Big data: the next big thing
Brendan Crossey sums up the key trends in big data and the implications for how we live and work.
Big data is the next big thing, although it’s not quite there yet. Like cloud computing, it means many things to many people. Many people are trying to understand what it’s all about and what it means to them.
In the past if you wanted to capture data you built a database, structured it and knew what you wanted to capture. That may take 18 to 24 months of a cycle from planning, through design and implementation. The world is changing so fast that 99 per cent of the information you want to capture is not being captured in databases like that. It’s in social media and mobile platforms.
Nearly everything has a pulse i.e. a device with the ability to log and capture data. Gathering real intelligence comes from capturing that data and being able to hold and contain it. That analysis is also now very much on-demand. The onus is on capturing all the information you can, as you don’t know exactly what you will need to analyse in the future, and then carrying out on-demand reporting against that, particularly to analyse the unstructured data.
It makes sense for a business to understand the trends in its facebook comments and ‘likes’. On a public health level, Google searches for ‘flu symptoms’ are now recognised as an early warning sign for epidemics and the company rapidly reports those trends to the marketing departments of pharmaceutical companies while increasing the cost of the relevant sponsored links. The same principle applies to any industry, including insurance, other financial services and indeed government.
Analytics plays a strong part in EMC’s work, enabled by our mass collection and storage of data (at petabyte level) for increasingly longer periods of time. The growth in unstructured data outstrips that in structured data 100 times over. IDC predict that data will grow by 44 times over the next decade and 99 per cent of that growth will be in unstructured data.
Our solutions allow people to scale out and capture that data on demand, almost instantaneously being able to ‘plug and play’ additional storage. The internet search companies are leading on the analysis but much of their technology is now open-sourced.
EMC is currently commercialising the Hadoop open source data analytics package by validating it and putting it on an enterprise-ready support programme, linked in to the Greenplum analytical programme to achieve the same for structured data. This will give companies the ability to put in any unstructured query and get the information back as quickly as possible, without pre-defined search terms from all sources.
Businesses now want to distribute that considerable amount of data very quickly across their workforce via facebook-type internal social media services, to work out what it means and what can be done to fix problems. The final aspect is installing Amazon-type predictive analysis, to capture those understand trends before we know what we’re looking for.
We’re only at the beginning of the curve. Financial services and retail are moving ahead fastest as they see the quickest return, and competitive advantage and customer response is key to those sectors. With social media, millions of people can hear about poor customer service in an instant. On the other hand, if your company is doing something right, you need to scale that out and tell the story as widely as possible.
Validity
Cross-referencing captured data with other data sources is essential to make sure that your information is valid. One comment by itself means nothing but you need to reference that and see if a pattern is emerging.
Motorola has been successful by going across to social media groups, finding the top 10 influencers and giving them first access to their devices. Not everyone is equal in the social media world.
Mortgage loan risk calculations are normally done on a monthly basis, a long time in financial services. Greenplum can deliver daily risk updates, taking in information from social media which can show changes in a householder’s buying patterns.
In connected health services, such as those offered by Path XL in Belfast patients are being treated in the community rather than in hospital but devices are feeding back real-time analytics. This, in turn, reduces the cost of care and a real-time alert also generates a faster response compared to less regular checks in hospital. More data in healthcare lends itself to better results, just as when ICT companies run a large number of simulations over a certain period of time.
Big data also enables international collaboration amongst health professionals, in real time, sharing information including very large pathology images. In a given year, 200 million pathology slides are created. If that gets online, it is a big data problem.
Google Ventures is boosting its internal expertise in data analytics, a sign that the web giant’s VC group wants to deepen its presence in the growing big data field and it has identified life sciences as an area for investment.
These online trends also encourage businesses to think about what activities are core and add value, compared to those which are not. In the real world, for example, businesses do not build their own roads and railways to connect office but instead rely on the services provided by others.
Windows 8 will bring about a further major change, as it consists of a series of apps using a cloud-based architecture separating the ‘front end’ and ‘back end’ of a system.
The real value of these services are when these services intersect with Cloud Computing, PWC now estimate
75 per cent of all companies in Ireland are using Cloud Services.
One third of all data being created is now living in or going through the Cloud and organisations need to discover, understand and protect this information easily and quickly.
Social impact
In employment, around 25 per cent of people are not getting to interview stage because of their facebook content. Many people change their facebook accounts when they leave university and put forward a professional image. Furthermore, businesses will contact previous employers to ask about the suitability of the applicant and whether that matches with the profile.
ICT companies are also looking for smart energy data as that technology develops, and the cost of power makes that a more important trend. However, the rapid emergence of facebook (even as a business tool) shows that it is impossible to predict future trends.
On a final note, Cisco predicts that, by 2016, there will be 10 billion mobile internet devices for a global population of 7.3 billion people. Social media is certainly changing the world of work and the next generation will be communicating and trading in a completely different way than we did five years ago.
Brendan Crossey is Head of EMC in Northern Ireland
EMC Computer Systems (UK) Limited
Forsyth House
Cromac Square, Belfast, BT2 8LA
Tel: +44 (0)7768 526172
Email: brendan.crossey@emc.com
