Many articles and books have been written comparing Big Data to a tidal wave, a metaphor for the onslaught of data as a voluminous, impactful giant wall of water. Having spent time at the Hadoop World/O’Reilly Big Data conference in NYC recently, I liken it more to a tsunami — a wave of not only great volume, but also of great velocity, and one with a wide range of potential impact depending upon one’s particular vantage point. Those companies who either ignore this wave or who are ill prepared to receive it are at risk of being washed up or washed out to the sea of competitive irrelevance.
Of course, unlike a real tsunami, there is great, positive potential from the Big Data phenomenon. Not since the .com buildup of the late 1990’s have I witnessed such energy and enthusiasm around an enabling set of technologies. Fascinating work is being done by a vast number of technology companies and individuals that are rapidly bringing Big Data technologies to the mainstream, focusing on developing quicker paths to business value. These efforts and their tangible results are being realized across three primary focus areas or phases of Big Data:
Data management (data generation/acquisition and aggregation)
Visualization and end user absorption
This portends rapid and extensive change in the execution of Big Data strategies as technologies converge, mature, and advance. Much of this change will happen both on the open and commercial sides of this technology, and these changes will require expanded skill sets and resources to create and maintain an effective Big Data strategy.
Until today, Big Data initiatives are almost wholly owned in that secretive domain of the data scientist who is largely responsible for all the phases of Big Data mentioned above. This is my tsunami warning to IT organizations and providers: IT departments will be called upon more and more to handle the data management phase in order to offload this effort from that scarce data scientist resource; this shift will be the result of concentrating the scientists’ focus on the analytics, visualization and actual business absorption of the analytical results.
To that end, IT organizations and their key partners need to quickly become enabled in the special data management needs associated with Big Data. This includes things like real time data acquisition from the increasing sensor-sphere known as the “Internet Of Things,” to the management of copious amounts of unstructured data in a widely distributed manner. This will become an art unto itself because the whole premise of Big Data value is about bringing the computation TO the data, in as optimal a fashion as possible.
Companies, like Datalink, are prepared now for what I see as the future schism in Big Data: IT being the experts in Big Data management and working to feed the needs of the data scientists and the business.