BIG DATA – Transformation from Hype to Real

By Kalaivani Arun, Big Data Solutions Architect , ideas2IT Technologies

Big Data has been creating a lot of excitement in the last 5 years. A 2015 IDG Enterprise Study states there has been a 125% increase in number of organizations that have implemented or deployed data driven projects in the last one year. Big Data not figuring in the Gartner hype cycle 2015 indicates that Big Data no longer remains a hype and is fast turning into reality.

As the conversion from Hype to Real happens, we also see a lot of Big Data projects getting derailed. In fact, the numbers are astoundingly more than regular technology initiatives. Only one-third of these projects are considered successful. So, how can Enterprises ensure more success with Big Data projects?

Here are our five key observations from our experience, to successfully build a data driven organization.

1) A strong business problem as a driver: An initiative for its own sake gets side-lined when a pressing business problem surfaces. One of our client organizations started a Big Data initiative and chose a platform to invest in. Mid way through, they discovered that the platform and getting all the data onto it was not something they really needed to be doing. Joining the Big Data bandwagon because of the hype around it, may not provide the necessary thrust to move the organization ahead. There has to be a strong business case that drives the initial set of projects in an organization. One need not look far for these business cases. There are so many use cases in any industry for Predictive analytics. Making business and technology folks aware of its application could yield great results by triggering ideas for innovative applications within your organization.

2)Lack of Skill sets: Some organizations start out by trying to recruit few data scientists and get them to carve out Big Data use cases and needs for the organization. This is often disastrous as data scientists often work towards solutions for defined problems. Discovering problem statements is best left to Business. It is best to start with a Chief Data Architect who is well versed with what goes on in the industry, can draw roadmaps, evaluate technologies and define the skill needs in the first place. This person can then decide whether the organization wants to ‘Build or Buy’ and if building, whether to outsource or build it themselves. A mature evaluation criterion for technologies, solutions and vendor capabilities can help fill the skill set gaps.

3)Realistic Expectations from Big Data projects: Making investments in data projects often come with an associated tag of what benefits do I get from this investment. While it is important to do a cost benefit analysis before the investment itself, expectations need to be set on the long term outcome and recurring benefits that may not even be quantifiable at the outset. Even smaller scale predictive analytics type projects often suffer from issues such as wanting to know the exact ROI/improvement in business outcome from that investment. Often times, it is a discovery once the project has started and at some point a ‘Go – No Go’ decision point has to be included before further investments are made. Clear expectation setting & publishing risks upfront help in ensuring that senior leadership and sponsors don’t get disillusioned if they do not see immediate returns.

4)Technology as a Means to an End: There are several technologies, players and multitude of products to harness the power of Data. The landscape is also exponentially expanding. As much as this provides flexibility and choice, it also creates confusion during selection. Making Technology choices without a 360 degree evaluation of the tools and technologies and its fit for purpose will often derail these initiatives. For example, Technology choices for Storage needs and Analytics needs are vastly different. Again within Analytics, Latency plays a crucial role in deciding the type of data storage and processing. Are users just consumers of Analytics, do they want to play around with models? There are several elements to consider before deciding on the complete Enterprise solution. Bottom-line is, Technology should be chosen to fit the need of the Business.

5)The Big Picture at the organization: The Chief Data Architect at the organization needs to have an end to end view of how the landscape is going to turn out to be based on the use. The organization strategy and drivers need to be clearly defined initially. What data needs to be covered, where data needs to be stored, how will it be processed, what are the typical use cases for this data, who are the users, how will they use it (dynamic/ static analysis), etc needs to be thought through. A big chunk of the initiatives may be driven based on business needs. Unless they are tied to the overall Data strategy and Information Architecture, these are bound to be smaller silos of initiatives often necessitating larger reengineering effort sooner than later. Worse still, immediately after having spent a lot on say, a Hadoop implementation, the ‘What Next’ question will pop up.

A well thought out strategy will clearly leverage each implementation and offer incremental benefits for each spend. It keeps expectations real across the organization, has backing of leaders and keeps the initiative alive until it realizes its purpose.

While all of this may sound like bigger hurdles to cross for an organization that has just started its Big Data journey, it helps to remember that it is initiatives like these that are going to be providing the business its competitive edge. The rest of world is getting on board, so why don’t we?


Send your email to