The Importance of Fast and Slow in Big Data

By Benjamin Schmidt, CTO, kWantera

The emergence of Big Data and Big Data technologies over the last few years has necessitated a sweeping change in the way companies react and interact to customer and business needs. While the field is still evolving, a few major trends are better understood. Most important in our experience has been the separation of fast moving application development from the slower moving Big Data infrastructure. This is critical to deriving business value from Big Data projects as rapidly as possible while continuing to derive value over the life of the project.

In the past couple of years, organizations of all sizes have attempted to rapidly implement Big Data technology stacks and put them to work immediately attempting to find business value. If not carefully planned one of two scenarios are likely to play out. In the first, a massive system is built, tested and deployed to solve the “today” problems. The database, provisioning mechanisms, network infrastructure, and application code are all built to a small design specification allowing for maximum speed. With a good engineering team it will go well and the system functions properly but when the next set of problems emerges engineers and technicians find it difficult to adapt the existing system and “bloat” occurs. New infrastructure is required, network gear requires upgrades, application code needs to modify to take advantage of newer system infrastructure, etc. Each successive iteration of the next problem contributes to the existing problems compounding them until eventually the response time to new business needs grinds to a halt.

In the second scenario, the system is designed to be future-proof against some likely business requests as they emerge. Engineers and technicians get to work building this new more elegant system. As development continues, the predicted business needs continue to evolve and new design specs are implemented necessitating some re-work. Perhaps data center layouts were inadequate, network gear was not sufficient, or business logic code was not abstract enough. Without anything deployed into production, the changes are relatively easy. But as you can imagine, this cycle can easily continue indefinitely without any business value being seen by management.

The issue is that Big Data systems require two different philosophies to coexist peacefully in order for the business value to be extracted over the long term. Namely, the slow moving technologies that underpin Big Data have to be separated from the faster moving business needs that use Big Data technologies. Without an effective strategy for managing these two parallel paths, the organization will end up in one of the scenarios above.

So how does an organization take action? At the outset, the most important decision is to organize your teams according to this design philosophy. Business logic and application development can operate on shorter time-scales and allow fast, flexible, and reactionary interactions with business needs. The slower moving Big Data components that execute those applications need to have milestones that extend much further into the future. A tight collaborations must exist between Big Data application developers and the infrastructure engineers that deploy the Big Data systems. With a clear understanding of the different delivery time-lines then the technical teams will be free to focus on adding value to the organization.

In many cases, this will require all the stakeholders to agree on this view of Big Data ecosystem and understand the new constraints that everyone will operate under. Many of the Big Data systems treats Big Data as a single monolithic entity but it is important to realize that Big Data investments cannot be onetime events. They should be seen as a continuous investment in the long term competitive edge for the organization and will need to grow with that organization. It is only when organizations come to focus on the fast and slow components of Big Data that these projects will be a success.