Big Data … for Humans
By Daniel Krasner, Co-Founder, KFit Solutions LLC
“Big data” is omnipresent in the technology and business community. While significant resources have and continue to be dedicated to solving big data problems, neither the term nor its application have ever truly been defined. One natural definition of “big data” can be derived from how fast volumes are increasing. We can imagine some function which quantifies this rate and whose properties can be used determine whether the amount of data is truly “big” or otherwise. The function can take into account market realities, resources, technical limitations, etc. and this definition would play nicely with engineering advances that have largely focused on the problems of moving, storing, and processing. However, in addition to the underlying infrastructure and technology, we have a user — the human interacting with the information.
Despite the fact that in many instances the user experience (UX) is what brings utility and purpose to technology, the term “big data” in application to UX becomes even more opaque. In practice, the greater the ease of interaction and the higher the utility, the less important these infrastructure details are to the user. Although emerging technologies are increasingly taking into account human-computer interaction (HCI) when approaching business solutions, the industry has a ways to go in bringing HCI to the forefront.
The idea of big or simply “too much” data is nothing new. Humans have dealt with volumes of information too large for an individual to handle for centuries. Archives and libraries are a natural example and can serve as a useful prism for viewing today’s emerging technologies. In many ways archival research was solving a “big data” problem and that solution entailed proper organizational structure and a process flow that enabled for flexible, efficient information retrieval and exploration. Archivists, card catalogues, and ISBN all contributed to taking volume size out of the data equation and creating a highly utilitarian process. From the perspective of the user, the ease of functionality of a well organized archive made the actual infrastructure details irrelevant. It did not matter whether there were one thousand or one hundred thousand volumes present, or how hard it was to transport or store them.
The archival data interaction process can serve as a guidepost for industry solutions and their current shortcomings.There is no shortage of examples where the “big” in big data translates to a rigid and awkward user experience. Businesses intelligence questions which require database queries, analyses, and reports but need different tools and potentially people to carry out each task. Document review that begins with a keyword search or statistical process but leaves the user with no clear starting point or no natural way to alter course once the process has begun. These are just a few of the many current workflow problems. They are not intractable. Today’s engineering, coupled with machine learning and artificial intelligence, is advanced enough to tackle them and to bring the human to forefront of information processing.
“Big data,”regardless of the precise definition, is a much different concept when viewed from the perspective of user experience as opposed to infrastructure engineering. Arguably it should never be applied to the former, as a high-utility solution should make volume irrelevant to the end-user. That is not to say that every problem can be solved without the intervention of high-level engineering or statistics, that there are no solutions out there addressing these issues, or that there isn’t a historical reality which has underlined technological priorities since the boom of “big data.” However, products and solutions that take into account and optimize the natural human process flow are those which will gain the market. In many ways the stage is set: data volumes are high, scaling infrastructure is relatively inexpensive, machine learning applications are sufficiently advanced, and value of efficiently derived information is unarguable. It is the marriage of these with human-computer interaction, or perhaps now human-data interaction, which will change the way we use and profit from the information of the future.