4 Important Lessons from Working with Big Data
By Ionela Postolache, COO, EAST VISION SYSTEMS
1) Always work with production numbers: they are unexpected and require immediate actions
When working with Big Data, especially in the Online Advertising domain, numbers can be unexpected. We must actually have a small percentage of the total traffic that hits the servers’ factored in as “unexpected behavior” and “natural discrepancies”. Which is why, working with production numbers is oh-so-important! for refining your algorithms and working paradigm.
No virtual emulators and custom-made simulation of various traffic sources can actually match the unexpected surprises the production environment does. Also, on simulators, data is behaving with fewer discrepancies due to the relative uniformity of its characteristics. What we suggest for anyone dipping their toes into this field is to let things take their natural course, migrate on production, and stand-by. There are companies who actually bring down production servers intentionally to train their teams react on crisis situations. And, rest assure, problems are bound to occur.
2) Sometimes custom-way is the only way
We have products that respond and adapt to many of the Big Data challenges. For instance, one of the most popular requirements we got was to handle event processing. More specifically, video event processing. This takes a lot of resources in hardware, which, for numbers reaching 20 millions hits per minute, can translate into pretty substantial sever bills. Naturally, we had to come up with a solution that can take this load of the infrastructure without loosing any data. It also had to be able to scale up to two times with a 75 % maximum recommended load on the hardware. So, we developed our proprietary software that does precisely that. The final bill dropped with 80 % and everybody was pretty happy about it.
Key is to not stop at what you can find on your first web search. Sure there are other solutions out there that behave somewhat similarly with ours. But they still need to undergo in house modifications. They still need to be acquired with possible licenses. And, most of all, they will never fit as perfectly as ours to our particular set of requirements. Least, the pre-exiting solutions are hard to debug should customization so require. There are cases when the original vendors need to be contacted to troubleshoot something. This is, obviously, not the case with something we build from scratch.
3) Have an outstanding recruiting process (for top 5 % real talent)
All this is not possible without an outstanding team behind everything. Is pretty difficult to expect quick, creative solution to live environments’ situations if your team members are not self-sufficient and, not only masters of the technology at had, but creative enough to find programmatic solutions under time pressure. And, as we all know, good people are hard to come by. Which is why, and this is almost as important as the production expertise, investing in the company branding and recruiting is an essential point.
However, to have such techies in your group, your brand has to be well established and your interactions with IT profile faculties a frequent routine.
4) Attend profile conferences to keep traction
Ok, so we have production numbers, solid methodologies proven over time and a great team to work our magic. Now what happens? Well, as in all activity domains, now, the business part comes into place. Interactions with clients and peers are instructive, informative and fun. And, where a better place to stay in touch than various profile conferences?
Several times a year, we pack our bags and our technically savvy egos and go meet the world. Sometimes is just to be on the loop with latest technologies to adopt ourselves. Sometimes, as it was the case this year, at EuroPython, we go to tell the world what we have been doing, share a little of our expertise with peers and hold a prestigious presentation.