Vicomi provides a platform for ‘emotional intelligence’. Using its software, major brands and publishers can add a widget that appears next to content published on their websites, enabling viewers to add emotional reactions (using emojis) while collecting data about these interactions. Vicomi then presents the data via interactive dashboards, helping their customers gain a better understanding of their audience and marketing efforts.
As Vicomi’s main business focus shifted to analytics and reporting, the company was looking for a way to improve the dashboards it was delivering to customers, while also cutting down the time it took to develop new dashboards or add features to existing ones.
Vicomi collects large volumes of data from the websites where its tracking pixel is implemented. With approximately 400 million views of its widgets every month, tens of millions of new records are created daily, and on a monthly basis these amount to several billions.
Data is kept as JSON files and stored in a data lake hosted on Amazon S3. For the past 3 years, the company’s analytical infrastructure was based on Spark and Databricks, which were used to process the data that was then presented to customers via dashboards developed in-house.
While the previous solution was delivering reasonably results, it was cumbersome and extremely difficult to maintain. Every change in the dashboards’ underlying data – such as adding a new column to the table being used for a specific report – would require manual coding in Python, and run the risk of breaking another part of the complex system that was in place.
Seeing the problems caused by this lack of flexibility, Vicomi’s CTO Sagi Waitzman decided to try Upsolver’s Data Lake Platform for S3 data management and preparation.
After taking a few weeks to get to know the ins and outs of Upsolver, Vicomi quickly started seeing the value of its self-service Data Lake Platform compared to the Spark and Databricks-based infrastructure that was in place.
While previously every change required writing Python, now Waitzman and his team could create new aggregations and tables on the fly through Upsolver’s drag-and-drop interface. This resulted in a massive reduction in DevOps time spent on analytics and reporting: if previously each changed required a week of meticulous coding and testing, the same could be achieved in minutes with Upsolver.
Today Upsolver is an essential part of Vicomi’s core product: customer-facing analytics and reporting on the data its system collects; and it has played a major role in improving the system’s performance (in addition to making it easier to maintain). “We’re already seeing a major improvement in the performance of Athena and time it take to update dashboards”, says Waitzman. “Working with the Upsolver team, I’m sure we’re going to see even better results in the future.”