Marketing technology and content analytics
Rip-and-replace for previous system based on Spark and Databricks to save time and improve performance
Tens of millions of events processed daily, billions monthly
Kinesis, Athena, ElasticSearch
Vicomi provides a platform for ‘emotional intelligence’. Using its software, major brands and publishers can add a widget that appears next to content published on their websites, enabling viewers to add emotional reactions (using emojis) while collecting data about these interactions. Vicomi then presents the data via interactive dashboards, helping their customers gain a better understanding of their audience and marketing efforts.
As Vicomi’s main business focus shifted to analytics and reporting, the company was looking for a way to improve the dashboards it was delivering to customers, while also cutting down the time it took to develop new dashboards or add features to existing ones.
Simplifying and accelerating DevOps work to produce analytics on streaming data
Vicomi collects large volumes of data from the websites where its tracking pixel is implemented. With approximately 400 million views of its widgets every month, tens of millions of new records are created daily, and on a monthly basis these amount to several billions.
The previous solution
Data is kept as JSON files and stored in a data lake hosted on Amazon S3. For the past 3 years, the company’s analytical infrastructure was based on Spark and Databricks, which were used to process the data that was then presented to customers via dashboards developed in-house.
While the previous solution was delivering reasonably results, it was cumbersome and extremely difficult to maintain. Every change in the dashboards’ underlying data - such as adding a new column to the table being used for a specific report - would require manual coding in Python, and run the risk of breaking another part of the complex system that was in place.
Seeing the problems caused by this lack of flexibility, Vicomi’s CTO Sagi Waitzman decided to try Upsolver’s Data Lake Platform for S3 data management and preparation.
"Since we’ve started using Upsolver, we can make any change we want, it happens in literally minutes and it just works."
- Sagi Waitzman, CTO at Vicomi
Agility and strong performance drive better, easier analytics for Vicomi’s streaming data
Minutes instead of weeks
to make changes to customer-facing dashboards
No coding needed,
with new tables and aggregations are created via Upsolver’s UI
maintaining the entire data lake infrastructure
After taking a few weeks to get to know the ins and outs of Upsolver, Vicomi quickly started seeing the value of its self-service Data Lake Platform compared to the Spark and Databricks-based infrastructure that was in place.
While previously every change required writing Python, now Waitzman and his team could create new aggregations and tables on the fly through Upsolver’s drag-and-drop interface. This resulted in a massive reduction in DevOps time spent on analytics and reporting: if previously each changed required a week of meticulous coding and testing, the same could be achieved in minutes with Upsolver.
“With Spark, it used to be that every dashboard was considered ‘untouchable’ - as long as it was working, we didn’t want to break anything. Since we’ve started using Upsolver, we can make any change we want, it happens in literally minutes and it just works.”
Sagi Waitzman, CTO at Vicomi
Today Upsolver is an essential part of Vicomi’s core product: customer-facing analytics and reporting on the data its system collects; and it has played a major role in improving the system’s performance (in addition to making it easier to maintain). “We’re already seeing a major improvement in the performance of Athena and time it take to update dashboards”, says Waitzman. “Working with the Upsolver team, I’m sure we’re going to see even better results in the future.”
By leveraging Upsolver to manage streaming data on S3 and deliver it for analysis with Athena, Vicomi managed to dramatically reduce the time required to develop new analytic features and reports.