VICOMI: How VICOMI Cut DevOps Time from Weeks to Minutes By Switching from Spark to Upsolver

CUSTOMER STORY

INDUSTRY: Marketing technology and content analytics
USE CASE: Rip-and-replace for previous system based on Spark and Databricks to save time and improve performance
DATA: Tens of millions of events processed daily, billions monthly
INTEGRATIONS: Kinesis, Athena, ElasticSearch
Minutes
instead of weeks to make changes to customer-facing dashboards
1 developer
maintaining the entire data lake infrastructure
No coding
needed with new tables and aggregations are created via Upsolver’s UI

Vicomi provides a platform for ‘emotional intelligence’. Using its software, major brands and publishers can add a widget that appears next to content published on their websites, enabling viewers to add emotional reactions (using emojis) while collecting data about these interactions. Vicomi then presents the data via interactive dashboards, helping their customers gain a better understanding of their audience and marketing efforts.

As Vicomi’s main business focus shifted to analytics and reporting, the company was looking for a way to improve the dashboards it was delivering to customers, while also cutting down the time it took to develop new dashboards or add features to existing ones.

The Goal

The data

Vicomi collects large volumes of data from the websites where its tracking pixel is implemented. With approximately 400 million views of its widgets every month, tens of millions of new records are created daily, and on a monthly basis these amount to several billions.

The previous solution

Data is kept as JSON files and stored in a data lake hosted on Amazon S3. For the past 3 years, the company’s analytical infrastructure was based on Spark and Databricks, which were used to process the data that was then presented to customers via dashboards developed in-house.

The pain

While the previous solution was delivering reasonably results, it was cumbersome and extremely difficult to maintain. Every change in the dashboards’ underlying data – such as adding a new column to the table being used for a specific report – would require manual coding in Python, and run the risk of breaking another part of the complex system that was in place.

Seeing the problems caused by this lack of flexibility, Vicomi’s CTO Sagi Waitzman decided to try Upsolver’s Data Lake Platform for S3 data management and preparation.

Minutes
instead of weeks to make changes to customer-facing dashboards
1 developer
maintaining the entire data lake infrastructure
No coding
needed with new tables and aggregations are created via Upsolver’s UI

The Results

After taking a few weeks to get to know the ins and outs of Upsolver, Vicomi quickly started seeing the value of its self-service Data Lake Platform compared to the Spark and Databricks-based infrastructure that was in place.

While previously every change required writing Python, now Waitzman and his team could create new aggregations and tables on the fly through Upsolver’s drag-and-drop interface. This resulted in a massive reduction in DevOps time spent on analytics and reporting: if previously each changed required a week of meticulous coding and testing, the same could be achieved in minutes with Upsolver.

Today Upsolver is an essential part of Vicomi’s core product: customer-facing analytics and reporting on the data its system collects; and it has played a major role in improving the system’s performance (in addition to making it easier to maintain). “We’re already seeing a major improvement in the performance of Athena and time it take to update dashboards”, says Waitzman. “Working with the Upsolver team, I’m sure we’re going to see even better results in the future.”

quote icon With Spark, it used to be that every dashboard was considered ‘untouchable’ – as long as it was working, we didn’t want to break anything. Since we’ve started using Upsolver, we can make any change we want, it happens in literally minutes and it just works.