<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=315693165909440&amp;ev=PageView&amp;noscript=1">


How VICOMI Cut DevOps Time from Weeks to Minutes
By Switching from Spark to Upsolver



Marketing technology and content analytics


Rip-and-replace for previous system based on Spark and Databricks to save time and improve performance


Tens of millions of events processed daily, billions monthly


Kinesis, Athena, ElasticSearch

The Backstory

Vicomi provides a platform for ‘emotional intelligence’. Using its software, major brands and publishers can add a widget that appears next to content published on their websites, enabling viewers to add emotional reactions (using emojis) while collecting data about these interactions. Vicomi then presents the data via interactive dashboards, helping their customers gain a better understanding of their audience and marketing efforts.

As Vicomi’s main business focus shifted to analytics and reporting, the company was looking for a way to improve the dashboards it was delivering to customers, while also cutting down the time it took to develop new dashboards or add features to existing ones.


The Goal

Simplifying and accelerating DevOps work to produce analytics on streaming data

The data

Vicomi collects large volumes of data from the websites where its tracking pixel is implemented. With approximately 400 million views of its widgets every month, tens of millions of new records are created daily, and on a monthly basis these amount to several billions.

The previous solution

Data is kept as JSON files and stored in a data lake hosted on Amazon S3. For the past 3 years, the company’s analytical infrastructure was based on Spark and Databricks, which were used to process the data that was then presented to customers via dashboards developed in-house.

The pain

While the previous solution was delivering reasonably results, it was cumbersome and extremely difficult to maintain. Every change in the dashboards’ underlying data - such as adding a new column to the table being used for a specific report - would require manual coding in Python, and run the risk of breaking another part of the complex system that was in place.

Seeing the problems caused by this lack of flexibility, Vicomi’s CTO Sagi Waitzman decided to try Upsolver’s Data Lake Platform for S3 data management and preparation.

Emotional Reactions


"Since we’ve started using Upsolver, we can make any change we want, it happens in literally minutes and it just works."

- Sagi Waitzman, CTO at Vicomi

The Solution

Agility and strong performance drive better, easier analytics for Vicomi’s streaming data

Minutes instead of weeks
to make changes to customer-facing dashboards

No coding needed,
with new tables and aggregations are created via Upsolver’s UI

1 developer
maintaining the entire data lake infrastructure

Data Architecture over S3

After taking a few weeks to get to know the ins and outs of Upsolver, Vicomi quickly started seeing the value of its self-service Data Lake Platform compared to the Spark and Databricks-based infrastructure that was in place.

While previously every change required writing Python, now Waitzman and his team could create new aggregations and tables on the fly through Upsolver’s drag-and-drop interface. This resulted in a massive reduction in DevOps time spent on analytics and reporting: if previously each changed required a week of meticulous coding and testing, the same could be achieved in minutes with Upsolver.

“With Spark, it used to be that every dashboard was considered ‘untouchable’ - as long as it was working, we didn’t want to break anything. Since we’ve started using Upsolver, we can make any change we want, it happens in literally minutes and it just works.”
Sagi Waitzman, CTO at Vicomi

Today Upsolver is an essential part of Vicomi’s core product: customer-facing analytics and reporting on the data its system collects; and it has played a major role in improving the system’s performance (in addition to making it easier to maintain). “We’re already seeing a major improvement in the performance of Athena and time it take to update dashboards”, says Waitzman. “Working with the Upsolver team, I’m sure we’re going to see even better results in the future.”

Streaming Data Analytics


By leveraging Upsolver to manage streaming data on S3 and deliver it for analysis with Athena, Vicomi managed to dramatically reduce the time required to develop new analytic features and reports.


Vicomi’s customers enjoy low-latency access to a wealth of data, and are highly appreciative of Vicomi’s ability to rapidly release new dashboards and drill-downs for richer insights.