Business intelligence and analytics software
Building and managing a data lake on S3; Running ad-hoc analytic queries on streaming data to better understand product data
Product logs - over 200BN records, 150 gigabytes of new data created daily, 20 terabytes overall
Sisense is one of the leading software providers in the highly competitive business intelligence and analytics space, and has been named a Visionary by Gartner for the past two years. Headquartered in New York and serving a global market, the company provides an end-to-end BI platform that enables users to reveal business insights from complex data. Sisense counts Nasdaq, Airbus, Philips and the Salvation Army among its customers.
Seeking to expand the scope of its internal analytics, Sisense set out to build a data lake in the AWS cloud in order to more effectively store and analyze product usage data - and following a recommendation from the AWS team, began looking into Upsolver’s Data Lake Platform.
Enabling self-service data lake operations, instead of a siloed engineering team
As an analytics company, data-driven decision making is part of Sisense’s DNA - and this includes product usage data, which the company uses to improve processes across its Sales, Customer Success and Product departments.
However, the rapid growth in its customer base created a massive influx of data that had accumulated to over 200bn records, with over 150gb of new event data created daily and 20 terabytes overall.
To effectively manage this sprawling data stream, Sisense set out to build a data lake on AWS S3, from which they could deliver structured datasets for further analysis using its own software - and to do so in a way that was agile, quick and cost-effective.
CTO and Co-founder Guy Boyangu, who was leading the process, says: “I wanted to make sure we’re creating a data lake rather than a data swamp, and with billions of records and over 1000 fields I knew this would be a challenge - especially because I didn’t want to devote an entire team of engineers just for this. From my experience, this team would quickly become a ‘silo’ that would not provide the agility we needed, which was absolutely crucial”.
- Transform data streams into structured fact tables that could then be sent to Sisense’s BI software
- Ability to quickly iterate and answer new business questions as they arise
- Self-service solution that would not require a dedicated data engineering team
An agile data lake solution for agile analytics
Implemented by one person, working in his spare time
From raw data to production-ready tables in a few weeks
No additional hires or technology required
Guy began experimenting with Upsolver in his ‘spare time’ amidst a host of additional responsibilities, but was delighted to discover that this was all he needed to do to start seeing results. After previously trying to run Athena directly on the 70bn stored records, which proved to be very cumbersome, Guy now had a simple way to enrich the data and deliver structured datasets to Sisense’s analytics platform.
Upsolver’s ability to rapidly deliver new tables generated from the streaming data, along with the powerful analytics and visualization capabilities of Sisense’s software, made it incredibly simple for the Sisense team to analyze internal data streams and gain insights from user behavior - including the ability to easily slice and dice the data and pull specific datasets needed to answer ad-hoc business questions.
“I saw a demo of Upsolver and it looked amazing, but I thought - ‘well, it’s just a demo’... then I started working with the software and could get exactly the structures I needed out of the hundreds of gigabytes we were streaming daily, and it just worked.”
Guy Boyangu, Co-founder and CTO, Sisense
“Upsolver is the shortest path from streaming to usable data.”
Upsolver is already in production, and provides powerful new querying capabilities for business analysts within Sisense. “The BI analysts are absolutely ecstatic to have this type of data, at this granularity, so easily accessible”, says Guy.
Working in tandem with Athena and Sisense’s own BI software, Upsolver will further simplify the way additional teams within the company make data-driven decisions: from product managers having a better understanding of how the software is being used, through sales representatives gaining real-time insights into prospect behavior, to data scientists working on machine learning algorithms.
“Our software lets users do things that are far beyond their original skillset when it comes to business intelligence - and Upsolver does the same thing for data engineering. I never used Spark-Streaming, Spark or Hive; but with Upsolver, I could handle massive amounts of streaming data and see real results, in the fraction of the time I thought it would take.”
Guy Boyangu, Co-founder and CTO, Sisense
- Ability to run new analytical queries on streaming data in Sisense
- Much-improved visibility into internal data
- Additional use cases around machine learning being rolled out
- Agile infrastructure that enables new queries and tables to be generated on the fly
- Data lake project could be handled internally and without the need to devote a team of engineers