Reinventing Pipelines for the Cloud Data Lake

Upsolver is a closely-knit group of data engineers and infrastructure developers who are obsessed with making data pipelines so easy to create that any data practitioner could do it, even when dealing with streaming and complex data and big data scale. We built the first version of Upsolver for our own use, and enjoyed eating our own dog food so much that we decided to bring this value to every company doing analytics in the cloud. We now claim numerous companies as customers spanning industries as diverse as oil & gas to gaming, feeding both operational analytics and customer-facing data products.

What we lost in the transition to big data

In the early days of Upsolver we recognized that something important was lost in the transition to big data. Decades ago, databases were invented to abstract away the complexity of coding over raw file systems, letting data teams “speak” in schemas and SQL, which let them transform data more easily and iterate faster.

In the new millenium, big data processing platforms like MapReduce and Spark were adopted to slash costs dramatically as volumes and velocity exploded. However, they arrived without many of the abstractions that made databases so easy to use.
The responsibility to bridge this gap fell to the new data engineering profession. They had to not only code transformations, but specify the query execution and orchestrate all of the jobs that would make the transformation cost-effective and performant.

Back to the future

Upsolver’s goal is to re-invent the simplicity of a database on top of the tremendous economic and “power at scale” advantages of a cloud data lake. That is why we focused on using SQL as the lingua franca for designing pipelines, and why we automate all of the pipeline plumbing that used to be taken for granted in traditional databases, but was left behind with the advent of Hadoop and Spark.

By eliminating any compromise across the dimensions of power, simplicity and affordability, we are transforming data pipelines for cloud analytics workloads. Data engineers who use Upsolver become much more productive, delivering data pipelines that normally would have taken months, in hours. Plus, SQL-savvy data consumers such as data analysts and data scientists now can transform complex raw data, without having to spend all of their time on data engineering.

Realizing the potential of the cloud data lake

This will widen the aperture of analytics, unlocking data that heretofore has been difficult to prepare. System logs, clickstreams, IoT sensor feeds, and other continuous data sources will be blended with static and historical data to create query-ready data sets that feed reporting or ad hoc queries with unmatched data freshness, as well as customer-facing data products.

Over the longer term, Upsolver will empower the cloud data lake to be a key component of every company’s data stack, acting as the central hub that connects raw data to analytics systems from cloud data warehouses to ML systems, graph databases and beyond.