To understand the value of using a tool such as Upsolver for ingest pipelines, we need to understand the types of challenges most organizations encounter when writing data to object storage such as Amazon S3. These include:

• Architectural challenges: Data lakes are often built on the premise of ‘store data now, analyze later’. While this is a useful paradigm for data exploration, it means you’re often building the lake without knowing exactly what you want to do with the data, which could lead to data being dumped into object storage without a clear strategy for ingestion and operationalization. This can become problematic since data which is not ingested and stored according to best practices will be very difficult to access further down the line, turning your data lake into a data swamp. Implementing these best practices requires a high level of technical expertise.
• Functional challenges around maintaining performance at scale – this includes continuous storage optimization which is best done upon ingestion, such as converting ingested files from JSON format to Apache Parquet, choosing the right partitioning strategy, and compacting small files. Exactly-once processing of event streams from Kafka or Kinesis, without losing or duplicating messages, is also notoriously difficult to implement. These issues are exacerbated by the data lake’s unstructured nature and lack of visibility which creates a reliance on ‘blind ETL’.

Automating Data Lake Ingestion