<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=315693165909440&amp;ev=PageView&amp;noscript=1">

Upstream

The Big Data Blog

4 Examples of Data Lake Architectures on Amazon S3

Sep 27, 2018 8:12:13 PM / by Eran Levy posted in Data Architecture, Data Lake

So you’ve decided it’s time to overhaul your data architecture. What’s next? How do you go about building a data lake that delivers the results you’re expecting.

Well, we’re strong believers in the notion that an example is worth a thousand convoluted explanations. That's why this post is all about real-life examples of companies that have built their data lakes on Amazon S3. Use it for inspiration, reference or as your gateway to learn more about the different components you'll need to become familiar with for your own initiative.

Read More

Apache Kafka with and without a Data Lake

Sep 13, 2018 4:09:13 PM / by Eran Levy posted in Apache Kafka, Data Architecture, Data Lake

Apache Kafka is a cornerstone of many streaming data projects. However, it is only the first step in the potentially long and arduous process of transforming streams into workable, structured data. How should you design the rest of your data architecture to build a scalable, cost effective solution for working with Kafka data? Let’s look at two approaches - reading directly from Kafka vs creating a data lake - and understand when and how you should use each.

Read More

Understanding Data Lakes and Data Lake Platforms

Sep 5, 2018 4:22:46 PM / by Eran Levy posted in Data Lake, Data Lake Platform, Data Architecture

The following article is an abridged version of our new guide to Data Lakes and Data Lake Platforms - get the full version for free here.

If you’re working with data in any capacity, you should be familiar with Data Lakes. Even if you don’t need one today, the rapid growth of data and demand for increasingly versatile analytic use cases (such as reporting, machine learning, and predictive analytics) could result in your organization outgrowing its data infrastructure much sooner than you currently foresee.

Read More

Let's face it: Big Data sucks.

Aug 29, 2018 3:18:51 PM / by Ori Rafael posted in Big Data, Data Lake

If you only read the bombastic headlines, you might be forgiven for thinking that Big Data is the name of a real-life superhero: fighting crime, busting traffic jams and even curing diseases. But when you work with data for a living, you quickly find out that underneath the shiny facade, ‘doing big data’ is also a major pain.

Read More