Build a modern data infrastructure

During this tutorial you will learn how to create a scalable and reliable data infrastructure using OSS libraries.

Given the limited amount of time we will focus mainly on the batch side but some useful insights will be shared about stream processing and OLAP.

The first part will introduce the main concepts while the second will be focused on building stuff.

Tutorial outline

First part

• Why bother?
• Unified data warehouse
• Stream VS Batch (Fast VS Slow data)
• Kafka? I'd rather use Redis
• Short term storage VS Long term storage
• Airflow

Second part

• Build a Python consumer
• Use Airflow to move things around
• Use Airflow to train machine learning models
• What's next?

Tools that we are going to use

• Redis streams
• Scylla
• Airflow
• Docker
• Pandas and Parquet
• Python ❤️

Prerequisites

• If I say clone this repo locally and download these docker images you don't freak out
• A good knowledge of Python