Data Handling & Data Engineering
18 talks from the 2026 edition.
-
Beyond Kafka and S3: Python Data Pipelines with HTTP-Native Bytestreams ▶ Recording 📝 Transcript
Johannes Dröge
Real-time bytestreams between systems in different organizations or secured environments, whether for batch dataset delivery or continuous streaming, are surprisingly hard. Traditional solutions fall short: message...
-
Building reliable data pipelines with polars and dataframely ▶ Recording 📝 Transcript
Oliver Borchert, Andreas Albert
If you have worked with real-world data before, you know that processing it can be challenging. Data often comes scattered across tables, in inconsistent encodings, with duplicated rows and is generally dirty. In...
-
Building Trust in Your Data Pipelines with Observability ▶ Recording 📝 Transcript
Stefan Dienst
In the daily work of a data engineer, building new data pipelines often takes priority, while maintaining them and ensuring their correctness becomes an afterthought. This focus can quickly turn into a pitfall:...
-
Dynamic Knowledge Graphs ▶ Recording 📝 Transcript
Jakob Leander Müller
Traditional RAG systems struggle to understand holistic connections in distributed, constantly changing knowledge sources that characterize real-world organizations. While document-based approaches using vector...
-
Exploring Germany's Urban Geography with Census and OpenStreetMap Data ▶ Recording 📝 Transcript
Travis Hathaway
When conducting studies of the urban form, an important resource many researchers turn to is the massive OpenStreetMap dataset. But, as extensive as this dataset is, it lacks one very important aspect about the...
-
Fight your garbage data: implementation of a pythonic data quality monitoring framework in PySpark ▶ Recording 📝 Transcript
Rostislaw Krassow, Joshua Finger
The timeless phrase “garbage in, garbage out” is even more important today with the growing usage of non-deterministic generative neuronal networks, which amplifies the effect of bad data quality. This presentation...
-
From Prompt to Production: How to use AI Code Assistants for Python Data Systems ▶ Recording 📝 Transcript
Serhii Sokolenko
**Code-generating LLMs have matured** to the point where they can reliably scaffold **data pipelines and data agents**, when used in a **supervised, engineering-first workflow**. This tutorial demonstrates how to...
-
From Row-Wise to Columnar: Speeding Up PySpark UDFs with Arrow and Polars ▶ Recording 📝 Transcript
Aimilios Tsouvelekakis
Python UDFs often become the slowest part of PySpark pipelines because they run row-by-row and pay a high cost crossing the JVM↔Python boundary. Spark’s Arrow-backed execution changes that cost model by moving data...
-
From Struggling to Mastery: A Practical Guide to Data Pipeline Operations ▶ Recording
Akif Cakir
How mature are your data pipeline operations? A Roadmap to Operational Excellence. Data teams often struggle to scale their pipeline operations, trapped in a cycle of manual fixes and reactive fire-fighting. But what...
-
How to Search Through 800 Billion Records in Real Time ▶ Recording 📝 Transcript
Mirano Tuk, Filip Bacic
Large-scale distributed systems rarely produce clean data streams. In practice, hundreds of services continuously emit overlapping updates, retries, corrections, and partial state. Turning that constant stream of...
-
Making my Apache Spark™ talk more interesting using AI ▶ Recording 📝 Transcript
Celeste Horgan
Writing talks is hard, but being a good conference speaker is even harder. Resultantly, this talk is recursive: I'll take a talk previously written for a London data science meetup on using Apache Spark and Apache...
-
Open Table Formats in the Wild™ - Reloaded: Vortexing Ducks over Floating Icebergs ▶ Recording 📝 Transcript
Franz Wöllert
Open table formats have *almost* freed us from vendor lock-in. They form a critical building block of the modern, composable data stack. The most prominent open table format is Apache Iceberg - not only because of...
-
Rediscovering single-node processing: When does it make sense to move from Spark to Polars? ▶ Recording 📝 Transcript
Jonas Böer
As data engineers, we are used to spinning up a Spark Cluster every time we want to do data processing and handle the overhead that comes with using such a mighty framework. But is this really necessary? In this talk...
-
Ship Data with Confidence: Declarative Validation for PySpark & Pandas ▶ Recording 📝 Transcript
Ryan Sequeira
Tired of data quality issues crashing your PySpark and Pandas pipelines? This talk introduces [dataframe-expectations](https://github.com/getyourguide/dataframe-expectations), a lightweight, open-source library for...
-
SQL is Dead, Long Live SQL: Engineering reliable analytics agent from scratch ▶ Recording 📝 Transcript
Mehdi Ouazza, Dumky de Wilde
Is it still worth learning SQL in 2026, or can we just "chat" with our data? This hands-on tutorial explores that exact question by pushing Text-to-SQL to its absolute limits. This won't be just happy paths; we will...
-
To nest, or not to nest? Nested data types in Polars with big data ▶ Recording 📝 Transcript
Daniel Finnan
Do you find yourself weighing up the pros and cons of using nested types in the Polars library - pondering whether you should encode your variables in structures using lists, arrays or opt for a flat format without...
-
Wetterdienst: Fast, Unified Access to Open Weather Data with Polars ▶ Recording
Benjamin
Weather and environmental data power analytics, ML, and operations—but APIs differ wildly and data prep is slow. Wetterdienst is a Python library that provides a unified, Polars‑first interface to multiple weather...
-
Zero-Copy or Zero-Speed? The hidden overhead of PySpark, Arrow & SynapseML for inference ▶ Recording 📝 Transcript
Petar Ilijevski
"Zero-copy" data transfer promises free communication between Spark's JVM and Python workers, but at 6 billion rows daily, the reality is far more complex. This session explores the low-level mechanics of distributed...