Wetterdienst: Fast, Unified Access to Open Weather Data with Polars
In this presentation, Benjamin Gutzmann, a Data Engineer at Otto Group data.works, introduces Wetterdienst, a Python library designed to simplify the complex process of accessing open weather data. Because weather and environmental APIs vary wildly in format and structure, data engineers often spend more time on "plumbing" than on actual analysis. Benjamin explains how Wetterdienst solves this by providing a unified, Polars-first interface that standardizes request patterns across multiple global services, including the DWD, NOAA/NWS, and ECCC.
Viewers will learn how the library normalizes inconsistent data into tidy, long-format DataFrames using SI units and UTC timestamps, while implementing robust caching and retry mechanisms to ensure pipeline reliability. Benjamin walks through the provider architecture and demonstrates practical workflows for station discovery, timeseries retrieval, and exporting data to databases. Whether you are building ETL pipelines or training ML models, this talk provides a blueprint for integrating weather data via Python, CLI, or REST API to accelerate your analytics and operations.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track Data Handling & Data Engineering and was classified suitable for novice domain / intermediate python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
Problem
Accessing weather data means wrestling with inconsistent APIs, formats, and units—slowing down data engineering and making pipelines hard to reproduce.
Solution
Wetterdienst is a Python library providing a unified, Polars-first interface to multiple open weather services (DWD, ECCC, EA, NOAA/NWS, Geosphere Austria, IMGW, Eaufrance, WSV, and more). It standardizes request patterns, returns tidy long-format data in SI units, and handles caching, timezones, and retries—so teams can focus on analysis instead of plumbing.
Core concepts:
- Polars-first — All data operations use Polars (v1.15+); pandas supported for some I/O
- Declarative request pattern — Provider → stations → values; tidy/long output by default
- Sensible defaults — UTC timestamps, SI units, humanized parameter names
- Reliability — Disk-based caching via diskcache, stamina-based retries, timezone handling
- Provider architecture — Consistent interfaces across DWD, ECCC, EA, NOAA/NWS, Geosphere, IMGW, Eaufrance, WSV, and more
- Multiple interfaces — Python API, CLI, and REST
Outline
- Introduction
- Journey — How Wetterdienst came to life
- Wetterdienst — Architecture, concepts, and request patterns
- Value — What wetterdienst offers you, me and everyone else
- Demo — Live: station discovery, timeseries retrieval, station metadata, climate stripes and more via app
Target Audience
Data engineers, scientists, and platform teams who need reliable weather data for analytics, ML, and operations.
Prerequisites
Basic Python and DataFrame experience (Polars or pandas); familiarity with ETL/ML pipelines helpful.
Key Takeaways
- A unified, Polars-first workflow to access and normalize open weather data
- Practical patterns for station discovery, timeseries retrieval, unit conversion, and caching
- How to integrate Wetterdienst via Python, CLI, and REST, and export to common formats and databases
Links
📦 Repo https://github.com/earthobservations/wetterdienst 📖 Docs https://wetterdienst.readthedocs.io/ 🌐 App https://wetterdienst.eobs.org/ 💡 Examples https://github.com/earthobservations/wetterdienst/tree/main/examples
