Automated testing with 400TB memory

SAP operates a large test infrastructure to test its in-memory database SAP HANA. In 2010, we started with a single Jenkins master with ten nodes. But to keep our testing time acceptable for the growing number of developers we had to scale up, which led to multiple different scaling challenges. The current test infrastructure is powered by more than thousand physical servers and provides different services like continuous integration, code coverage and code linting for a huge C++ project. These services are essential for developing and shipping new SAP HANA versions. This talk provides insights into how we scaled and improved our test infrastructure. All custom implementations like improved scheduling, expressive test configuration and caching of artifacts were implemented in our favorite programming language Python. With the flexibility and power of Python it has been easier to extend, optimize and adapt the infrastructure for new requirements like different CPU architectures and newer operating system versions.