SAP operates one of the largest test infrastructure to test its in-memory database SAP HANA. The infrastructure provides different services like continues integration, code coverage and code linting for a huge C++ project with Python test coding. These services are essential for the development teams and quality specialists. Without these services developing and shipping of new SAP HANA version wouldn’t be possible. In 2010, we started with a single Jenkins master with ten nodes. But to keep our testing time acceptable for the growing number of developers we had to scale up and that led to multiple different scaling challenges. The current test infrastructure is powered by more than thousand physical servers. Scaling of the infrastructure was only possible with custom optimizations like improved scheduling, expressive test configuration and robust tooling implemented in our favorite language Python. With the flexibility and power of Python it’s possible for developers to implement complex test scenarios to verify features and mitigate regressions. On infrastructure side, it has been easier to extend, optimize and adapt the infrastructure for new requirements like different CPU architectures and newer Operating systems versions.
This talk provides insights and stories how we scaled and improved our test infrastructure and how new technologies like Linux Containers can improve automated testing and software quality assurance.