ALT Treasure Data is processing millions of distributed SQL queries every day on the cloud. Upgrading the query engine service at this scale is challenging because we need to migrate all of the production queries of the customers to a new version while preserving the correctness and performance of the data processing pipelines. To ensure the quality of the query engines, we utilize our query logs to build customer-specific benchmarks and replay these queries with real customer data in a secure pre-production environment. To simulate millions of queries, we need effective minimization of test query sets and better reporting of the simulation results to proactively find incompatible changes and performance regression of the new version. This paper describes the overall design of our system and shares various challenges in maintaining the quality of the query engine service on the cloud.

ALT Modern web applications rely on underlying database engines to manage the increasing amount of user data. Unfortunately, the semantic gap between the application layer and the database layer easily leads to performance and correctness problems that severely affect the experience of web users. In this talk, I will discuss our work in understanding and tackling database related problems in modern web applications. Guided by our empirical study of this type of problems in open-source world, we developed checkers that found thousands of database-related problems in popular web applications, and IDE tools that help design database-friendly web pages.