How to benchmark application performance from the user's perspective

What kind of performance does your application have, and how do you know? More to the point, what kind of performance do your end users think your application has? 

In this era of rapid growth and unpredictable traffic surges, understanding your application's scalability is not just a technical concern — it's a strategic imperative for success. Providing end users with optimal performance is of course non-negotiable, and benchmarking it is a key way to meet their expectations. 

Nor do you want to simply benchmark your application’s components — you need to benchmark entire critical user journeys (CUJs) end-to-end, as the user perceives them, to provide a holistic view of the application's performance under real-world conditions. Benchmarking components in isolation may overlook potential bottlenecks and performance issues that arise from the interplay of various components, network latencies, and external dependencies. By simulating complete user flows, you gain insights into the actual user experience, enabling you to identify and address performance issues that impact user satisfaction and engagement.

In this blog post, we discuss why it is important to incorporate end-user perceived performance benchmarking into modern application development, and how to incubate an organizational culture that benchmarks applications from the get-go — and that continues to benchmark over time. As part of the Google Kubernetes Engine (GKE) engineering team, we also show you how we use the open-source Locust tool to simulate complex user behavior to use as part of your end-to-end benchmarking practice.

Why benchmarking is indispensable 

There are many reasons you should institute robust benchmarking practices as part of your application development process: 

• Proactive performance management: Early and frequent benchmarking helps developers identify and address performance bottlenecks early in the development cycle, which in turn can help to conserve resources, accelerate time-to-market, and deliver smoother product launches. Additionally, integrating benchmarking into testing protocols can add a critical safety net, safeguarding code quality and user experience by promptly detecting and addressing performance regressions.

• Continuous performance optimization: Applications are dynamic, subject to continuous evolution, scaling, and changes in user behavior. Regular benchmarking facilitates ongoing monitoring of performance trends, allowing developers to evaluate the impact of new features, updates, and system modifications, thus helping the application remain consistently performant and responsive in the face of change.

• Bridging the gap between development and production: As part of a development process, benchmarking real-world workloads, images, and scaling patterns provides a realistic assessment of application performance in a production environment. This helps developers proactively address potential issues and smooth transitions from development to deployment.

Benchmarking scenarios to mimic real-world load patterns

As a developer, your goal should be to benchmark your applications in conditions that closely mirror real-world scenarios, including deployment, scaling, and load patterns. This approach tests how well applications handle unexpected traffic surges without compromising performance or user experience.

As part of the GKE engineering team, we perform extensive benchmarking across various scenarios to test and optimize our cluster and workload autoscalers. This helps us understand how our autoscaling mechanisms respond to varying demands, maximizing resource utilization and maintaining optimal application performance.

Locust for realistic load testing and performance benchmarking

Locust is a sophisticated yet simple-to-use load-testing tool that helps developers simulate complex user behavior through scripting, offering a comprehensive understanding of application performance under realistic conditions. By defining and instantiating "users" who perform specific tasks, Locust enables the creation of various load scenarios.

In one example benchmark, we employed Locust to simulate users accessing a web server and requesting the 30th Fibonacci number. This generated a consistent load of approximately 200 ms per request, with each connection closed and re-established to ensure load balancing across different pods.

Locust makes it relatively easy to simulate these kinds of intricate user interactions within your application. It can generate up to 10,000 requests per second on a single machine, and can scale further through distributed deployment out-of-the-box. It provides granular control over user count and spawn rate via custom load shapes, enabling you to emulate real-world load patterns with users who exhibit diverse load profiles. It natively supports HTTP/HTTPS protocols for web and REST requests, and is extensible to a wide array of systems, including XML-RPC, gRPC, and various request-based libraries/SDKs.

Accompanying this blog post, we provide a GitHub repository to illustrate an end-to-end benchmark of a pre-release autoscaling cluster environment. You are encouraged to adapt it to align with your specific requirements.

Benchmarking end-users’ perceived performance is not just a best practice; it's imperative for delivering exceptional user experiences. By proactively integrating benchmarking into the development process, developers can discover whether their applications remain performant, responsive, and capable of meeting varying user demand.

Tools like Locust help simulate real-world scenarios, so you can gain deeper insights into your application performance under diverse conditions. Performance is an ongoing pursuit. Let benchmarking guide you toward delivering exceptional user experiences.

At Google Cloud, we’re always working to make GKE faster, more scalable and efficient under the hood. Upgrade your Kubernetes environments early to capitalize on the latest improvements in GKE, like the Cluster Autoscaler improvements introduced in GKE v1.29.