The failure of a mission-critical web application can be costly.
Recently, the stock-trading app Robinhood went down for 24 hours during what was probably one of the stock market's most significant swings.
The reason for this major outage, according to the folks at Robinhood, was that their system struggled with unprecedented load and crashed. How many users and how much long-term revenue do you think Robinhood lost because of bad performance?
Remember that top engineering organizations consider performance not as nice-to-have, but as a crucial feature of their products. But most engineering teams do not regularly test the performance and scalability of their infrastructure, and most lack the tools to properly do so.
Although you can find plenty of commercial tool options out there, for the right organization, free and open-source tools may be a good alternative—or the perfect complement to your commercial tool set.
The value of performance
Before talking tools, let's consider load times and the value of performance. When you understand that performance is key to a great user experience, you need tools that will do one thing very well, and that's to measure the user's perceived load time.
There are many performance rules out there, but ultimately, load time is the only performance metric that matters.
How fast is fast enough for a web application? Here's a quick overview of key performance metrics:
- Under 100 milliseconds is perceived as instantaneous.
- A 100ms to 300ms delay is perceptible.
- One second is about the limit for the user's flow of thought to stay uninterrupted.
- Users expect a site to load in 2 seconds.
- After 3 seconds, 40% of visitors will abandon your site.
- 10 seconds is around the limit for keeping the user's attention.
So, as you can see, most software users want instant response.
Because of this, the importance of performance is increasing, and engineering teams need to treat performance as a feature. The goal of a performance load test is to understand how your applications behave under heavy load conditions.
To get started, you need to understand the baseline performance of your application and know that the performance of each transaction is unique.
For example, in an e-commerce application, a home page transaction is likely highly cached and very fast, whereas a checkout transaction is more complicated and must talk to a payment service, shipping service, etc.
To ensure that users have a great experience, you must test the most common flows for your users and understand performance both in the browser and on the server. To get the job done, you’ll need server-side, client-side, and performance tools, and you can find free and open-source options that fall into each of these categories.
But before jumping into load-testing tools, don't make the mistake of overlooking how your application performs with just one user.
Don’t ignore one-user performance
Believe it or not, most folks skip the most obvious place to start with performance—verifying that their application can perform well for one user before even starting concurrent load or stress testing.
James Pulley, practice manager of performance engineering and test at TEKsystems Global Services and a podcaster at PerfBytes, reminded me in our last TestGuild performance podcast that most engineers focus only on how their applications perform for many concurrent users.
How many, though, can answer whether the app scales for only one person? You have to ask that question before even thinking about a load test.
Tools for understanding client-side performance
Most modern applications spend more time in the browser than on the server side. The reason is that newer applications now are using JavaScript frameworks such as React and AngularJS.
These feature-rich front ends add a new layer that needs to be measured. So measuring the rendering time of an application that is under test on a client's local machine becomes critical.
Two of the best tools to use to understand client-side performance are:
- Google PageSpeed Insights, a service that analyzes the content of a web page and generates suggestions to make your pages load faster. Reducing page load times reduces bounce rates and increases conversion rates.
- Google Lighthouse is an open-source, automated tool for improving the quality of web pages. Your front-end developer should at a minimum be using Lighthouse metrics, which are available in Google Chrome tools.
Tools for understanding real-world performance
Sitespeed.io is my favorite tool for evaluating client-side performance from real browsers.
This open-source tool analyzes your website's speed and performance based on performance best practices and timing metrics. You can analyze one site, analyze and compare multiple sites, or let your continuous integration server break your build when you have exceeded your performance budget.
It is not always possible for teams to modify the applications to optimize client-side performance. Fortunately, Google invested in making ngx_pagespeed and mod_pagespeed as web server extensions to automate performance improvements without the need for code changes.
Here’s a description of each, as well as one other useful tool:
- Google ngx_pagespeed speeds up your site and reduces page load time. And this open-source nginx server module for this Pagespeed tool automatically applies web performance best practices to pages and associated assets (CSS, JavaScript, images) without requiring you to modify your existing content or workflow.
- Google mod_pagespeed also speeds up your site and reduces page load time. This open-source Apache HTTP server module automatically applies web performance best practices to pages and associated assets (CSS, JavaScript, images) without requiring that you modify your existing content or workflow.
- WebPagetest.org provides deep insights into the performance of the client side in a variety of real browsers. This utility will test a web page in any browser, from any location, over any network condition—and it's free.
Reuse your functional tests to get performance data
When getting started with performance testing, don't overlook another asset your software team probably already has—a functional testing script.
Some open-source functional test tools can be leveraged to gather some high-level performance info that should help your team.
- Cypress has a method that can capture Google Chrome's performance data for test runs. This allows you to find and fix common performance issues such as slow-time-to-first-byte.
- Selenium has a few ways to capture performance data as well. One common approach is using a HAR file. This is done using the BrowserMob Proxy (BMP). BMP lets you manipulate HTTP requests and responses, capture HTTP content, and export performance data as a HAR file.
- To use JMeter with Selenium Webdriver you can install the JMeter Webdriver Sampler. This is useful for testing the performance of AJAX, GWT-based web applications, and simulated user actions.
Once you have a good understanding of your client-side front-end performance, it's time to move on to some server-side load testing tools.
Tools for understanding server-side performance
Apache Bench and Siege are great for quick load tests from a single endpoint. If you just need to get a sense of the requests per second for an endpoint, these are great solutions.
A more advanced approach—and my personal preference—is Locust.io, an open-source load-testing framework that allows complex transactions and can generate high levels of concurrency with ease. I've included it in the list below, along with a few other tools to consider.
- Locust.io is great for understanding the performance on the server side.
- Bees with Machine Guns' authors describe it as "a utility for arming (creating) many bees (micro EC2 instances) to attack (load test) targets (web applications)."
- Multi-Mechanize is an open-source framework for performance and load testing that runs concurrent Python scripts to generate load (synthetic transactions) against a remote site or service. It's commonly used for web performance and scalability testing, but you can also use it to generate a workload against any remote API accessible from Python.
- Siege is an HTTP load-testing and benchmarking utility designed to let web developers measure code under duress, to see how it will stand up to load on the Internet. Siege supports basic authentication, cookies, and HTTP and HTTPS protocols, and lets the user hit a web server with a configurable number of simulated web browsers. Those browsers place the server "under siege."
- Apache Bench is useful for benchmarking your Apache HTTP server, to get an idea of how Apache performs.
- Httperf provides a flexible facility for generating varied HTTP workloads and measuring web server performance. The focus is not on implementing a particular benchmark but on providing a robust, high-performance tool that facilitates the construction of both micro- and macro-level benchmarks. The three distinguishing characteristics of httperf are its robustness, which includes the ability to generate and sustain server overload; its support for the HTTP/1.1 and SSL protocols; and its extensibility to new workload generators and performance measurements.
- Apache JMeter is for testing performance both on static and dynamic resources (files, servlets, Perl scripts, Java objects, databases and queries, FTP servers, and more). You can also use it to simulate a heavy load on a server, network, or object to test its strength or analyze overall performance under different load types. Finally, consider using it to make a graphical analysis of performance or to test your server/script/object behavior under a heavy concurrent load.
- Element is an open-source Puppeteer node library that uses a browser-based load-testing tool. You create scripts using Typescript, and can use them against your web app in the same way that your customers do, opening a browser and interacting with page elements.
Choices and trade-offs
While you have many open-source tools from which to choose for performance management, it's not always wise to build and manage your own performance testing tools and infrastructure.
You may determine that it's not worth the engineering resources required to load test when you can pay for a proven commercial product or service. Such tools let you more easily build, execute, and analyze performance tests, but you have to have the budget pay for it.
Should you go with open source, or will a commercial tool provide more value? Only you can decide if it makes sense to use open source—and if your organization has the engineering resources required. If you do, many of the tools I've discussed above should go right into your toolbox.
Taken together, these free and open-source tools offer a path to get started on capacity planning and load testing the server side, optimizing and performance testing the client side, and monitoring performance from end to end to derive meaningful insights from performance tests.
What are your favorite tools? Add your comments below.
This story was updated in March 2020 from an earlier version.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
