One major roadblock that prevents teams from successfully maturing their performance efforts is a lack of training. Sometimes, just knowing where to start can be a challenge.
Before getting into specific training resources, however, it might be helpful to define performance engineering and describe how it differs from performance testing.
In their book Effective Performance Engineering, Todd DeCapua and Shane Evans describe performance engineering as a cultural shift in the way organizations view their essential processes. Performance engineering embraces practices and capabilities that build quality and performance throughout an organization.
It helps your organization increase revenue, customer attraction and retention, brand value, and competitive advantage—all while meeting or exceeding the expectations of your end users.
For its part, performance engineering means understanding how all the parts of the system fit together, knowing which metrics matter, and building in performance from the first design.
This list of tools and resources is a good place to begin. It can help transform your organizations into performance engineering centers of excellence.
Maturity levels and how they relate to performance engineering
DeCapua describes these five main stages of maturity:
- Level 1: Randomly breaking things
- Level 2: Performance testing
- Level 3: Performance analytics
- Level 4: Lifecycle virtualization
- Level 5: "Performance" engineering
Performance engineering is the top layer, Level 5, which all organizations should strive to reach.
Once you understand what performance engineering is, you still need to convince management, and getting everyone on board with a performance engineering transformation can be a challenge.
The web performance optimization (WPO) stats compiled here can help. Case studies and experiments demonstrate WPO's impact on user experience and business metrics. If you're trying to convince your organization that performance testing is important to your software development efforts, this is a great place to start. Keep in mind that WPO is just one small piece of performance engineering.
One-user and front-end performance
As companies begin to shift left in their performance activities, one-user performance is one area that is often overlooked.
James Pulley, an independent consultant working with PerfBytes, said in a recent TestGuild performance podcast that most engineers focus on how their applications perform for many concurrent users.
It's equally important, though, to know whether an app also scales for one. Client-side, one-user performance is important because testing has moved to the front end, with a focus on user experience (UX) performance.
Modern applications spend more time in the browser than on the server side, with many apps using JavaScript frameworks such as React and Angular front ends.
These feature-rich front ends add a new layer that needs to be measured. So the measurement of the rendering time of an application that is under test on a client's local machine becomes critical.
This also leads to applications becoming more complex, with heavier front ends that use up more resources. Because of this, there's a need to also include performance testing from the user's perspective.
Guy Arieli, CEO of Experitest, believes that mobile UX performance is even more significant than the apps' performance. When you work with your browser on your desktop, you are connected via Wi-Fi. And the bandwidth that you're using will be relatively high compared to anything you will find in 3G or even 4G, especially with changing network conditions and where the reception is not very good.
A pillar of performance engineering is that performance needs to be baked into your software from the beginning, with developers creating performant code in your product from the start.
Here are some resources to help you test all of the above.
Tools to capture front-end performance
Webpagetest is the standard for measuring performance from users' perspectives and for capturing visual metrics. It is a free service that provides a website speed test; it allows you to run your test from multiple locations using real browsers at real consumer connection speeds. You can also create your own instance of a web page test and run it behind the firewall to verify your intranet pages or pages that are under development. Webpagetest give you test location options that include testing from an Android or iOS device as well.
Detox Instruments is a performance analysis and testing framework designed to help developers profile their mobile apps to better understand and optimize their behavior and performance. It can help your developers find issues early in the development cycle for mobile apps.
Cristian Daniel Marquez Barrios, a front-end developer with over 10 years of experience, suggests that your front-end developers should be using, at a minimum, Google Lighthouse metrics. These are available in Google Chrome tools. When you open DevTools in Chrome and navigate to the Audits tab, you can find the Lighthouse audit. This will measure the performance of your site, simulating mobile on a desktop version of a browser, so it's a really good tool to start with.
Sitespeed.io is a set of open-source tools that make it easy to measure the performance of your website. It includes not only the software that can run performance tests, but also Grafana (see below).
Grafana collects and presents results from several test rounds, which means you can monitor the performance of your site. The tool also allows you to take a lot of additional actions, such as verifying the performance of all the pages of your site, or selected pages. You can also simulate different network connections, and much more.
Other tools Sitespeed.io gives you include:
- Coach, which can help make your web page faster. Coach acts like a modern version of YSlow, and gives you advice for how your page could be faster.
- Browsertime; this lets you automate running JavaScript in your browser to collect performance-timing metrics of your website.
- Compare, which matches two HAR files and finds regressions.
- PageXray; this converts a HAR file to a JSON format that is easier to read.
- Throttle, which simulates slow network connections on Linux and macOS X and helps you see how the application performs when bandwidth is limited.
- Chrome-HAR, which creates Chrome HAR files based on events from the Chrome Debugging Protocol.
Other client-side performance resources
If your application is very heavy on JavaScript client-side functionality, you're going to want to measure that as well. Here are some other tools to analyze web pages, monitor performance, and find front-end code quality issues.
- YSlow analyzes web pages and determines why they're slow, based on Yahoo's rules for high-performance websites.
- YellowLab Tools contain an online test to help speed up heavy web pages.
- Varvy tests your site to determine whether it follows the Google guidelines for speed.
- Benchmark.js is a benchmarking library that supports high-resolution timers and returns statistically significant results.
- PerfMap allows you to create a front-end performance heat map of resources loaded in the browser using the Resource Timing API.
- Phantomas is a PhantomJS-based web performance metrics collection and monitoring tool.
- Webperf Heat Maps generates performance heat maps of web pages.
- Zombie.js is a lightweight framework for testing client-side JavaScript code in a simulated environment. No browser is required.
Performance testing example sites
If you're just getting started or don't have an app to run performance tests against, it's useful to have an example app to test against to help you as you learn. Here are a few web pages designed solely for this purpose.
- If you're an old-time Mercury automation engineer, you'll probably remember the sample site called Mercury Tours that shipped with WinRunner, QTP, and UFT. You can also use it as an example performance testing site.
- Another old-time favorite website for performance testing is a full web application called JpetStore; it's built on top of MyBatis 3, Spring 5, and Stripes.
- Blazemeter also has a popular test site; it's called BlazeDemo.
Resources for performance testing newbies
In addition to Effective Performance Engineering, the book already mentioned above, here are a few other resources:
- The Every Computer Performance Book, by Bob Wescott. This is a short, witty book on how to solve and avoid application and/or computer performance problems.
- PerfBytes is a must-listen podcast focusing on all things performance testing.
- Señor Performo’s blog is a fun site for beginners, explaining basic concepts with silly examples.
- Performance Testing Guidance for Web Applications is a free e-guide that shows you an end-to-end approach for implementing performance testing.
- The Test Guild Performance and Site Reliability Engineering podcast is a great show for beginners in the performance space.
- Designing for Performance, by Lara Callender Hogan; this book helps you approach projects with page speed in mind, showing you how to test and benchmark the most critical design choices.
- Practical Scalability Analysis with the Universal Scalability Law, by Baron Schwartz; this helps you understand the connections between scalability and other disciplines, such as performance optimization and the study of queuing.
[ Related story: 6 test automation tools developers love ]
Performance testing tools
Once you have a website to test against and a basic understanding of performance testing, you can take a look at some open-source performance testing tool options:
- JMeter is the most popular open-source tool in the load-testing space. Apache’s JMeter describes itself as a Java application that was designed specifically for load testing with the ability to measure performance.
- Taurus has the user-friendly option of allowing you to write your tests in YAML. You can describe a full-blown script in about 10 lines of text, which gives your teams the ability to describe their tests in a YAML or JSON file.
- Gatling is a stress tool built on Scala, Akka, and Netty. It allows you to test and measure your application’s end-to-end performance.
- Locust is a simple-to-use, distributed, Python-based load-testing tool.
- k6 is a developer-centric, open-source load-testing tool for testing the performance of your back-end infrastructure.
- Grinder is a load-testing Java framework that makes it easy to run a distributed test that leverages a host of load injector machines.
- Fortio is a load testing library, command line tool, advanced echo server, and web UI in go (golang). It allows you to specify a set query-per-second load and record latency histograms and other useful stats.
Performance monitoring tools
Monitoring performance is a key element to ensure that your application will perform perfectly in the wild. However, if you look at Gartner's Magic Quadrant for Application Performance Monitoring report, all the tools listed are paid tools.
For the folks who don’t have a budget for paid tools, here are some free APM solutions you should check out.
- Pinpoint is currently the most popular application performance management tool on Github.
- JavaMelody monitors JavaEE apps.
- Skywalking monitors microservices and container-based architectures.
- Stagemonitor monitors Java server applications.
- Dynatrace's Appmon offers a personal license you can use free for life in your local environment.
- The Prefix web request profiler can help you validate the behavior of your code and find hidden exceptions in your code. It can also find slow SQL query issues.
Google performance resources
Google has a bunch of great tools to help you understand how to improve the performance of your web and other applications. Here are a few:
- Google Developer Tools can help you get started analyzing run time performance issues.
- As previously mentioned, Google Lighthouse is an open-source, automated tool that audits performance, accessibility, progressive web apps, and more.
- Web Fundamentals contains explanations, examples, and recommendations that focus on low-effort, high-return performance wins.
- Google PageSpeed reports on the performance of a page on both mobile and desktop devices, providing suggestions about how that page can be improved.
- Chrome User Experience Report is a treasure trove of metrics, powered by real-user measurement data. Knowing this data can help you create more realistic tests for how your application will be used in production.
Test infrastructure performance resources
Paul Bruce, DevOps advocate and automation and performance engineer at Neotys, says that in his experience, the traditional folks managing performance are missing the huge dangers and opportunities of the Kubernetes world.
Here are some Kubernetes performance testing resources:
- Kubernetes perf-tests is a repository dedicated to storing various Kubernetes-related performance test-related tools. If you want to add your own load test, benchmark, framework, or other tool, contact one of the owners.
- JMeter Cluster supports Kubernetes and OpenShift.
- Measuring of API performance describes a test plan for quantifying the API performance of container cluster systems.
- Load Testing Kubernetes is a video from the Cloud Native Computing Foundation to help you optimize your cluster resource allocation in production.
Let's not forget Docker. Here are some Docker images that have performance testing tools baked into them from the beginning:
- JMeter Docker is a lightweight Docker image for Apache JMeter.
- Jenkins Performance Plugin allows you to capture reports from popular testing tools. Jenkins will generate graphics with a trend report of performance and robustness. It allows you to set the final build status as good, unstable, or failed, based on the reported error percentage.
- Keptn is a control-plane for DevOps automation of cloud-native applications. Keptn uses a declarative approach to build automation for delivery and operations, which can be scaled to a large number of services.
Site reliability
Because I'm old school, I still call many software development activities "performance testing," but I’ve begun to see a trend that has this term falling by the wayside.
Mark Tomlinson, the "performance sherpa," has an interesting theory. He says that if you listen to developers speak, they tend to use the term "webperf" rather than "performance testing."
So it's very clear that developers and engineers are doing performance work. Web designers are doing performance work. Infrastructure teams are doing performance work. And none of them self-identify as "testers" in their organizations.
So, if you’re like me and still call it performance testing, don’t ignore resources that also exist for site reliability engineers or infrastructure teams. Here are some examples:
- The Site Reliability Engineering Book: In this free e-book, members of Google's SRE team explain how their engagement with the entire software lifecycle has helped Google build, deploy, monitor, and maintain some of the largest software systems in the world.
- Chaos Engineering: In this report, members of the Netflix team that developed chaos engineering explain how to apply these principles to your own system. By introducing controlled experiments, you'll learn how behavior from component interactions can cause your system to drift into an unsafe, chaotic state.
- Chaos Monkey randomly terminates virtual machine instances and containers that run inside your production environment. Exposing engineers to failures more frequently gives them incentives to build resilient services.
If you’ve tried using Chaos Monkey in the past, you may have found it difficult to use and understand. There’s another product built on top of it called Gremlin that makes it easier to get started.
- Gremlin Free provides the ability to run both Shutdown and CPU attacks on your hosts or containers, controlled by a simple user interface at no cost.
That's my list. What did I miss? Share your suggestions in a comment below.
For more great tips and practical information on performance testing and engineering, join us for the PerfGuild online conference, which runs June 1-2, 2020.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
