It’s no surprise that after the success of last winter’s online test automation conference, Guild Conferences founder Joe Colantonio would follow up with another event, this one on performance testing.
With nearly three days of prerecorded content, plus live half-hour Q&As, the entire event can't be covered in one article, but here are five deep insights and one performance nugget.
[ Special: PerfGuild performance testing conference ]
Agile should make performance testing easy
Alex Podelko, a performance engineer at Oracle, talked about reinventing performance testing. Podelko compared the classic model, where software isn’t testable until it's code-complete, to today’s agile environments, where a simple working system is available within two or four weeks of project start. Podelko recommended involving a performance engineer throughout the project. Performance tends to degrade as projects get larger and more complex. Running the same performance testing script between sprints (or more often) means finding performance problems earlier—and tracking down the source of the problem becomes much easier.
Yet performance testing is not easy, or at least not as easy as it could be, even in an agile environment. Podelko suggested these scenarios when deciding what types of performance testing to do:
Test on cloud, physical servers, or production? There are scenarios for each
One of the most-asked questions in performance testing is what kind of test servers to use. The general tradeoff is that building clones of the software under test in the cloud is easy, though not very realistic, while physical test servers that match production capacity are expensive. Podelko suggested that each has its place:
- Use cloud clients and the production system for system validation for high load. This will have low repeatability, because the database is changing, but high validity.
- Use an internal lab for performance tuning and troubleshooting. This will have high repeatability but not visibility into true, end-to-end performance at scale.
- Use cloud servers for early and light continuous testing. This will have lower costs and lower scope but may not give results that are realistic—unless production is in the same cloud environment with the same setup.
Debugging the internal lab is probably more than just running the same predefined script over and over—something both Podelko and Mark Tomlinson discussed during their talks.
Automated performance testing pipelines and exploration: A mix of both
Automated performance testing pipelines tend to be repeatable: Create a test server, put the latest build on it, create so many users, simulate activities with those users, measure performance, compare to a baseline, send a message if there is an error. To facilitate getting fast feedback, they tend to be small and powerful.
Occasionally, you may need to do another kind of test. You might notice that a certain activity is slow, or it might show up as slow in the server logs. And you might know a change is localized to a certain API and want to test just that API at load.
That is the sort of work Podelko suggests performing in the lab. It likely won’t happen with one end-to-end run. Instead, you’ll vary the load and system configuration, looking at different pieces, perhaps in isolation, to identify the weakest part—or to determine if the particular subsystem you are performance-testing is the weakest part at all.
Which leads to the next step, performance investigation.
Prefer performance investigation over simple performance testing
The conference had two general styles of talks: tool presentations and talks on the practice of performance testing. Tomlinson’s talk, on advanced performance exploration, covered this in some detail. Tomlinson pointed out that a computer has four essential properties: disk, CPU, network, and memory. When one of these properties becomes overly busy, it can become a bottleneck. For example, if a system is calculating pi to a thousand digits, you’ll probably wait for the CPU to do all the calculations, even if disk, memory, and network are all fine.
Tomlinson suggested the utilities Top, for Unix, and Task Manager, for Windows, as simple monitoring tools. Once you find the slowest component (API, database, web server, etc.), monitor with those tools to find the resource that is slowing down the process.
Perhaps you don’t need the performance script at all. Many applications start out visibly slow on the first round, using a strategy of “make it work, make it right, make it fast.”
One place to start is with performance tuning. It’s possible the “working right” system can handle only a handful of users. “System crashes at ten simultaneous users” is a lot less helpful than “Search API fails at 10 simultaneous users because of CPU on API server.” That kind of information looks more like performance and capacity engineering, and less like testing.
Tomlinson explained that everything in the system is a computer, from the servers to the routers and the load balancer. It’s possible that even the antennas on the router are little computers (CPU, disk, memory) that are coordinated by a larger computer on the router. The process of performance investigation, then, is finding the bottleneck resource, then the reason it is a bottleneck (say, the CPU is causing a delay), then tweaking either the system resources, configuration, the code, or some other element and retesting.
A computer can have several levels: the pure hardware, the operating system, any system or framework software, and finally the custom application code that the computer is running. That’s four different levels for every computer, and multiple computers for any system, including the front-end web servers, API servers, database servers, load balancers, and other components.
Tomlinson walked the audience through these layers, suggesting testing at every level to identify where the problem is:
Tomlinson wasn’t the only presenter to talk about performance engineering as a discipline. Todd DeCapua, the director of technology and product innovation for CSC, presented a slightly different model.
Move from load testing to performance engineering
While he wanted to avoid the term maturity model, DeCapua agreed that simple load testing is something to evolve beyond. In his talk on “effective performance engineering,” DeCapua defined load testing as system- and hardware-focused, with a goal of either “breaking” the system or meeting some service-level agreement (SLA). Tests typically run inside the data center, using prerecorded packet transmissions.
Those tests see how long data takes to get off the server but don’t take transmission to the end client into account, and they don’t look at how long it takes to render images on the screen. Once the tests simulate the full user experience, they move up to what DeCapua calls “performance testing.” After performance testing, he looks to run tests in a continuous integration system, catching regressions quickly. Once that’s complete, he suggests the use of service virtualization to test each component in isolation, to know both when and where regressions are introduced.
Finally, a performance nugget
Tomlinson’s podcast, PerfBytes, has a regular section called “news of the damned,” where he and his co-host, James Pulley, discuss recent IT performance failures. Asked by the moderator for the single most common problem, Tomlinson stated that nine times out of ten, it is the lack of a content delivery network (CDN)—a company that stores static files on multiple servers, throughout the world, to save load on an organization’s servers. If your company wants to do grownup-level website traffic, get a CDN.
Putting the lessons together
There’s more than one way to do performance testing; a website designed for 50 customer service agents will need much less work than a major e-commerce site with 24/7 availability and customers from all over the world. Still, there are some commonalities: model customer use, test often, and add more value by identifying bottlenecks, not just declaring pass/fail.
The process of performance testing is investigative. Strive to test early and often and to find more valuable information over time.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
