Cloud load testing (CLT) is one of the areas of software development practice that seem to have generated a lot of hype over the last few years. And the frequency at which I hear people talking about its capabilities is only increasing.
CLT gives you the capability of driving load to your application(s) under test from cloud environment(s), so that you get quick results from your load and performance test scenarios. But many of the discussions I have with folks interested in cloud load testing get stuck at a feature and function level. None of that really matters.
What I’d first like to focus on are the reasons you should be using CLT. I will then describe the relationship between cloud computing and performance, including what makes cloud load testing different. And finally, I will help you try out three popular CLT packages so you can get a feel for how these tools work in your own environment.
The 3 Bs of cloud load testing
When I explain why teams should be using CLT, I start with the three Bs, as follows.
Business
You need to get performance and load-testing capabilities quickly and easily to the lines of business (LoBs). As awareness about the correlation between performance and revenue and other key indicators for organizations has grown, so has LoBs' concern that the technology solutions that enable their business are performing as they need to.
The objectives of LoBs are to meet specific targets for achievement. In the last five to ten years, this has typically required a significant increase in the rate at which LoBs contract with third parties or build out their own technology capabilities to enable faster achievement of their business targets and goals.
Along with this trend, what was formerly considered core IT capabilities are now moving to LoBs, which are tasked with achieving business results on their own. That means bringing CLT capabilities to the LoB, and this trend coincides with the growing quantity of bimodal IT and how that manifests itself within many organizations. Specifically, the phenomenon of an LoB designing and developing technology capabilities outside of central IT is likely here to stay. But the LoB still needs the ability to verify and validate performance, among other key indicators.
Burst
In every business and industry there are times of promotions, sales, or forecast increase in end-user activity. This increased activity creates a “burst” within the business and the applications and infrastructure that enable organizations to operate and deliver products and services to end users.
CLT provides an easy way to proactively create and execute burst tests. This means that, if your normal on-premises performance-testing tool is licensed for 10,000 Vusers but you want to leave the script the same while tripling the volume to 30,000 Vusers, you can simply run your 10,000 Vusers and use CLT to generate the remaining 20,000 Vusers. That provides you with your target 30,000-Vuser scenario and associated results prior to this happening in production.
Having a capability to quickly and easily ensure that your organization will be able to support these bursts in business will not only enable you to deliver value to your end users, but also meet and exceed financial (and other) targets for your businesses in a cost-aware way.
Benefits
There are three major benefits to CLT.
Time to value: CLT enables you to quickly get your desired results more quickly.
Scalability: Often, up-and-coming organizations have less on-site and more cloud-based resources, which reduces short-term capital expense. Cloud resources allow you to scale with your business and provide this capability on demand from the cloud for millions of virtual users.
Results: Anything that does not provide results is overhead and reduces value. You want to execute and deliver automated results and be able to focus on running the business with confidence.
The cloud and performance
Now let's consider how CLT can improve performance and how it's specifically different from other modes of load testing.
The cloud and performance are talked about seemingly more than any others in the enterprise, for in the digital age, time equals money. I mean that literally, because this old axiom has actually been quantified. Amazon calculated that a one-second delay in the loading time of its website could result in $1.6 billion in lost sales annually.
Conventional wisdom once held that a web page had a full eight seconds to load before visitors would give up and hit the back button. Today, web users are more discriminating, and their attention begins to wander in half that time, at best. Web properties that are slow or offline are subject to all kinds of negative effects, ranging from lower Google rankings to suppressed sales and conversion rates.
As websites and apps have migrated to the cloud, enterprises have lost some control over issues such as performance and latency. The cloud provider may have an SLA and an uptime assurance, but unless you’re actively testing those promises, you may never really know if the provider is honoring its guarantees.
What's different about cloud load testing?
Enter CLT, or cloud load testing (also known as cloud performance testing). Enterprises are using this increasingly popular technique to demand more from their cloud infrastructures. In fact, a recent YouGov survey of IT executives found that more than 70 percent of enterprises are already using CLT tools in their organizations.
CLT is becoming an essential tool in any IT director’s arsenal, and adopting this type of testing makes sense for a great many reasons.
First, compared to traditional load testing, CLT is incredibly simple. For the most part, running a cloud load test involves setting a few parameters or writing a simple script and then clicking the run button. Because the resources required to run these tests are all based in the cloud, there are no complex hardware stacks to manage or racks of servers to spin up. The hard work is all handled behind the scenes, over the web. Cloud load tests run quickly and can give you insight into the performance of your web-based application based on thousands of virtual users.
CLT also offers a significant upgrade to traditional testing methodologies by giving testers access to, literally, the entire world. Tests can be run from nodes located around the globe, all at once, or staggered over time. These nodes can be clustered and segmented into groups defined by various device types, network speeds, and operating systems, and variables can be changed as needed—or kept exactly the same, to ensure that the same test is repeatable should you make a change to your cloud environment. The configurability of CLT simply cannot be matched by traditional means.
And finally, if the above benefits of CLT don't sway you, note that it is generally very affordable. Licensing can be flexible and based on a pay-as-you-go or pay-as-you-use arrangement, letting you spin up resources only when needed. If your cloud testing needs are only periodic, it doesn’t make sense to shell out major coin for a system you’ll only use once in a while.
A look at the cloud load-testing market
The CLT market is relatively compact, composed of both major tech veterans and smaller startups. Information about the actual breakdown of this market is tough to come by, so we commissioned YouGov to survey IT professionals to find out which CLT tools they were using, including both enterprise-grade tools and more casual load-testing packages. The survey focused on eight of the most commonly used CLT packages. Here’s a brief look at each one, including two designed specifically for the enterprise.
- Hewlett Packard Enterprise StormRunner Load — An enterprise-class product designed to weave CLT into an agile development environment through its own scripting system. Scales above 1 million virtual users and produces over 20 analytical charts and graphs based on its testing results. Integrates with CI server tech, including Jenkins, Hudson, and TeamCity.
- SOASTA CloudTest — An enterprise-class testing driven by a complex and intricate visual testing environment. Works with both websites and apps. Scales up to over 1 million users and supports 47 global testing locations. Integrates with Jenkins and Travis CI platforms. Offers on-demand or on-premise versions of its software.
- LoadImpact — Though it scales up to 1.2 million virtual users, the more common use case for LoadImpact is more casual, with plans priced on a monthly subscription basis to handle testing at the level of 1,000 to 8,000 virtual users. Plugs in to Jenkins and TeamCity.
- LoadStorm — Similar to LoadImpact but with considerably higher virtual user limits, with subscription plans ranging up to 500,000 concurrent users. (Custom plans scale into the “millions.”) Relies on a point-and-click scripting system to generate loads, but most plans also include some free consulting time if you have trouble.
- SendGrid Loader — One of the simplest and least expensive CLT products in common use, Loader has only two plans: A free one supporting 10,000 virtual users (a huge number compared to other free or demo services), and a paid one that supports 100,000 users. Does not include more advanced testing options such as geolocation of virtual users. Integrates with Jenkins and TeamCity.
- Spirent Blitz — Blitz steps up the more casual cloud test plan that supports virtual users from multiple regions, but maxes out at 50,000 users per region in its most expensive plan. Supports multistep transactions and integrates with Bamboo CI server.
- SmartBear LoadComplete — Unlike other casual services (which operate via a web interface), this is a Windows desktop application with a visual test-scripting system, à la StormRunner Load. Virtual user support for mainstream plans is capped at 5,000 virtual users; an “enterprise” plan lets you climb above 10,000.
- BlazeMeter — Another casual product, with support for 3,000 concurrent users on its standard subscription plan, though an “enterprise” plan promises unlimited virtual users. Integrates with numerous CI tools, including Jenkins, Bamboo, Travis, and TeamCity.
In the survey, StormRunner Load clearly dominates the market, with one-third of those surveyed saying they preferred this platform. The remainder of the field each commands between 7 percent and 12 percent of the market, based on those surveyed.
Getting started with three popular cloud load-testing apps, step by step
Ready to give CLT a try? Here’s a step-by-step guide on how to sign up for, install, and get going with three popular packages.
HPE StormRunner Load
First, visit the StormRunner Load download page and click on the Free Trial link. On the subsequent page, click Start Trial. This will give you access to ten tests, each with up to 100 concurrent API virtual users or 30 UI virtual users, over the next 30 days. Fill out some basic contact info (no credit card is required to demo any of these services) and complete the process by clicking the Sign Up link. A confirmation will be sent to your email account, after which your instance of StormRunner Load will be created.
Sign in again through the link provided via email and click Launch next to StormRunner Load. StormRunner Load offers three options for creating a script to launch your first cloud load test.
- VuGen is a tool that can record web-based applications at the transport level.
- TruClient is a quick automation scripting tool that lets you create a script in real time as you record online actions.
- TruAPI is a scripting tool that lets you handle transactions, logging, and other more complex interactive tasks.
Any of these three systems can interface with the StormRunner Load system. The first two are Windows executables, while the third is a JavaScript application.
TruClient is probably the fastest option of the three for a beginner. Download and install the application, then run it. To create a new web-based test script, select TruClientWeb, then your browser of choice. From there, click the recording icon (red circle) and walk through the steps you want, to load test, open a site, navigate its structure, search, or whatever you’d like to test. Click the stop icon (blue square) to finish. Save the script.
Return to the StormRunner Load website. Click the Load Tests tab and select the upload icon to your script. You can configure the number of virtual users, the duration/ramp-up/locations of the users, and other advanced options using the icons on the left. Click Run Test when you’re ready to go. StormRunner handles the rest of the work, provisioning, running, and analyzing the test results. Once the test begins, graphs displaying your load test results will be generated in real time.
SOASTA CloudTest
Visit the SOASTA CloudTest page and click Free Trial, which supports up to 100 concurrent virtual users. Note, however, that this package (CloudTest Lite) is not a true CLT product, since the loads are all generated by your desktop. You’ll have to upgrade to the standard package to access true cloud-based loads. Sign up and wait for the email from CloudTest, which offers all the info you need to get started.
Download the linked software package and unzip it to a local hard drive. It’s designed for a virtualized environment, so you’ll need VMware Player installed as well. Launch the software from your virtual environment and activate it. (Licensing information is included in the setup email.)
Now you can create your test. A full user guide is available, but the basics are similar to StormRunner. Use the recording system to create your script (here called a Test Clip) as you walk through the steps you want the test to undertake. Once your Clip is ready, it can be run, monitored, and evaluated.
BlazeMeter
No time for installing software and writing scripts? Visit the BlazeMeter home page and click Try It Now. Sign up, and you’re on your way. (The free trial is valid for 14 days.)
Unlike the more enterprise-class tools discussed above, BlazeMeter requires no software to be downloaded. Just enter a URL on the next page and the number of concurrent users you’d like to test. (A maximum of 20 users is supported.) Note that this tool will test access to a single web page at a time and doesn’t support scripted tests. (More advanced options, such as setting multiple testing regions or testing multiple steps in a transaction, aren’t included in the free demo.)
After you tell the system to start running, that’s really it. BlazeMeter immediately gets to work and provides results in real time. Click through the tabs on the results page to dig deeper into the various categories of analysis.
Image credit: Flickr
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
