Web services are quickly moving to cloud architecture, and DevOps and testing are following close behind. While cloud architectures are highly scalable, you'll have some challenges to overcome if you want to provide the maximum benefit to your business.
Many articles, books, and videos talk about why testing in the cloud is good, but they don't always talk about how to determine whether cloud-based testing will meet your needs.
Here are the benefits of using cloud-based testing, some of the challenges and considerations for testers thinking about making the transition, and a brief example of what successful cloud-based testing, development, and deployment looks like.
The promise of cloud-based testing
Companies benefit from using cloud-based testing services because someone else is hosting the hardware and delivering the software. That potentially saves money and development cycles that you would otherwise need to do this work in-house by creating your own testing lab. And unlike when you use your own hardware and software, cloud-based testing platforms can be scaled up or down to meet business demand.
Testers benefit when they use cloud-based tools to increase the scope of browsers and environments that are covered in UI or functional tests. Cloud services such as Browserstack, SauceLabs, Firebase Test Lab, and AWS Device Cloud give test teams the ability to test against multiple environments, browsers, and mobile operating systems without having to maintain the necessary hardware.
Cloud-based testing suites help release engineering and DevOps teams spin up test environments with different configurations and data, or create fully automated release processes, when combined with tools such as Heroku or Docker. Using these tools in the cloud creates opportunities for collaboration among your entire product development team.
Challenges in the cloud
So, if the cloud is the next best thing, then why isn’t everyone on board? Getting from the ground to the cloud can be a little tricky, but if you know the challenges associated with the transition, you can more clearly articulate your needs and address concerns.
What happens when services combine?
One concern with using cloud-based testing services is that many companies use several services to meet different needs. The more moving parts there are to a system, the more access challenges you’ll face when bringing the pieces together. Imagine, for example, writing and running browser tests. Say that the tester is developing and running those tests in a local environment with a relatively speedy Internet connection, and the tests pass.
Enter the cloud. Now you want the tests to run with the CI system, which is hooked into a cloud-based browser test service. For all of their convenience, cloud-based browser test services can introduce more complexity, since they may be accessed through a company VPN and then rely on the speed and reliability of the cloud browser test service. One possible consequence of this might be flaky tests that timeout due to a lag that was not introduced in the local office environment.
Other challenges include what happens when major cloud services go down. For example, during the summer of 2013, Amazon Web Services went down for less than 30 minutes. But the result was that hundreds of companies could not provide services for their customers during that time, and they couldn't test properties that required access at AWS. This kind of third-party dependency can be a major challenge to using cloud services.
Provisioning and access
Cloud services often come with licensing and pricing that may make it difficult to provision access for all who need it.
Tip: When planning to use third-party cloud services, first determine who needs access to what services, and how often.
Having flexible configuration for different kinds of users or different aspects of the product for testing is a goal of many cloud-based testing systems. Sometimes this goal can be achieved through virtualization and containers. The challenge of using this technology is that many testers know what they want but may not know where to find information or how to have that critical conversation with release engineers or DevOps.
Test environments: Configuration in the cloud
Seaver's Law of software engineering states that "Persistence means always having to say you're sorry." Persistence defeats the purpose of having a clean test environment. It can confound tests and cause false positives. Nonetheless, test data is often necessary.
One of the challenges to testing in the cloud lies in finding ways to create a test environment that contains all of the necessary app configurations and test data. This process can be time-consuming for testers and developers alike.
Ease of use: An example
The CumulusCI project is one open-source system that overcomes the challenges of testing in the cloud. It is a Python framework that integrates with Salesforce Developer Experience (SFDX) and allows users to create configurations and data for each deployment of the application by way of the command line, making setup and configuration easy for developers and testers. This further integrates with MetaCI, a framework that uses Heroku to support the needs of Salesforce organizations and burst capacity architectures.
The upper limits of deployments are high enough that testers and developers can comfortably create multiple instances of different organizations needed to do browser testing, manual testing, and even feature development. Because the organizations are cloud-based and contained, destructive testing can be done without fear, even on feature branches. This custom CI system is open source, making it a great resource for companies looking to better understand another cloud CI solution or what might be necessary to build their own.
How to get ready for cloud testing
Are you ready to try testing in the cloud? Here are the first steps you can take to bring this technology to your team:
- Investigate what cloud services your company is already using. There may be ways to get more from services you already have, and you'll want to make sure that new services you add will work well with your current architecture.
- Determine who needs access to what services. Often, this means making sure that people have the access they need to do their jobs, which may mean extending service plans with growth or using a service that scales with business needs and charges accordingly.
- Determine who owns access to services. Sometimes, a single DevOps engineer has responsibility for owning knowledge about cloud environments and configurations. That’s not a good idea, because that person becomes a single point of failure for development and testing.
- Use trials of cloud services to learn about the pros and cons of each. Most companies offer free trials of some sort; for example, Browserstack, SauceLabs, and Micro Focus offer free trials that allow you and your teams to experiment with how cloud testing tools will work in your toolchain.
Follow these steps and you'll be well on your way to increasing your productivity in both testing and development.
What are your tips for overcoming the challenges of testing in the cloud? Post your comments below for discussion.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
