Etsy is known in tech circles for its great culture, visualization and monitoring tools, as well as strong DevOps practices. A fantastic example of this is the painless and friction-free way that its DevOps teams deploy code. Their continuous delivery pipeline allows anyone—developers, infosec, IT operations—to deploy, and their commitment to automation has made the process so safe that new engineers push to production their very first day on the job.
What's most remarkable, however, is the velocity as which Etsy's engineers push out new code. With good test and deployment automation in its continuous delivery pipeline, Etsy is able to deploy code more than 80 times per day, safely, easily, and painlessly. You can too, if you follow smart DevOps practices. But first, let's look at how Etsy does it.
The Etsy model
At Etsy, the whole process is automated, making it fast, reliable, repeatable, and safe. It begins with developers running 4,500 unit tests on their own workstations, a step that takes less than a minute to complete. Next, they check the code into trunk, which triggers over 7,000 automated tests on Etsy's continuous integration servers. These tests are distributed to several servers, and run in parallel, so they take no longer than 11 minutes to complete.
Next up are smoke tests and functional tests, which run test cases and end-to-end tests in a staging environment. From there, an engineer simply pushes a button to get the code to QA, and pushes another button to send the code to production.
But it’s not just Etsy that is seeing results. I have studied these practices in more than 20,000 teams for the past few years (with my colleagues Gene Kim and Jez Humble, together with the team at Puppet), and the gains have been astounding. High-performing teams deliver software 200 times more often, with 2,500 times faster lead times when compared to lower-performing teams.
Even better, this throughput comes without sacrifices to stability: High performers report 24 times faster recovery from failures and three times lower change fail rates. These statistics are compelling, but how can your organization achieve this kind of IT performance?
3 technical practices that make it happen
Three types of practices contribute to superior throughput and stability: technical, process, and cultural. While all three areas are essential for a true technology transformation and superior IT performance, I’ll focus on a few interesting technical practices in this article: test automation, deployment automation, and test data management.
Test automation
Test automation is much more than just not running tests manually; it’s the practice of running automated tests continually, throughout the development process. Research shows that automated tests are an important piece of continuous delivery, and that they drive IT performance. You can see the role that test automation played in Etsy’s deployment pipeline, and how it enabled them to deploy code quickly and easily.
Deployment automation
Deployment automation is essential to continuous delivery because it removes (or reduces) manual intervention in the deployment process, thereby speeding up delivery of software, and reducing errors that occur with manual intervention.
The importance of test and deployment automation was clear at Autotrader.com. The company went from releasing web services quarterly to almost weekly. This amazing move allowed them to delight their customers as they responded to market conditions, but it also meant that they needed to test their software across an increasingly complex environment — one that now must address several platforms and devices.
Using a DevOps approach to unify teams, they worked to significantly increase their test, build, and deployment capabilities. These efforts have paid off. Autotrader reports that it:
- Reduced test, build, and deploy times from weeks to just minutes.
- Decreased software defects by 25%
Test data management
Good test data management (TDM) requires having adequate data to run your testing suite, and having appropriate and adequate data on demand. Research has found that this is an important part of continuous delivery. It also plays a part in predicting IT performance. Test data requires careful maintenance, and is becoming an increasingly important part of continuous delivery.
Nordstrom tackled TDM after its analysis showed that data was creating bottlenecks in its software development and delivery processes. They dedicated a team to solve the problem, and focused on test data masking and subsetting, test data automation, and integration with tooling. Its efforts paid off, with the teams reporting the following benefits:
- Faster time to market.
- Improved quality and customer experience.
- Significant productivity improvements and savings.
- Improved testing efficiencies.
A word of caution here: Your teams should minimize their reliance on test data to run automated test suites whenever possible. The reason: Creating and managing data to be used in your test suites can be a significant overhead cost (e.g., don’t use a full copy of the production database.) Furthermore, many tests can be done without using data at all (e.g., many unit tests should be able to be run without external test data). Some teams may also run into problems when data is not masked or hashed properly. Good TDM includes right-sizing TDM.
Automation and testing are key
The experiences of Esty, Autotrader.com, and Nordstrom demonstrate how the technical practices of test automation, deployment automation, and test data management can have a great impact on IT performance for development teams, and also deliver quality features and functions that matter to customers.
But that's only part of the story.
In addition to several other important technical practices, such as continuous integration, trunk-based development, and the use of version control for all production artifacts, these three technical practices must go hand-in-hand with other processes.
You'll need to use visualizations to make business decisions and work in process limits. And you should establish cultural practices, such as blameless postmortems, and foster a culture that values learning, to truly bring about transformational change in your organization.
Image source: Flickr
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
