Many people talk about the CAMS model of DevOps: culture, automation, measurement, and sharing. In practice, however, many teams and organizations focus on the technology and cultural pieces of DevOps transformations and neglect measurement and sharing.
Measurement is the secret ingredient to any DevOps transformation, because it’s so influential when done right. It is a key aspect of any technology transformation: With metrics in hand, you will be prepared to leverage the insights you gain from your experiments, capitalize on your successes, and change course quickly when you don’t see a benefit in something you have built.
Small improvements are the key to large gains, and with metrics in hand, you will be prepared to take advantage of these small improvements.
[ View Nicole Foresgren's presentation at DevOps Enterprise Summit London below ]
And because they capture information that is clearly defined, measured, reported, and shared with others, metrics help you communicate clearly across the business. This touches on the sharing aspect of CAMS. And because these metrics will shape incentives, which in turn shape behavior, metrics also shape your culture.
Here's how to understand metrics and measurement to better drive your DevOps.
What measures are most important?
I've spent quite a bit of time applying data science to DevOps, and I am often asked, "What is the one metric that matters for DevOps?" Unfortunately, there is no easy answer, and there isn’t really one metric that matters in all cases. In fact, any time you capture only a single metric, that is the metric that will be gamed. I’ll address this more in just a minute.
The most important thing when you capture metrics, in DevOps or any area, is to follow an easy framework that outlines two key types of measures: inputs and outcomes. (DevOps Research and Assessment, or DORA, offers Metrics Workshops that address this. We’ll be offering one workshop the day after DevOps Enterprise Summit London, and another workshop the day before the upcoming DevOpsDays Minneapolis. I’ve offered this workshop in various forms for years, and the framework it presents is a powerful, easy-to-understand way to think about measurement.)
Your outcome is the goal. It might be customer satisfaction (whether your customer is external or internal), improved response time, or improved test suite coverage. Depending on where you sit in the organization, your outcome may be broader or narrower.
Your inputs are things that you and your team believe can affect your outcome and that you can control. This last piece is important, because if you want to take an iterative, experimental approach to improving your outcome, you must be able to affect the inputs you identify. (Yes, there may be other inputs that you do not have control over; those are for another discussion.) But if, for example, your outcome is improved customer satisfaction, your inputs might include improving response time to tickets, meeting SLAs, or improving the user interface.
IT performance as a metric
For an applied DevOps example and a more direct answer to the question, “Is there one metric that matters?” let’s go with this: If I had to pick my favorite metric in technology transformations, it would be IT performance.
IT performance is a family of metrics that I have found in my research with DORA and Puppet to absolutely matter in DevOps and technology transformations. This one set of metrics is a strong indicator of high-performing teams and is predictive of organizational performance, as measured by profitability, productivity, and market share. These metrics reflect throughput and stability, and by capturing two balancing aspects, they are difficult to game.
IT performance consists of four measures: deployment frequency, lead time for changes (code commit to code deploy), mean time to restore (MTTR), and the change failure rate. The first two are throughput measures; the last two are stability measures. Our research has shown that you can combine these into a single measure of IT performance. I'll use this as the output measure.
Now that I have identified the outcome measure, I want to select a few input measures that I believe will influence IT performance. The State of DevOps Report 2017 identifies over 20 key capabilities that drive improvements in IT performance, and these fall along familiar categories: technical (version control, test automation, deployment automation, trunk-based development), process (work-in-progress limits, visual management, visualization of the value stream), and cultural (team culture following the Westrum typology, a learning culture, job satisfaction). You can select from one of these or pick your own and experiment.
[ DevOps Enterprise Summit: Experts share lessons learned ]
The final step: Match the data to your measures
Once you've selected the input and output measures, the final step is to identify what measures to use for your outcomes and inputs and where those measures will come from. Some measures may be easy to find in your systems, such as the lead time or change fail rate, while other measures may be difficult to find or not exist yet.
Don’t be discouraged if this is the case! Most organizations we work with are in this situation and are still adding metrics and monitoring capabilities to their systems and organizations in an effort to be more data-driven.
Try it yourself
If you haven’t taken this approach to data and measurement yet, I encourage you to try it. Sit down with your team and identify your goals for the month or quarter. Use those goals to identify an outcome, and then identify the inputs that you think will drive that outcome.
Next, identify metrics that you think you have available to proxy for that outcome and those inputs. This will be tricky the first time or two that you do it, but before long it will be easy to do and will make identifying assumptions or gaps in thinking an easy task.
An added benefit to this exercise is that your team will have a clear explanation of what it is working on, why, and how it is measuring it. And you can use that to communicate to others in your organization.
Want to know more about using metrics to drive DevOps? View my recent presentation, "The Key to High Performance: What the Data Says," from DevOps Enterprise Summit London.
Or post your questions and comments below.
Image: Flickr
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
