It’s almost impossible for engineering teams to pinpoint their cycle times, bottlenecks, and continuous improvement efforts without measuring and identifying their successes and failures. You can achieve continuous improvement only by enabling a learning culture and removing bottlenecks while identifying areas for improvement.
Metrics act as guidance for engineers, giving them an idea of what they need to strive for and improve upon. That’s why you need an engineering metrics program. They provide a quantifiable measure of key performance indicators (KPIs) you can use to measure growth and success over time. Some key metrics examples are lead time, time to code review, time to merge, and commit-to-deploy time.
Recently, I discussed metrics programs and continuous improvement with some of the top minds in software engineering: Charity Majors, the founder and CTO of Honeycomb.io; Dana Lawson, vice president of engineering at GitHub; and Kathryn Koehler, director of productivity engineering at Netflix. (You can watch the entire discussion here.)
They shared advice on how to create continuous improvement in engineering organizations, and discussed ways of creating and using an engineering metrics program with metrics from personal experiences and improvement frameworks. Here are the key takeaways.
The SPACE methodology: A good way to predict developer productivity
The SPACE Framework is an engineering metrics program that’s currently making waves in the industry. Developed by Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler, this framework centers on the idea that defining, measuring, and predicting developer activity can lead to safer and faster software delivery.
SPACE, which stands for satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow, enables you to understand your teams and how they're doing. While this methodology is still in its early days, Netflix, GitHub, and others are already implementing it within their dev teams.
Netflix's Koehler explained the key to success when adopting a metric system: “There's no unicorn metric for anybody. People who are at the undergraduate level of productivity measure activity and think, ‘Oh, lines of code, butts on seats, deployment today.’ That's what you can get your head around, and that's what productivity means. But the SPACE methodology is really around well-being and satisfaction.”
The key to the SPACE methodology is the understanding that productivity and satisfaction are intrinsically connected, and you can see this reflected in other methodologies as well.
The DevOps Research and Assessment model: The precursor to SPACE
The DevOps Research and Assessment (DORA) model, which preceded the SPACE Framework, lets you measure your software delivery performance and compare it to the rest of the industry. When developing the model, the DORA team identified four key metrics for performance: deployment frequency, mean lead time for changes, change failure rate, and time to recovery.
With any metrics program, the most important thing to remember is that each data point and metric will be unique to your organization.
“The metrics of DORA are a good foundation to start understanding your organization,” said GitHub's Lawson. “But as with any analytics, it's a trap just to look at activity. It's a trap just to go read these in a vacuum. It's not going to tell you the whole picture of what's going on. There are so many different variables, and the variables are going to be unique to the team that's actually working. Because you can have a company with the scale of GitHub or Netflix or Honeycomb, and you have all these little units, and your data engineers may have a different cycle time, and that's totally freaking fine.”
Should there be a fifth component of DORA, and, if so, what would it be? The resounding answer was yes, and it should focus on protecting your developers’ stress levels and their out-of-office hours.
Use data and metrics to help avoid burnout
Not only do engineering metrics programs such as SPACE and DORA give you a baseline understanding of how your organization is performing and where you can improve, but—most importantly—they give your dev teams a starting point for goals and objectives. Each metric will be unique to your software delivery lifecycle.
Because mental health should not be ignored, you can also use metrics to ensure that your developers are not being overworked. According to Honeycomb.io's Majors, “People feel they're responsible for the stuff that they've built because, by default, we care about what we've built.” But over time, pulling developers back to their work during their off hours can accumulate and lead to burnout, making metrics and data essential to the time spent on each task and prioritizing what matters most.
Successful implementation of a metrics program also means that you do not grade people based on the metrics, but just ensure they’re aware of them.
As Koehler said, “Unfortunately, what gets graded gets gamified, and so it's important to reinforce with people that this is a tool for you to know where you're investing and how you're improving. I am not looking at it to adjust your compensation.”
Is data at the core of your continuous improvement practices?
An engineering metrics program is important not only for your management team, but for your dev teams as well. Data helps management track performance and progress through the organization while also enabling engineers to focus on the tasks that matter and guiding them to the metrics they should be striving for.
The well-being of your engineers is of the utmost importance. A successful engineering metrics program takes into account the workload of your engineers, the mean time between interruptions, and more. So be mindful of the duties of your engineers and protect their time at work—and away from work.
Interested in learning more about continuous improvement and other dev-related topics? Join the 1,200-plus other engineering leaders in the Dev Interrupted Discord community.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
