I was working as an agile coach at a large financial company when another agile coach came to me and proposed forming teams that could serve as "impediment-removal engines" (IREs). He had noticed that many of our teams were struggling with the same problems.
Many of those impediments were systemic and outside the scope of influence of the teams. So he proposed creating teams made up of people who had more organizational influence than did the individual project teams and could resolve those systemic issues.
It worked, eventually, but it turned out that one IRE team wasn't enough. Here's how the strategy evolved—and the benefits we achieved.
I wasn't convinced, at first
My initial reaction was, "Good idea; I would love help, but I am really busy." I went back to my teams to focus on helping them improve. While we were able to make progress on many issues, some of the teams had gotten to the point where their biggest impediment was a roadblock they couldn't remove themselves.
When you optimize your software development process, you need to prioritize addressing the impediment that has the biggest effect on your ability to get work done.
In some cases, working on and resolving a less acute blocker will make your process defective by making your worst blocker more pronounced. In our case, improving our process was getting harder.
Turns out Michael, the other agile coach, wouldn't take no for an answer. He returned about a week later and asked me again. When I asked him which issue would be the first addressed, he replied that many of the teams were being held back by a lack of testing environments. This was a huge issue for my teams also, and I agreed that we needed to do something about it.
Creating the strategy
Michael mapped out the idea of creating a set of teams that would be charged with taking on the impediments that were beyond the scope of influence of the individual teams. The teams would be made up of people higher up in the organizational hierarchy.
Each team would have the goal of either resolving specific issues or escalating to a team with a wider sphere of influence. The team would organize itself just as the project teams that worked under them did and use the same agile process those teams used.
Level 1
At the first level, the team would be made up of a director and the managers who reported to him or her. This team would vet the blockers their teams sent to it. The project teams had to ask the IRE team to take on specific blockers. The one thing the project teams had to do was show the IRE team what they had done to resolve the issue themselves.
Part of making empowered teams is to make sure they have what they need to resolve impediments themselves. It would be ideal if teams could resolve all of their issues, but in many organizations that isn't possible for a variety of reasons.
Rather than telling a project team to just keep trying to resolve an impediment it doesn't have the ability to resolve, creating a clear escalation path helps give the team a way to start the work on an organizational impediment. At the same time, this team can also hand that issue to the people who have the appropriate level of influence within the organization to actually make the changes needed.
Level 2
The second level of the IRE team was made up of directors and the vice president they worked for. This team would consider only issues that the first IRE team couldn't resolve. This means that the second team should see far fewer issues than the first, problems that required people with even larger spheres of influence to resolve.
In many organizations, the more entrenched systemic blockers are, the more you need people higher up the hierarchy to resolve them. Organizations that can't or won't escalate systemic issues to those that can resolve them will limit or halt their teams' ability to get work done and create successful software.
Impediments are things that prevent getting work done. Allowing work-blocking issues to fester only hurts morale and motivation.
Level 3
In our organization, we also needed a third-level IRE team that included the vice presidents and the senior vice presidents to whom they reported. This team's mission was to take on the organizational impediments that required a company-wide sphere of influence to resolve.
This team took on very few issues, but resolving issues raised to Level 3 would unblock work across the entire organization and make noticeable productivity gains. Depending on the depth of your organizational hierarchy, you might need more than three levels of IRE teams.
Tackling the issues
While I was already fairly busy coaching my teams, I felt this opportunity to get some long-term blockers resolved would be such a boon to productivity that I shouldn't pass it up. To help Michael start putting the IRE teams in place, he asked me to help the director leading the portfolio I was coaching set up an IRE team.
I would help him set up his team and coach the team members through the first few critical phases. The good news was that this team was a group of people that already knew one another and had worked together before, so we were able to get up and running quickly.
Addressing the lack of test environments
The first issue we decided to take on was the lack of test environments. This was a long-term issue that many teams at this company suffered from.
It is also an impediment at many organizations. Best practices for system management before the cloud was to install a server once and keep re-purposing it for other work. Since the cloud, virtualization, containers, and infrastructure as code have created best practices, this has shifted to purpose-built servers that live only as long as needed. If it is a test server, then the server needs to live only as long as it takes to test the software.
But some organizations haven't graduated out of the maintained-server mindset and still have a high cost of standing up a server. Sometimes there is even a high cost of changing the software installed on a server. These practices can lead to teams juggling servers for development and test.
It can also lead to standoffs and refusals to give up resources—i.e., servers—because you don't know whether you will get them back when you need them. When server access is a scarce resource, people will hoard servers.
One of the teams in the portfolio had already tried to take on this issue. It had gone as far as to show how much time and money was wasted in the shell game of servers over a six-month period. What it could directly account for was equal to about 1.5 full-time equivalent (FTE) of lost time and money. On a seven-person team, 1.5 people not working for a six-month period is a lot.
While everyone agreed that this isn't a good situation, the team didn't have the ability to get more environments or to get system administrators to make it easier to switch the applications under test. All of the team's attempts to change were blocked by people outside their influence.
How the issue was resolved
The IRE team took the data the project team had created showing the magnitude of the problem. It spoke to its counterparts in the groups that create and maintain servers and in the groups that deploy and configure the software.
The team's counterparts agreed that this was a problem—and then explained that there wasn't anything they could do. They lived and died by SLAs (service-level agreements require performing specific operations in specific time periods and meeting specific performance metrics tied to environments). They were within their SLA as it stood.
In their world, production systems had a very tight and strict SLA and all lower environments had less restrictive guarantees. Dev and test servers had a "best-effort" SLA, so there were no agreements to deploy quickly to them. In the worse cases there were no agreements to even fix a broken dev or test server; those could languish in such a state indefinitely.
To be fair, most dev and test change requests and maintenance tickets were addressed in one or two weeks. This time frame may seem reasonable, but a team with a two-week sprint and no server won't be able to complete its work on time.
This issue needed to be escalated to the higher-level IRE teams because it involved budgets and services that spanned portfolios and business units. Also, the teams had started to see faster response times for their change requests. This was because the IRE team had worked with the operations group to tighten up the SLAs around lower environment change requests so that the teams would have less idle time waiting for updates.
The organization had started down the path of allowing teams to experiment with infrastructure as code so that teams could spin up new requirements on demand, making it easier for them to have the environments they needed. This will be a multi-year effort, since they are just getting into designing their systems to take advantage of newer virtual and cloud-based technologies.
An unexpected benefit
The organization in this case was a financial institution that had been founded 80 years earlier. It has been using information technology since the 1970s, and at times, it felt as if a few of its systems were still those original mainframe applications created in the very early days of corporate computing.
Many of the vice presidents, directors, and managers I worked with were talented, experienced technologists who understood the value of agile and DevOps. One drawback was that they had never worked as a team member on an agile team.
While they could understand the team practices, they had a hard time feeling the pain their agile teams felt. Getting them on an IRE team where they used the same agile methodology as their teams really helped them feel how the process worked.
They started to complain about the same things their teams complained about. They started to experience the same issues, process difficulties, and delivery problems they used to beat their teams up for. This gave them a better appreciation for what their teams accomplished and a deeper level of empathy for their teams.
In the end, the IRE approach helped the organization move away from viewing agile and DevOps transformation as an attempt to "fix the software teams." Instead, the organizational mindset morphed into one where people at multiple levels bought into agile and understood it a gut level.
Want to know more? During my STARWEST Virtual+ conference session, "How to Be an Impediment-Removal Engine," I'll offer more tips about how to create your own IRE teams. The conference runs October 5-8, 2020.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
