Six years ago, J.P. Morgan Asset Management's equities trading desk was growing dissatisfied with the slow pace of software development and implementation. Our equities technology team at the time still followed a traditional, quarterly software-release cycle, and that release rate wasn't keeping up with the more sophisticated demands of the market.
By 2017, however, we had moved to a more-than-daily release cycle.
J.P. Morgan Asset Management manages assets on behalf of our clients, and the equities trading desk executes over $1 trillion a year; this represents 1% of the world's equity trades. For every 0.01% improvement we can make to the execution of those trades, we can save our clients $100 million.
Because we operate in a fast-changing marketplace with continually evolving regulation, it is important to be able to deliver technology improvements and enhancements quickly and safely.
Why the traditional approach wasn't working
Back in 2012, our technology team followed a traditional release cycle, with different teams managing different aspects of the product lifecycle. It was a rigid plan, build, and operate model with formal handoffs between development and operations teams and infrequent collaboration. Releases were challenging to execute and done generally once every three months.
Each release was a big one, aggregating lots of software changes. This meant that each release carried a lot of risk and could lead to expensive rollbacks if problems cropped up.
[ See more: DevOps Enterprise Summit 2018 Las Vegas ]
What's more, as the pace of business accelerated, our business users grew frustrated by having to wait for several weeks, or perhaps up to a quarter, for the changes they needed to be made so that they could be more effective. They wanted to be able to react to changes in the market more quickly and deliver benefits to clients faster. They struggled to do that under our quarterly release schedule.
These factors were increasing our cost of delivery while dulling the positive impact we could make on the business. We knew that the traditional answer of just adding more developers wouldn't solve the problem. We needed to make a different kind of change.
This was the start of our DevOps journey. It's been a long, iterative process, but we're more responsive to the business, we're more collaborative than ever, and we've gotten there in a highly regulated environment that has zero tolerance for failure.
Our path to success led us to tackle four major challenges. Identifying and overcoming these challenges allowed us to speed up delivery of software, increase the quality of our work, and minimize risk along the way. Here's how we did it.
1. Breaking down the wall between dev and ops
One of the biggest initial barriers to speeding up our release process was the segregation between developers and operations.
These teams were siloed, and the most interaction they had was when they were firefighting issues—not the best way to build collaboration between the teams. We needed to interact more frequently and be proactive to truly solve fundamental issues that were causing recurring problems in the delivery lifecycle.
We knew that if we could establish a more collaborative environment between the teams, it would be easier to move toward lean and agile methods of software delivery. So one of the very first steps we took was to get these teams physically located together.
We knew that seating the developer and operations teams next to one another would help build relationships and a mutual understanding. It also made it easier to establish a swarm mentality—where everyone pitched in to solve problems as they arose.
But we didn't stop there. The goal was to establish a "one team" culture that had us running events together, socializing together, doing lunch together, and so on. It really helped us to fully establish a mutual understanding of the kinds of concerns and problems that each side was facing.
With that kind of connection occurring, we also ramped up the level of communication among team members. We held daily Scrum meetings to ensure that everyone was operating in the same physical space and were all on the same page.
Collaboration and efficiency improvements allowed us to go from quarterly releases to monthly releases. By 2015, we reached the milestone of 100 annual releases.
2. Replacing tools that couldn't keep up
Once we broke down the people barriers, it became apparent that we used different tools across our formerly two separate teams. That was the next blocker in our one-team strategy. In the old model, developers had their own tool set, and operations had another. Each produced its own set of metrics, which meant there was no single source of truth.
For example, developers used one set of tools to release into test environments, and a separate, centralized deployments team used another set of tools to release into production. At that point, the deployment package would enter a bit of a black hole and was slowing down the process.
We decided to move to a new set of automated deployment tools, which all of our new, combined team used. This also allowed us to bring deployment back into our delivery pipeline so anyone in the team could manage deployments end-to-end while ensuring that the process had all the required controls and was fully auditable.
Much of the change processes had built up from regulatory demands, but over time people had lost sight of the initial requirements and the processes had become bureaucratic and cumbersome. We had to step back and re-evaluate the initial requirements.
Working with our technology risk partners, we started to draw up new automated processes that would meet the requirements for control while meeting our requirements for streamlining.
The more systematic we could make the process, the happier our tech risk partners were, since it delivered the repeatability and tractability they needed. And through that we ended up agreeing on a new set of tools that were highly streamlined and secure.
3. Addressing technical debt that bogged down processes
Prior to starting our DevOps journey, our release process was time-consuming and high-risk. Releases were rarely issue-free, so they would consume additional time in support and emergency patching.
A lot of this came down to technical debt. Automated tests weren't at the level they needed to be, and so they relied heavily on manual testing that took a long time. Our goal was to get to a stage where automated testing would be sufficient for business approval of a release. That way, we would reduce the cost of a release, so we could deliver in smaller increments that would carry less risk.
Given the low level of tolerance for failure we had as a financial institution, we needed a way to help us transition to automated tests and minimize the impact any issues could create.
While we had rigorous testing procedures and expensive user acceptance testing (UAT) environments, issues could still make it to production. One of the methods we used to reduce issues further was to follow a two-phase release process.
For every release, we first deployed an early-access version to a restricted set of users, rather than deploy to everyone at once. If issues were introduced or functionality wasn't to users' liking, users had a one-click functionality available to them to roll back to the current version.
The issue would be logged and, because our releases were much smaller with fewer changes, we could find the root cause of the problem fairly quickly and resolve the problem while the trader continued to work as normal.
The combination of automated testing, smaller releases, and lower blast radius for issues accelerated our rate of change and reduced the frequency and severity of issues. It became a virtuous cycle of improvement.
4. Deepening the business and technology relationship
One of the biggest elements that precipitated our DevOps changes was obviously the business need for faster, higher-quality updates to the trading platform. Given our prior track record, the business was concerned that delivering more frequently would mean more issues. Not only did we need to improve our processes, but we needed to gain their trust.
That started by being fully transparent with our business partners, acknowledging where the problems lay, detailing our strategy to put it right, and making sure they were involved in the process.
For many it was counterintuitive that moving faster could actually be safer, so we had to spend a lot of our effort demonstrating the power of modern development techniques and repeatedly proving the benefits.
We doubled down on the idea of a very customer-centric environment. We focused on serving our users as customers and took the time to meet with individual users in person to explain changes coming up and walk them through those changes. Not only were we co-locating developers and operations, but we started getting our tech team in closer physical proximity with our user base as well. Some users needed more handholding than others, and we recognized that.
Doing those things showed our users that we really cared and that we had their interests at heart. Rather than just being there when there were issues, we were also highly visible when releases went well to reinforce the message that our new mode of delivery was really working.
The numbers speak volumes
By overcoming these four challenges through DevOps and lean Kata, along with the use of improvement themes, we've experienced tremendous gains. Our release frequency increased from quarterly at the outset to 100 releases in 2015, 200 in 2016, and over 400 in 2017. More importantly, the frequency and severity of incidents arising around releases have been reduced significantly.
As a result, we've gone from arm's-length relationships with our business users to having our users be our strongest advocates. For organizations like ours that are facing a similar long-haul journey, we recommend breaking down problems into small steps and being prepared for a long journey.
Here's why that's important. At the outset, it was difficult to get developers and operations staff that had been in their roles for a long time to believe that we could actually change things to this extent.
By mapping what "awesome" looked like and breaking out the smallest challenges that stood in the way of that, we came up with a massive list of things we wanted to change. From there, we could say, "What are three things on this list that we can actually solve in the next few weeks?"
By continuing to move forward on these small problems that were contributing to the bigger challenges, we started to make progress. We'd tackle three issues, then another three. Before we knew it, a few months had passed and we were completing a long list of achievements that further gained buy-in from more people on the team. That's what really helped us build momentum.
Kenneth McLeish, head of equities technology at J.P. Morgan Asset Management, co-authored this article.
This is just a quick summary of our multi-year journey. To learn more about how Kenneth and I accelerated software development and what we're doing next, drop into our session, "How we partnered with our business to go from quarterly to daily releases in a highly regulated environment," at DevOps Enterprise Summit Las Vegas. The conference runs from October 22 to October 24, 2018. TechBeacon readers can get $300 off registration using code DEVOPS300.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
