Earlier this week, major news sources focused on the war between Amazon and Walmart leading up to Amazon Prime Day, a one-day sale for Amazon's premium—Prime— customers that was touted as Black Friday in July. On Wednesday, the day of the event itself, it seemed to me that every tweet, blog, and news network was covering Amazon's lackluster deals. The Twittersphere was having a field day with #PrimeDayFail. By midday Wednesday, a bazillion barrels of virtual ink had been used in creating the publicity, most of it negative, around Amazon's Prime Day failure.
Despite the bad press, I'm not 100% convinced that Amazon Prime Day was a "failure." From a performance perspective, Amazon learned a lot this week about its website's capacity to handle dramatically increased traffic. What if this was part of Amazon's plan all along?
Think about it
This was a game changer in several important ways.
First, Amazon's own press release touted the event as "bigger than Black Friday":
"Beginning Wednesday, July 15, just after midnight PT, Prime members can shop thousands of exclusive lightning deals with new deals added as often as every ten minutes throughout the day. With more deals than Black Friday, members will enjoy thousands of deals from sought-after devices and tools to popular toys and pet supplies. Anyone can join Prime, to start a free 30-day trial and participate in Prime Day visit amazon.com/primeday."
Second, the publicity not only drove high interest among Amazon Prime customers but also brought Walmart to the game with their own sale lasting 90 days. Walmart's efforts included slams against Amazon for offering deals only to its premium customers, who pay $99 per year for free shipping. In May, Walmart confusingly announced its own "Prime-like" program for $50 a year.
Who won this week?
For me, the bottom line—beneath all the publicity, disappointing sale items, and news analysis around Amazon and Walmart—is that Amazon achieved a remarkable baseline for performance indicators. If you're a performance guy like me, it doesn't matter whether your massively advertised sales day generated thousands of Twitter sound bites about cat pillows and 55-gallon drums of lubricant and other useless products people never expected to see featured on your 20th anniversary sale.
What matters is how many people showed up. How did the website perform when confronted by millions of eager buyers, particularly those who like your service enough to pay extra for it?
Performance issues didn't spoil the day
Some Amazon customers reported site outages as they tried to find deals or conduct transactions. It isn't clear if there was a major performance issue, although I have two colleagues who experienced problems.
The image on the left is from a cell phone, the one at right from a laptop. In any case, there were few complaints about Amazon Prime's website performance—there were far more complaints about what customers found when they got there.
Amazon is a formidable company, at the top of their game, celebrating 20 years of success. Plus, they thrilled shareholders with the highest stock price ever. In my view, what we saw at the end of the day, literally the end of July 15th, was nothing less than a brilliant marketing stunt.
We saw them driving huge web volume. Did they do this to the point that they had certain performance issues with their site? Maybe. Michael Zuber, VP of go-to-market sales for HP Software Americas, made this comment to me regarding Amazon Prime Day: "Another example of legendary marketing, creativity, and end-user demand generation ending with some users experiencing frustration."
Then again, maybe that's not such a bad problem to have, as long as it's part of a test you're conducting.
The facts about Prime Day
Let's look at what we know for certain about this event.
- They drove huge production traffic and volume.
- The social media coverage was amazing, with reams upon reams of Tweets, Facebook posts, LinkedIn comments and articles, and Google+ activity.
[Note: I wonder if Amazon (and their competitors) were able to get results on the impact to brand, revenue, competitors, and customers from these events on July 15, 2015; and measure the results of certain orchestrated changes in applications and/or infrastructure, then see the impact it had in the market from several of these KPI's, including social media activity and the tailing influences?] - They also had much of the industry talking about it. The top practitioners and thought leadership in several circles I am in were discussing the Amazon Prime event and other ancillary impacts the morning of the 15th. I spoke to a number of my peers as well, some reaching out and sharing their experience with me saying, "You need to write this story." I reached out to Burt Klein, senior advisor at CA Technologies. I contacted James Pulley, Keith Lyon, Tammy Everts, and Scott Moore, a renowned performance guru. All of us were talking about the event and the impact in the market and to end users.
Speculation about Prime Day: Performance testing in production?
Now, here's some speculation, both by me and by several others who know their stuff when it comes to performance.
In my opinion, Amazon used Prime Day as performance testing in production.
Was this a dry run for things like the upcoming Black Friday, Cyber Monday, and holiday shopping season? We're getting close to those dates—the top 100 retailers in the world have a code freeze and reduce their software releases around the 15th of September. That's less than two months away. Maybe Amazon wanted to create a marketing frenzy, drive all this traffic in actual production, and possibly pushing out different code bases at intervals throughout the day to see how that code might perform.
Maybe they were also testing various software releases and infrastructure configurations on the back end, to see how they would perform under different load scenarios. This is what we refer to as A-B testing: observing what happens when you make a variety and/or combination of changes and the resulting outcome of these in production and to your users. My friend and colleague James Pulley, performance evangelist at PerfBytes, suggested as much. Regarding Walmart and Amazon, he said, "Two large providers with a strong history of being ahead of the curve on performance: Can we consider today's performance problems to be the 'canary in the coal mine' for coming issues this fall?"
And there was this from Burt Klein, senior advisor at CA Technologies: "We all can plan for a spike in activity, but it is still guess work as to the magnitude of that spike. What do we do when our actual user request is five times the peak projection? The new challenge is to identify and implement systems that are resilient enough to allow an organization to throttle the thru-put while delivering a kind unavailable message rather than bring the full system to its knees."
And everyone's watching
There's one more thing we can say for certain. Amazon absolutely disrupted the market and their competitors. Other companies like Walmart, Wayfair, etc., are paying close attention. There are hundreds of news articles about the event, before, during, and after. A Harvard Business Review article published a few days before Prime Day states the case nicely:
"Remember the vicious discounting wars that occurred during the Internet bubble era? Amazon emerged a clear victor in that virtual retail grab. It's time for brick and mortar retailers to be on the lookout."
Amazon isn't just going after brick and mortar companies, either. With Amazon Prime, you can stream videos on demand, for example, which for $99 a year is cheaper than Netflix at $108.
Wow. They've created their own holiday day. Prime Day.
Yes, it's possible that I'm giving Amazon too much credit here. Maybe this was simply a miscalculation, a yard sale pure and simple that was billed as something considerably more exciting. But again, by targeting their Prime customers with this sale, they were risking less, given the membership fees these customers have already ante'd up. Most customers will continue using Amazon because of what Prime offers them and that alone will keep them loyal, even if many were massively annoyed on Wednesday.
None of this matters, though, for Amazon's technology team (some have called them "one of the best"), because the results they have created will keep both their big data analytics team and performance engineers busy. The metrics they were able to obtain on Prime Day provides a field day for analysis. They're looking at mountains of fresh data from their own consumer base (and their competitors) and studying all that behavior and those actions based on events happening in the real world. The performance engineers at Amazon will have a number of interesting weeks ahead of them.
For the rest of us, it'll be interesting to see how the events of July 15, 2015 (will it still be called Prime Day?) and the results generated will be leveraged by Amazon. How will they use their performance engineers to optimize their applications and infrastructure for the experience of their end users? I look forward to seeing how Amazon continues its tradition and reputation in the market from the last 20 years, while delivering value to shareholders.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
