This article is part of an ongoing series of Performance Retrospectives that assess real-world application performance issues in the news, analyze what might have happened, and offer up best practices that just might help you avoid similar problems.
At 7:57 AM on Monday, May 4, the NETGEAR VueZone support forums erupted with complaints from customers who were unable to use their cloud-based home security cameras to remotely monitor activity in their homes. VueZone cameras, part of the Internet of Things (IoT) in the home, use mobile apps and a web-based service from NETGEAR to let consumers remotely monitor web-connected security cameras at home.
What happened
Over a 10-day period from May 4 to May 14, 2015, NETGEAR VueZone users reported numerous issues, but the biggest complaint was the inability to remotely communicate with home security cameras using the VueZone service.
Why it happened
Based on user comments and an Amazon blog post, it appears that the root cause of this incident was a change in an Amazon API. One user elaborated on the issue in a NETGEAR VueZone support forum, reporting "an error from the Amazon cloud." As with many of the connected, smart devices in the home that make up the IoT, these cameras depend on services hosted in the cloud and APIs to connect them. In a technical blog from Amazon at about the same time as this incident was resolved, Milind Gokarn posted "Serving Private Content Through Amazon CloudFront Using Signed Cookies."
The business impact
VueZone cameras sell for about $200.00 per unit and users pay a monthly fee for the web-accessible monitoring service. The monthly fee varies depending on number of cameras and level of service. We don't know the exact revenue numbers involved, but using a conservative estimate of $10.00 per month for the service, and assuming NETGEAR has 100,000 subscribers, the business would generate $12 million per year, or $32,877 per day. By that measure, a 10-day outage adds up to $328,767 in lost revenue, if the vendor is obligated to offer credits. You can see how the costs of such an outage can quickly add up. Keep in mind that these estimates don't include the cost of finding and fixing the problem, losing a percentage of customers who canceled their subscriptions, and suffering damage to brand image, among other business impacts.
Takeaways: Test for resiliency
As businesses increasingly rely on third-party services and APIs for services, the number of dependencies rise. When something goes awry in the cloud, the impact on the customer—and future revenues for the business—can add up quickly. It's critical for business success to Identify how to test for such dependencies and ensure system resiliency when changes or failures occur.
As the adoption of IoT and connected devices in the home and in business continues to accelerate, more of these incidents will occur, unless businesses test thoroughly beforehand. Developers and testers need to find the issues, not just the first time but with every change. Organizations need to factor in how to test for all dependencies. A thorough discovery of issues must include not only the dependent systems and services but also the service provider's own systems as well.
Keep learning
Learn from your SecOps peers with TechBeacon's State of SecOps 2021 Guide. Plus: Download the CyberRes 2021 State of Security Operations.
Get a handle on SecOps tooling with TechBeacon's Guide, which includes the GigaOm Radar for SIEM.
The future is security as code. Find out how DevSecOps gets you there with TechBeacon's Guide. Plus: See the SANS DevSecOps survey report for key insights for practitioners.
Get up to speed on cyber resilience with TechBeacon's Guide. Plus: Take the Cyber Resilience Assessment.
Put it all into action with TechBeacon's Guide to a Modern Security Operations Center.
