Processing...
All posts in: "Outage Reports"

Comcast Fiber Outage Rips the Internet

Posted by on June 29th, 2018
July 2nd, 2018

Earlier today, beginning at 9:20am Pacific Time, Comcast suffered a network outage on its backbone that left millions of Internet users in the United States—even beyond their own subscriber base—unable to reach critical sites and services. The outage wasn’t contained to a particular region, as Internet users from Seattle to New York were affected. The significance of this outage was not only in its massive scope but also in its length, which stretched for nearly three hours—effectively an eternity in Internet timescale.… Read More »

AWS US-East-2 Outage and Monitoring Microservices

Posted by on May 31st, 2018
May 31st, 2018

On the morning of May 31st, 2018, shortly after midnight Pacific Time, users reported an outage in Amazon Web Services’ (AWS) US-East-2 region. An issue with Internet connectivity caused this data center to lose reachability to the Internet, impacting the functioning of any apps with external dependencies such as API services and integration with 3rd party SaaS applications.… Read More »

Network Monitoring to Measure CDN Performance

Posted by on May 2nd, 2018
May 2nd, 2018

Content Delivery Networks (CDNs) have become the architectural cornerstone of network design for enterprises looking to deliver fast and highly available digital experience to their customers. When you rely on a third-party service, like a CDN, to deliver business critical services, it is crucial to continuously monitor their performance and SLA commitments. Because when any one of the cogs in your well oiled digital experience machine fails, your business fails.… Read More »

Anatomy of a BGP Hijack on Amazon’s Route 53 DNS Service

Posted by on April 24th, 2018
April 24th, 2018

How does one steal crypto-coins? By hacking DNS and BGP – the two cornerstone protocols governing the Internet. With any cryptographic system, one way to compromise it is to be a man in the middle (MITM) when a transaction is taking place. Both ends of the transaction believe they’re talking to a legitimate party at the other end, but in reality, a malicious actor sits in the middle and alters the information being passed through to their advantage. In case of cryptocurrency, it could be the public key of the transfer recipient. … Read More »

Understanding Internet Exchanges via the DE-CIX Outage

Posted by on April 13th, 2018
April 13th, 2018

Internet Exchanges (IX) play an important “efficiency” role in the practical matter of physically connecting the networks that comprise the Internet. IXs are facilities that provide space, power and connectivity to member networks—typically Internet Service Providers (ISPs), telecom, cable and mobile carriers, content providers, web enterprises and sophisticated enterprises such as large financial services companies. This infrastructure is used by these networks to arrange peerings exchange regional, national and/or international IP traffic. By being able to peer with many different networks in one facility, networks can cost-effectively gain lower-latency connections than if they had to reach those networks via multiple transit hops. The availability of multiple peerings also increases network and transit path resiliency. … Read More »

Amazon AWS Outage a Lesson in Managing Cloud First Risks

Posted by on March 2nd, 2018
March 6th, 2018

On Friday, March 2nd, 2018, a power outage hit the AWS-East Region (Ashburn), affecting hundreds of critical enterprise services like Atlassian, Slack and Twilio. Significant corporate websites and Amazon’s own service offerings were impacted as well. ThousandEyes operates a global set of software agents performing Internet-aware network monitoring. We were able to catch this outage in the act, which rippled very quickly outward towards other services. Even customers relying on Amazon private connectivity service known as AWS Direct connect were affected.… Read More »

How GitHub Successfully Mitigated a DDoS Attack

Posted by on March 1st, 2018
March 5th, 2018

On Wednesday, February 28th, 2018 at 9:15am Pacific Standard Time, GitHub, the popular web-based hosting service for software development, was a victim of a powerful DDoS attack that impacted its global user base of 20M. It was one of the largest DDoS attacks, with attack traffic peaking at 1.3Tbps. Concerned users, some who were in the midst of a code check-in, took to Twitter to express their feelings while simultaneously looking for answers. Unfortunately for GitHub, their status page was also affected, adding a bit of anxiety to the equation.… Read More »

A Tale of Two Trading Sites Told by Network Monitoring

Posted by on February 22nd, 2018
February 22nd, 2018

How can network monitoring help understand online trading website performance? In the first days of February, 2018 the stock market went on a bit of wild ride, with high levels of volatility amidst heavy trading sessions. On February 5th, the Dow skidded almost 1600 points before recovering slightly to a loss of 1175 points for the day—its worst daily point drop ever. Asian markets soon experienced their own swoon. Then on February 6, U.S. markets bounced back—regaining nearly 570 points, while the FTSE experienced its biggest drop since Brexit. As of February 9, the volatility continued.… Read More »

Top Internet and Cloud Outages of 2017

Posted by on January 17th, 2018
January 17th, 2018

Now that we have rung in the new year and settled into our daily routine, let’s take a few minutes to reflect on the state of the Internet and key outages that shook the world in 2017. Over the past few years, as we analyze many such outage events, we’ve noticed an undeniable trend that governs today’s cloud era: The Internet is the backbone of human communication. We start and end our day with it. Be it calling a ride to work or ordering dinner on the way back, the Internet has a part to play in our quality of life. Which is why when it breaks, it causes havoc.… Read More »

Comcast Suffers Outage Due to Significant Level 3 BGP Route Leak

Posted by on November 7th, 2017
November 8th, 2017

On Monday morning, November 6th 2017, from 9:45am-11:25am Pacific, Comcast suffered a nationwide outage sending several million users of the popular Internet Service Provider into a frenzy. Our analysis has revealed that Level 3 leaked over a thousand routes belonging to Comcast subsidiary networks and their customers. The immediate effect was that instead of traffic going through the Comcast backbone to reach these networks, traffic was forced to go via Level 3.… Read More »