Tata Cable Cut Affects SaaS Services Worldwide

Posted by on February 25, 2015

At 15:35 UTC (21:05 Mumbai, 8:35 Pacific) on February 25th, Tata Communications reported a cable cut between India and Singapore affecting nearly 350 Gbit per sec of capacity. We received a notification via one of our hosting providers in India. Even prior to the notification, we began seeing issues across a range of services where traffic was flowing to and from India. From more than 60 services that we were testing, we saw packet loss increase to over 3% in Asia and rise above 20% in India and Malaysia.

Figure-1
Figure 1: Packet loss increased in Asia from 1% baseline to over 3% immediately after the cut.

Our agents in New Delhi and Kuala Lumpur reported elevated packet loss and latency to many international services that transit Tata and Bharti Airtel, the two primary international transit providers to India and a major providers to Malaysia. For several hours this packet loss ranged from 10-30% and latency increased by 200-300ms. You can interact with the report.

Figure-2
Figure 2: Packet loss exceeds 30% and latency more than doubles from New Delhi to international destinations.

Tata’s Undersea Cable Network

Tata Communications operates the largest subsea fiber optic cable network in the world and is a Tier 1 network provider. Tata has capacity on 3 cables between India and Singapore: SEA-ME-WE 3 and 4 (which are shared with other operators) and Tata Indicom (owned solely by Tata). This cable cut appeared to affect the Tata Indicom Cable (TIC), as we saw traffic affected on the Chennai to Singapore route, with severe congestion in Singapore.

Figure-3
Figure 3: Tata Indicom Cable (TIC) from Chennai to Singapore. (source)

Global Services Affected

Many services with data centers in East Asia and the western United States were affected by the Tata cable cut, given the importance of Tata as a transit provider between India and Singapore. Services that were impacted from India included Box, Docusign, Facebook, Oracle CRM, RingCentral, Salesforce and SAP.

Figure-4
Figure 4: Path trace from New Delhi to storage provider Box in California
shows traffic transiting Tata, with packet loss in Singapore.

Other services that are served from Europe and the eastern United States generally have transit providers from Mumbai to France or Italy via one of the many undersea cables on that route: SEA-ME-WE 3 and 4, FLAG, I-ME-WE, SEACOM, AAE-1 and EIG. In these cases we saw elevated levels of packet loss given congestion, though less severe and for a shorter duration than on the East Asian routes. One of these examples was NetSuite, which also saw packet loss despite transiting Tata’s network via Europe.

Figure-5
Figure 5: Traffic to NetSuite from New Delhi transits Tata via Mumbai and France
after the cable cut, with packet loss at the Tata edge.

Instability occurred throughout Tata’s network, with POPs as far away as Chicago and San Jose registering >50% packet loss. We tracked packet loss and latency until 18:15 UTC, nearly 3 hours, before many Indian networks switched traffic to Bharti Airtel. This often caused increased latency, as traffic peered in suboptimal locations, such as London for traffic destined for Australia. We can see the routing changes away from Tata in Figure 6.

Figure-6
Figure 6: A hosting company (in green) removes Tata (dotted red line) as an upstream provider at 19:00 UTC.

Interested in understanding more about internet performance and sharing your results? You can monitor and report on the performance of the services mentioned above for free with ThousandEyes Lite.

Processing...