When Routes Go Wrong: Solving BGP AS-Path Prepending Errors

Posted by on October 21, 2014

One scenario we captured and demonstrated during Networking Field Day 8 involved a BGP change at Country Financial. Follow along with this interactive share link if you would like to see the data for yourself, or watch the recorded presentation from NFD8 below.

Around 17:25 UTC our cloud agents started alerting to connection issues with Country Financial’s website. We saw a dip in availability and alerting on this test for over an hour (Figure 1).

Figure 1: Availability to Country Financial below 50% for over an hour.
Figure 1: Availability to Country Financial below 50% for over an hour.

In order to get some context into Country Financial’s network, let’s take a look at a time before the event. When we look at the Path Visualization at 17:15 UTC on September 4th we see one upstream internet service provider, Access2Go (Figure 2).

Figure 2: All traffic to Country Financial’s web server, green node on the far right, travels through the Access2Go network, shown as a blue node.
Figure 2: All traffic to Country Financial’s web server, green node on the far right, travels through the Access2Go network, shown as a blue node.

The BGP Route Visualization confirms this (Figure 3). Autonomous System (AS) 40948 belongs to Access2Go and is shown in grey, indicating it is the provider. The origin network, AS 10511, is registered to CC Services Inc, another name for Country Financial.

Figure 3: Country Financial’s AS 10511 is show in green with ISP Access2Go’s AS 40948 in grey.
Figure 3: Country Financial’s AS 10511 is show in green with ISP Access2Go’s AS 40948 in grey.

At 18:30 UTC, after the event, we looked again at the Path Visualization. It appears that during the last hour the provider changed from Access2Go to Qwest (Figure 4).

Figure 4: Country Financial is now served by Qwest.
Figure 4: Country Financial is now served by Qwest.

To confirm this we can look at the BGP Route Visualization view. However instead of seeing AS 209 (Qwest) in grey as the only provider, we also see AS 15011 (Jaguar Communications). Even more interestingly, none of the monitors are peering with AS 15011 (Figure 5).

Figure 5: Country Financial’s AS 10511 is connected to both AS 209 (Qwest), the correct upstream ISP, and AS 15011 (Jaguar Communications), a BGP prepending misconfiguration.
Figure 5: Country Financial’s AS 10511 is connected to both AS 209 (Qwest), the correct upstream ISP, and AS 15011 (Jaguar Communications), a BGP prepending misconfiguration.

Qwest (AS 209) is the provider, however during the provider change someone trying to increase the cost of the path using AS-path prepending mistyped 15011 instead of 10511. I had a quick look at the raw data and confirmed this is what happened.

Figure 6: Raw BGP data showing the AS path from AS 3561 (Saavis) on the left to AS 10511 (Country Financial) on the right, with two correctly prepended paths and one incorrect path to AS 15011 (Jaguar Communications).

What is BGP prepending? BGP prefers routes with the shortest AS path. To manipulate the inbound route taken, administrators can manipulate the AS path by prepending, or adding AS numbers to a route advertisement so that a route looks less favorable. A typical use case for prepending would be on multi-homed networks, where administrators will use prepending to disfavor a route that has higher costs or is the backup link.

Most of the time routing changes take place and they go fine – however sometimes just one keystroke can introduce latency, errors, or even unintentional peering sessions. If you’re interested in learning more about BGP, take a look at the Visualizing and Troubleshooting BGP Webinar.

Processing...