Creating the Magic Behind Performance Improvement Through Peering

Incapsula global points of presence (PoPs) are designed to accelerate website performance by serving customers content located closest to where their requests originate from. Interconnection points with local ISPs help us bring traffic even closer to our PoPs to further reduce round trip time and give users a better experience.

Peering is the ability of one entity to connect to other entities. In this context, it refers to the network-to-network relationship.

In this post, I explain the process and results of our peering efforts.

The Challenge

When a user requests one of our protected web pages, we would ideally like to respond to the request from an edge network location that provides the lowest latency.

In reality, the internet has its own methods of routing and prioritization. By design BGP routing gives traffic a higher priority to the fastest route. The factors that influence it include autonomous system numbers (ASNs), network operators, transit providers, and Tier 1 providers that have weighed the costs versus results and applied economical reasoning to determine routing prioritization. The truth is financial considerations ultimately influence the best route in favor of the profitable route.

How Peering Works

Transit providers use different routing paths depending upon the ASN involved.

When two ASNs peer, their exchange prefixes will receive priority, enabling regional packets to change hands between those ASNs at the earliest point or according to a specific business consideration. The physical infrastructure through which ASNs exchange traffic between their networks is called the internet exchange point (IX).

Peering allows us to optimize our relationship with another ASN (that would otherwise belong to local ISPs) or other network, with whom we share interests. Our potential peering partner wants to offload the traffic as early as possible, and our goal is to receive it as close as possible to its point of origin. This is because once our edge network receives the user’s request, we can start accelerating its handling.

A few weeks ago, we established an interconnect agreement with a local ISP in Vietnam. Here you can see how response time dropped from a median of 180 ms to a median of 48 ms:

When looking at our backbone, the shift of traffic from Hong Kong to Singapore is clearly visible in the greenish yellow section of the chart below:

When we did not have a direct peering agreement with that specific ISP in Hong Kong, traffic was exchanged over the internet. Once the peering agreement was in place, the Singapore IX, with whom we had a direct link, became a favored point of routing.

Another example demonstrates the impact peering has on response time, captured by real user monitoring (RUM) in Indonesia:

* Median of 95ms

And here you see the results once we established a peering agreement with regional ISPs. The median latency dropped by half:

* Median of 54ms

How Do We Do It?

The first step is to get the list of all relevant peers in a specific peering point. PeeringDB is a great tool to start with. By searching for relevant ASNs in IX, you’ll get a report similar to this one: PeeringDB. Another option is a local IX indexing service, such as https://www.hkix.net/hkix/participant.htm.

Next, we can start mapping the relevant ASNs.

In this pie chart, we can see the breakdown between ASNs in our network at one specific PoP/location

To identify a good potential match, we look at two factors:

1. Large traffic exchange with Incapsula
2. The ratio of ingress and egress traffic to a specific ASN

We use one of our internal real-time tools to identify an ASN as described above and examine real traffic measurements from our layer 7 appliances to get the RPS.

Once we identify a specific ASN to peer with, we can approach them and establish peering and exchange routes / prefixes according to their peering policy.

Finally, we look at Cedexis reports to view and validate the performance improvement from a real user perspective.

In addition, we also look at the data from the bottom up for details, such as searching specific sites with high latency and analyzing what we can do to improve their end user experience.

Summary

We have recently put a lot of peering efforts into the APAC region and are already seeing nice results. Similar efforts are taking place in other regions. And as we keep adding more peers to our network, we continue to invest in new POP deployments. You can track them online on the Incapsula Global Network Map.