Peering
Peering is voluntary interconnection of administratively separate Internet networks for the purpose of exchanging traffic between the customers of each network. The pure definition of peering is settlement-free or "sender keeps all," meaning that neither party pays the other for the exchanged traffic, instead, each derives revenue from its own customers. Marketing and commercial pressures have led to the word peering routinely being used when there is some settlement involved, even though that is not the accurate technical use of the word. The phrase "settlement-free peering" is sometimes used to reflect this reality and unambiguously describe the pure cost-free peering situation.
Peering requires physical interconnection of the networks, an exchange of routing information through the Border Gateway Protocol (BGP) routing protocol and is often accompanied by peering agreements of varying formality, from "handshake" to thick contracts.
How peering works
The Internet is a collection of separate and distinct networks, each one operating under a common framework of globally unique IP addressing and global BGP routing.
The relationships between these networks are generally described by one of the following three categories:
Transit (or pay) - You pay money (or settlement) to another network for Internet access (or transit).
Peer (or swap) - Two networks exchange traffic between each other's customers freely, and for mutual benefit.
Customer (or sell) - Another network pays you money to provide them with Internet access.
Furthermore, in order for a network to reach any specific other network on the Internet, it must either:
Sell transit (or Internet access) service to that network (making them a 'customer'),
Peer directly with that network, or with a network who sells transit service to that network, or
Pay another network for transit service, where that other network must in turn also sell, peer, or pay for access.
The Internet is based on the principle of global reachability (sometimes called end-to-end reachability), which means that any Internet user can reach any other Internet user as though they were on the same network. Therefore, any Internet connected network must by definition either pay another network for transit, or peer with every other network who also does not purchase transit.
Motivations for peering
Peering involves two networks coming together to exchange traffic with each other freely, and for mutual benefit. This 'mutual benefit' is most often the motivation behind peering, which is often described solely by "reduced costs for transit services". Other less tangible motivations can include:
Increased capacity for extremely large amounts of traffic (distributing traffic across many networks).
Increased control over your traffic (reducing dependence on one or more transit providers).
Improved performance (attempting to bypass potential bottlenecks with a "direct" path).
Improved perception of your network (being able to claim a "higher tier").
Government regulations, or the desire to avoid the appearance of being a monopoly.
Physical interconnections for peering
The physical interconnections used for peering are categorized into two types:
‧Public peering - Interconnection utilizing a multi-party shared switch fabric such as an Ethernet switch.
‧Private peering - Interconnection utilizing a point-to-point interconnection such as a patch-cable or dark fiber between two parties.
Public peering
Public peering is accomplished across a Layer 2 access technology, generally called a shared fabric. At these locations, multiple carriers interconnect with one or more other carriers across a single physical port. Historically public peering locations were known as network access points (NAPs), today they are most often called exchange points or Internet exchanges ("IXP" or "IX"). Many of the largest exchange points in the world can have hundreds of participants, and some span multiple buildings and colocation facilities across a city.
Since public peering allows networks interested in peering to interconnect with many other networks through a single port, it is often considered to offer "less capacity" than private peering, but to a larger number of networks. Many smaller networks, or networks who are just beginning to peer, find that public peering exchange points provide an excellent way to meet and interconnect with other networks who may be open to peering with them. Some larger networks utilize public peering as a way to aggregate a large number of "smaller peers", or as a location for conducting low-cost "trial peering" without the expense of provisioning private peering on a temporary basis, while other larger networks are not willing to participate at public exchanges at all.
A few exchange points, particularly in the United States, are operated by commercial carrier-neutral third parties. These operators typically go to great lengths to promote communication and encourage new peering, and will often arrange social events for these purposes.
Private peering
Private peering is the direct interconnection between only two networks, across a Layer 1 or 2 media that offers dedicated capacity that is not shared by any other parties. Early in the history of the Internet, many private peers occurred across 'telco' provisioned SONET circuits between individual carrier-owned facilities. Today, most private interconnections occur at carrier hotels or carrier neutral colocation facilities, where a direct crossconnect can be provisioned between participants within the same building, usually for a much lower cost than telco circuits.
Most of the traffic on the Internet, especially traffic between the largest networks, occurs via private peering. However, because of the resources required to provision each private peer, many networks are unwilling to provide private peering to "small" networks, or to "new" networks who have not yet proven that they will provide a mutual benefit.
Peering agreements/contracts
Throughout the history of the Internet, there have been a spectrum of kinds of agreements between peers, ranging from handshake deals to peering contracts which may be required by one or both sides. Such a contract sets forth the details of how traffic is to be exchanged, along with a list of expected activities which may be necessary to maintain the peering relationship, a list of activities which may be considered abusive and result in termination of the relationship, and details concerning how the relationship can be terminated. Detailed contracts of this type are typically used between the largest ISPs, and the ones operating in the most heavily-regulated economies, accounting for about 1-2% of peering relationships overall.
History of peering
The first Internet exchange point was the Metropolitan Area Ethernet, or MAE, in Tysons Corner, Virginia. When the United States government decided to de-fund the NSFNET backbone, Internet exchange points were needed to replace its function, and initial governmental funding was used to aid the MAE and bootstrap three other exchanges, which they dubbed NAPs, or "Network Access Points," in accordance with the terminology of the National Information Infrastructure document. All four are now defunct or no longer functioning as Internet exchange points:
‧MAE-East - Located in Tysons Corner, VA, and later relocated to Ashburn, Virginia
‧AADS - Located in Chicago, Illinois
‧SprintNAP - Located in Pennsauken, New Jersey
‧PacBell NAP - Distributed throughout California
As the Internet grew, and traffic levels increased, these NAPs became a network bottleneck. Most of the early NAPs utilized FDDI technology, which provided only 100 Mbit/s of capacity to each participant. Some of these exchanges upgraded to ATM technology, which provided OC-3 (155 Mbit/s) and OC-12 (622 Mbit/s) of capacity.
Other prospective exchange point operators moved directly into offering Ethernet technology, such as gigabit Ethernet (1000 Mbit/s), which quickly became the predominant choice for Internet exchange points due to the reduced cost and increased capacity offered. Today, almost all significant exchange points operate solely over Ethernet, and most of the largest exchange points offer ten gigabit Ethernet (10,000 Mbit/s) service.
During the dot-com boom, many exchange point and carrier neutral colocation providers had plans to build as many as 50 locations to promote carrier interconnection in the United States alone. Essentially all of these plans were abandoned following the dot-com bust, and today it is considered both economically and technically infeasible to support this level of interconnection among even the largest of networks.
Depeering
By definition, peering is the voluntary and free exchange of traffic between two networks, for mutual benefit. If one or both networks believes that there is no longer a mutual benefit, they may decide to cease peering: this is known as depeering. Some of the reasons why one network may wish to depeer another include:
A desire that the other network pay settlement, either in exchange for continued peering or for transit services.
A belief that the other network is "profiting unduly" from the settlement free interconnection.
Concern over traffic ratios, which related to the fair sharing of cost for the interconnection.
A desire to peer with the upstream transit provider of the peered network.
Abuse of the interconnection by the other party, such as pointing default or utilizing the peer for transit.
Instability of the peered network, repeated routing leaks, lack of response to network abuse issues, etc.
The inability or unwillingness of the peered network to provision additional capacity for peering.
The belief that the peered network is unduly peering with your customers.
Various external political factors (including personal conflicts between individuals at each network).
In some situations, networks who are being depeered have been known to attempt to fight to keep the peering by intentionally breaking the connectivity between the two networks when the peer is removed, either through a deliberate act or an act of omission. The goal is to force the depeering network to have so many customer complaints that they are willing to restore peering. Examples of this include forcing traffic via a path that does not have enough capacity to handle the load, or intentionally blocking alternate routes to or from the other network. Some very notable examples of these situations have included:
‧BBN Planet vs. Exodus Communications[1]
‧PSINet vs. Cable & Wireless[2]
‧AOL Transit Data Network (ATDN) vs. Cogent Communications[3]
‧Teleglobe vs. Cogent Communications[citation needed]
‧France Télécom vs. Cogent Communications[4]
‧France Télécom (Wanadoo) vs. Proxad (Free)[5]
‧Level 3 Communications vs. XO Communications[citation needed]
‧Level 3 Communications vs. Cogent Communications[6]
Modern peering
The modern Internet operates with significantly more peering locations than at any time in the past, resulting in improved performance and better routing for the majority of the traffic on the Internet. However, in the interests of reducing costs and improving efficiency, most networks have attempted to standardize on relatively few locations within these individual regions where they will be able to quickly and efficiently interconnect with their peering partners.
The primary locations for peering within the United States are generally considered to be[citation needed]:
‧San Francisco Bay Region (San Jose CA, Palo Alto CA, Santa Clara CA, San Francisco CA)
‧Washington DC / Northern Virginia Region (Washington, DC, Ashburn VA, Reston VA, Vienna VA)
‧New York City Region (New York NY, Newark NJ)
‧Chicago Region (Chicago IL)
‧Los Angeles Region (Los Angeles, CA)
‧Dallas Region (Dallas, TX, Plano, TX, Richardson, TX)
For international traffic, the most important locations for peering are generally considered to be:
‧Amsterdam, Netherlands
‧London, United Kingdom
‧Frankfurt, Germany
‧Tokyo, Japan
‧Hong Kong, China
‧Seoul, South Korea
‧Miami, FL, USA
Exchange points
The largest individual exchange points in the world are AMS-IX in Amsterdam, followed closely by LINX in London. The next largest exchange point is generally considered to be JPIX in Tokyo, Japan. The United States, with a historically larger focus on private peering and commercial public peering, has a much smaller amount of traffic on public peers compared to other regions which operate non-profit exchange points. The combined exchange points in multiple cities operated by Equinix are generally considered to be the largest and most important, followed by the PAIX facilities which are currently owned and operated by Switch and Data. Other important but smaller exchange points include LIPEX and LONAP in London UK, DE-CIX in Frankfurt Germany, NYIIX in New York, and Nap of the Americas (or NOTA) in Miami, Florida.
URLs to some public traffic statistics of exchange points include:
‧AMS-IX
‧DE-CIX
‧LINX
‧MSK-IX
‧TORIX
‧NYIIX
‧LAIIX
‧TOP-IX
‧Netnod
Peering and BGP
A great deal of the complexity in the BGP routing protocol exists to aid the enforcement and fine-tuning of peering and transit agreements. BGP allows operators to define a policy that determines where traffic is routed. Three things commonly used to determine routing are local-preference, multi exit discriminators (MEDs) and AS-Path. Local-preference is used internally within a network to differentiate classes of networks. For example, a particular network will have a higher preference set on internal and customer advertisements. Settlement free peering is then configured to be preferred over paid IP transit.
Networks that speak BGP to each other can engage in multi exit discriminator exchange with each other, although most do not. When networks interconnect in several locations, MEDs can be used to reference that network's interior gateway protocol cost. This results in both networks sharing the burden of transporting each others traffic on their own network (or cold potato). Hot-potato or nearest-exit routing, which is typically the normal behavior on the Internet, is where traffic destined to another network is delivered to the closest interconnection point.
Law and policy
Internet interconnection is not regulated in the same way that public telephone network interconnection is regulated. Nevertheless, Internet interconnection has been the subject of several areas of federal policy. Perhaps the most dramatic example of this is the attempted MCI / Sprint merger. In this case, the Department of Justice signaled that it would move to block the merger specifically because of the impact of the merger on the Internet backbone market. In 2001, the Federal Communications Commission's advisory committee, the Network Reliability and Interoperability Council recommended that Internet backbones publish their peering policies, something that they had been hesitant to do before hand. The FCC has also reviewed competition in the backbone market in its Section 706 proceedings which review whether advanced telecommunications are being provided to all Americans in a reasonable and timely manner.
Finally, Internet interconnection has become an issue in the international arena under something known as the International Charging Arrangements for Internet Services (ICAIS).[7] In the ICAIS debate, countries underserved by Internet backbones have complained that it is unfair that they must pay the full cost of connecting to an Internet exchange point in a different country, frequently the United States. These advocates argue that Internet interconnection should work like international telephone interconnection, with each party paying half of the cost.[8] Those who argue against ICAIS point out that much of the problem would be solved by building local exchange points. A significant amount of the traffic, it is argued, that is brought to the US and exchanged then leaves the US, using US exchange points as switching offices but not terminating in the US.[9] In worst case scenarios, traffic from one side of a street is brought to Miami, exchanged, and then returned to another side of the street. Countries with liberalized telecommunications and open markets, where competition between backbone providers occurs, tend to oppose ICAIS.
Peering requires physical interconnection of the networks, an exchange of routing information through the Border Gateway Protocol (BGP) routing protocol and is often accompanied by peering agreements of varying formality, from "handshake" to thick contracts.
How peering works
The Internet is a collection of separate and distinct networks, each one operating under a common framework of globally unique IP addressing and global BGP routing.
The relationships between these networks are generally described by one of the following three categories:
Transit (or pay) - You pay money (or settlement) to another network for Internet access (or transit).
Peer (or swap) - Two networks exchange traffic between each other's customers freely, and for mutual benefit.
Customer (or sell) - Another network pays you money to provide them with Internet access.
Furthermore, in order for a network to reach any specific other network on the Internet, it must either:
Sell transit (or Internet access) service to that network (making them a 'customer'),
Peer directly with that network, or with a network who sells transit service to that network, or
Pay another network for transit service, where that other network must in turn also sell, peer, or pay for access.
The Internet is based on the principle of global reachability (sometimes called end-to-end reachability), which means that any Internet user can reach any other Internet user as though they were on the same network. Therefore, any Internet connected network must by definition either pay another network for transit, or peer with every other network who also does not purchase transit.
Motivations for peering
Peering involves two networks coming together to exchange traffic with each other freely, and for mutual benefit. This 'mutual benefit' is most often the motivation behind peering, which is often described solely by "reduced costs for transit services". Other less tangible motivations can include:
Increased capacity for extremely large amounts of traffic (distributing traffic across many networks).
Increased control over your traffic (reducing dependence on one or more transit providers).
Improved performance (attempting to bypass potential bottlenecks with a "direct" path).
Improved perception of your network (being able to claim a "higher tier").
Government regulations, or the desire to avoid the appearance of being a monopoly.
Physical interconnections for peering
The physical interconnections used for peering are categorized into two types:
‧Public peering - Interconnection utilizing a multi-party shared switch fabric such as an Ethernet switch.
‧Private peering - Interconnection utilizing a point-to-point interconnection such as a patch-cable or dark fiber between two parties.
Public peering
Public peering is accomplished across a Layer 2 access technology, generally called a shared fabric. At these locations, multiple carriers interconnect with one or more other carriers across a single physical port. Historically public peering locations were known as network access points (NAPs), today they are most often called exchange points or Internet exchanges ("IXP" or "IX"). Many of the largest exchange points in the world can have hundreds of participants, and some span multiple buildings and colocation facilities across a city.
Since public peering allows networks interested in peering to interconnect with many other networks through a single port, it is often considered to offer "less capacity" than private peering, but to a larger number of networks. Many smaller networks, or networks who are just beginning to peer, find that public peering exchange points provide an excellent way to meet and interconnect with other networks who may be open to peering with them. Some larger networks utilize public peering as a way to aggregate a large number of "smaller peers", or as a location for conducting low-cost "trial peering" without the expense of provisioning private peering on a temporary basis, while other larger networks are not willing to participate at public exchanges at all.
A few exchange points, particularly in the United States, are operated by commercial carrier-neutral third parties. These operators typically go to great lengths to promote communication and encourage new peering, and will often arrange social events for these purposes.
Private peering
Private peering is the direct interconnection between only two networks, across a Layer 1 or 2 media that offers dedicated capacity that is not shared by any other parties. Early in the history of the Internet, many private peers occurred across 'telco' provisioned SONET circuits between individual carrier-owned facilities. Today, most private interconnections occur at carrier hotels or carrier neutral colocation facilities, where a direct crossconnect can be provisioned between participants within the same building, usually for a much lower cost than telco circuits.
Most of the traffic on the Internet, especially traffic between the largest networks, occurs via private peering. However, because of the resources required to provision each private peer, many networks are unwilling to provide private peering to "small" networks, or to "new" networks who have not yet proven that they will provide a mutual benefit.
Peering agreements/contracts
Throughout the history of the Internet, there have been a spectrum of kinds of agreements between peers, ranging from handshake deals to peering contracts which may be required by one or both sides. Such a contract sets forth the details of how traffic is to be exchanged, along with a list of expected activities which may be necessary to maintain the peering relationship, a list of activities which may be considered abusive and result in termination of the relationship, and details concerning how the relationship can be terminated. Detailed contracts of this type are typically used between the largest ISPs, and the ones operating in the most heavily-regulated economies, accounting for about 1-2% of peering relationships overall.
History of peering
The first Internet exchange point was the Metropolitan Area Ethernet, or MAE, in Tysons Corner, Virginia. When the United States government decided to de-fund the NSFNET backbone, Internet exchange points were needed to replace its function, and initial governmental funding was used to aid the MAE and bootstrap three other exchanges, which they dubbed NAPs, or "Network Access Points," in accordance with the terminology of the National Information Infrastructure document. All four are now defunct or no longer functioning as Internet exchange points:
‧MAE-East - Located in Tysons Corner, VA, and later relocated to Ashburn, Virginia
‧AADS - Located in Chicago, Illinois
‧SprintNAP - Located in Pennsauken, New Jersey
‧PacBell NAP - Distributed throughout California
As the Internet grew, and traffic levels increased, these NAPs became a network bottleneck. Most of the early NAPs utilized FDDI technology, which provided only 100 Mbit/s of capacity to each participant. Some of these exchanges upgraded to ATM technology, which provided OC-3 (155 Mbit/s) and OC-12 (622 Mbit/s) of capacity.
Other prospective exchange point operators moved directly into offering Ethernet technology, such as gigabit Ethernet (1000 Mbit/s), which quickly became the predominant choice for Internet exchange points due to the reduced cost and increased capacity offered. Today, almost all significant exchange points operate solely over Ethernet, and most of the largest exchange points offer ten gigabit Ethernet (10,000 Mbit/s) service.
During the dot-com boom, many exchange point and carrier neutral colocation providers had plans to build as many as 50 locations to promote carrier interconnection in the United States alone. Essentially all of these plans were abandoned following the dot-com bust, and today it is considered both economically and technically infeasible to support this level of interconnection among even the largest of networks.
Depeering
By definition, peering is the voluntary and free exchange of traffic between two networks, for mutual benefit. If one or both networks believes that there is no longer a mutual benefit, they may decide to cease peering: this is known as depeering. Some of the reasons why one network may wish to depeer another include:
A desire that the other network pay settlement, either in exchange for continued peering or for transit services.
A belief that the other network is "profiting unduly" from the settlement free interconnection.
Concern over traffic ratios, which related to the fair sharing of cost for the interconnection.
A desire to peer with the upstream transit provider of the peered network.
Abuse of the interconnection by the other party, such as pointing default or utilizing the peer for transit.
Instability of the peered network, repeated routing leaks, lack of response to network abuse issues, etc.
The inability or unwillingness of the peered network to provision additional capacity for peering.
The belief that the peered network is unduly peering with your customers.
Various external political factors (including personal conflicts between individuals at each network).
In some situations, networks who are being depeered have been known to attempt to fight to keep the peering by intentionally breaking the connectivity between the two networks when the peer is removed, either through a deliberate act or an act of omission. The goal is to force the depeering network to have so many customer complaints that they are willing to restore peering. Examples of this include forcing traffic via a path that does not have enough capacity to handle the load, or intentionally blocking alternate routes to or from the other network. Some very notable examples of these situations have included:
‧BBN Planet vs. Exodus Communications[1]
‧PSINet vs. Cable & Wireless[2]
‧AOL Transit Data Network (ATDN) vs. Cogent Communications[3]
‧Teleglobe vs. Cogent Communications[citation needed]
‧France Télécom vs. Cogent Communications[4]
‧France Télécom (Wanadoo) vs. Proxad (Free)[5]
‧Level 3 Communications vs. XO Communications[citation needed]
‧Level 3 Communications vs. Cogent Communications[6]
Modern peering
The modern Internet operates with significantly more peering locations than at any time in the past, resulting in improved performance and better routing for the majority of the traffic on the Internet. However, in the interests of reducing costs and improving efficiency, most networks have attempted to standardize on relatively few locations within these individual regions where they will be able to quickly and efficiently interconnect with their peering partners.
The primary locations for peering within the United States are generally considered to be[citation needed]:
‧San Francisco Bay Region (San Jose CA, Palo Alto CA, Santa Clara CA, San Francisco CA)
‧Washington DC / Northern Virginia Region (Washington, DC, Ashburn VA, Reston VA, Vienna VA)
‧New York City Region (New York NY, Newark NJ)
‧Chicago Region (Chicago IL)
‧Los Angeles Region (Los Angeles, CA)
‧Dallas Region (Dallas, TX, Plano, TX, Richardson, TX)
For international traffic, the most important locations for peering are generally considered to be:
‧Amsterdam, Netherlands
‧London, United Kingdom
‧Frankfurt, Germany
‧Tokyo, Japan
‧Hong Kong, China
‧Seoul, South Korea
‧Miami, FL, USA
Exchange points
The largest individual exchange points in the world are AMS-IX in Amsterdam, followed closely by LINX in London. The next largest exchange point is generally considered to be JPIX in Tokyo, Japan. The United States, with a historically larger focus on private peering and commercial public peering, has a much smaller amount of traffic on public peers compared to other regions which operate non-profit exchange points. The combined exchange points in multiple cities operated by Equinix are generally considered to be the largest and most important, followed by the PAIX facilities which are currently owned and operated by Switch and Data. Other important but smaller exchange points include LIPEX and LONAP in London UK, DE-CIX in Frankfurt Germany, NYIIX in New York, and Nap of the Americas (or NOTA) in Miami, Florida.
URLs to some public traffic statistics of exchange points include:
‧AMS-IX
‧DE-CIX
‧LINX
‧MSK-IX
‧TORIX
‧NYIIX
‧LAIIX
‧TOP-IX
‧Netnod
Peering and BGP
A great deal of the complexity in the BGP routing protocol exists to aid the enforcement and fine-tuning of peering and transit agreements. BGP allows operators to define a policy that determines where traffic is routed. Three things commonly used to determine routing are local-preference, multi exit discriminators (MEDs) and AS-Path. Local-preference is used internally within a network to differentiate classes of networks. For example, a particular network will have a higher preference set on internal and customer advertisements. Settlement free peering is then configured to be preferred over paid IP transit.
Networks that speak BGP to each other can engage in multi exit discriminator exchange with each other, although most do not. When networks interconnect in several locations, MEDs can be used to reference that network's interior gateway protocol cost. This results in both networks sharing the burden of transporting each others traffic on their own network (or cold potato). Hot-potato or nearest-exit routing, which is typically the normal behavior on the Internet, is where traffic destined to another network is delivered to the closest interconnection point.
Law and policy
Internet interconnection is not regulated in the same way that public telephone network interconnection is regulated. Nevertheless, Internet interconnection has been the subject of several areas of federal policy. Perhaps the most dramatic example of this is the attempted MCI / Sprint merger. In this case, the Department of Justice signaled that it would move to block the merger specifically because of the impact of the merger on the Internet backbone market. In 2001, the Federal Communications Commission's advisory committee, the Network Reliability and Interoperability Council recommended that Internet backbones publish their peering policies, something that they had been hesitant to do before hand. The FCC has also reviewed competition in the backbone market in its Section 706 proceedings which review whether advanced telecommunications are being provided to all Americans in a reasonable and timely manner.
Finally, Internet interconnection has become an issue in the international arena under something known as the International Charging Arrangements for Internet Services (ICAIS).[7] In the ICAIS debate, countries underserved by Internet backbones have complained that it is unfair that they must pay the full cost of connecting to an Internet exchange point in a different country, frequently the United States. These advocates argue that Internet interconnection should work like international telephone interconnection, with each party paying half of the cost.[8] Those who argue against ICAIS point out that much of the problem would be solved by building local exchange points. A significant amount of the traffic, it is argued, that is brought to the US and exchanged then leaves the US, using US exchange points as switching offices but not terminating in the US.[9] In worst case scenarios, traffic from one side of a street is brought to Miami, exchanged, and then returned to another side of the street. Countries with liberalized telecommunications and open markets, where competition between backbone providers occurs, tend to oppose ICAIS.
Comments