Cisco Nonstop Forwarding for BGP: Deployment & Troubleshooting
1.0 Overview
Cisco Nonstop Forwarding with Stateful Switchover (NSF with SSO) is a Cisco innovation for routers with dual route processors. Cisco NSF with SSO allows a router that has experienced hardware or software failure of an active route processor to maintain data link layer connections and continue forwarding packets during the switchover to the standby route processor. This forwarding can continue despite lost routing protocol peering arrangements with other routers. Routing information is recovered dynamically, in the background, while packet forwarding proceeds uninterrupted.
Cisco NSF for BGP is a combination of internal system modifications to the various NSF-capable hardware platforms, and external enhancements to the BGP-4 protocol. The modifications to the BGP protocol (BGP Graceful Restart) have been submitted to the Internet Engineering Task Force (IETF):
http://www.ietf.org/internet-drafts/draft-ietf-idr-restart-06.txt1
This document will detail specific changes to the BGP protocol. In addition, it will explore common deployment scenarios for BGP NSF and the basic troubleshooting techniques available to analyze the functionality of this new technology.
2.0 Benefits of Cisco Nonstop Forwarding
In pursuit of higher revenues and profitability, Service Providers and Enterprises are increasingly putting more mission-critical, time sensitive services on their IP infrastructure. One of the key challenges in this business model is achieving and delivering high network availability. This network availability is measured and billed appropriately via a Service Level Agreement (SLA); therefore, users must address the following issues:
•Increase network and node availability during planned or unplanned software restarts, peer resets, and/or hardware (e.g., Route Processor [RP]) changes. In other words, negatively impacting events must be minimized, and maintenance windows decreased.
•Minimize topology changes seen in the network. Topological changes could cause route flapping, consuming expensive CPU cycles on the router, and producing packet jitter and undesirable traffic patterns through sub-optimal routing.
•Reduce Capital Expenditures (CapEx) and Operational Expenditures (OpEx) in deploying a highly available network.
Cisco NSF addresses these issues by:
•Providing transparent RP switchover during a hardware or software fault on a router
•Masking the impact of any failure by localizing the associated topology changes so they do not cascade throughout the entire network
•Reducing CapEx and OpEx by providing intelligent unattended failover within a single router.
•Maintaining original capital investment by enabling features on existing Cisco platforms and route processors
3.0 Protocol Enhancements
This section provides an examination of how BGP Graceful Restart works. For a more complete discussion, refer to the IETF Internet draft at http://www.ietf.org/internet-drafts/draft-ietf-idr-restart-06.txt.
Cisco routers that support dual RPs and Cisco NSF with SSO can maintain Layer 2 data link connections and sufficient forwarding information to continue processing packets during a RP switchover. The ability to maintain Layer 2 connections during such an event is referred to as Cisco SSO.2 The ability to continue forwarding packets is implemented by using the existing information in the Forwarding Information Base (FIB) and the Cisco Express Forwarding (CEF) tables until the routing protocols can reconverge. This document will refer to a router going through an RP switchover as the Restarting Router.
However, all of these innovations would be for naught if routers that were peered with the router that performs the switchover (hereafter, the Peer Routers) did not continue to forward packets to it. In order for a Peer Router to continue packet forwarding, several conditions must be met:
•The Restarting Router and the Peer Router must each agree to support BGP Graceful Restart
•The Peer Router must not prematurely declare the Restarting Router as unavailable
•The Peer Router must not communicate any state change in the Restarting Router to any of its peers. This avoids the network-wide detrimental effect on performance associated with the sudden failure of a router
•The Peer Router must send BGP updates to help the restarting NSF router reacquire its BGP Routing Information Base (RIB)
• The Peer Router must signal the completion of the initial routing update by sending the End-of-RIB marker (discussed below)
•In the interim (before the Restarting Router has reacquired the routing information), the Peer Router must mark any routes associated with the Restarting Router as "stale", but continue to use those routes for packet forwarding
The protocol modifications begin when the initial BGP connection is established. Both the NSF-capable router and its peer indicate their understanding of the BGP Graceful Restart mechanism by exchanging a new BGP capability (#64) during the initial BGP OPEN that establishes the session.
Note that the Peer Router will send Capability 64, regardless of whether it is capable of restartability. Capability 64 does not alone indicate restartability. It can indicate that the router in question has implemented the BGP enhancements specified in the IETF draft. Thus, a Cisco 7200 Series Router that is BGP graceful-restart configured will still advertise Capability 64 to its peers, even though it does not support dual RPs and cannot restart BGP.
Additionally, the NSF-capable router will provide a list of Address Family Identifiers (AFI) and Subsequent Address Family Identifiers (SAFI) for which it has the capability to maintain forwarding state across a BGP Restart. The AFI and SAFI indicate different types of protocols for which BGP can carry information. This would include protocol support, including IPv4, IPv6, MPLS, and Unicast/Multicast routing.
Figure 1 illustrates the significant fields within the new Capabilities 64 exchange, and provides a brief discussion of their usage. Table 1 offers more detailed information about each field.
Figure 1
Format of BGP Graceful Restart Capability 64
When the NSF-capable router performs a route processor switchover, the TCP connection to the Peer Router is cleared; a Peer Router that does not support BGP restart then clears all routes associated with the Restarting Router and no longer forwards packets to it. With BGP Graceful Restart, the Peer Router marks all routes to the Restarting Router as stale, but continues to use them for packet forwarding, based upon the knowledge that the Restarting Router will re-establish the BGP session shortly and that it maintains the capability to forward packets in the interim.
When the Restarting Router's newly active RP opens the new BGP session, it will again send the Graceful Restart capability (#64). However, this time, the restart bit in the Restart Flags portion of the capability exchange will be set. This notifies the Peer Routers that the restart of the BGP process on the Restarting Router caused the disconnect/reconnect.
While continuing to forward packets, the Peer Router refreshes the Restarting Router with any relevant BGP updates. The Peer Router indicates completion of this process by sending an End-of-RIB (EOR) marker. The EOR marker for IPv4 is a BGP update message that is of the minimum length—23 bytes. The EOR does not contain any routes to be added or withdrawn. Essentially, it is an "empty" update, whose sole purpose is to indicate that all available routes have been sent. The EOR marker helps speed convergence, because it allows the router to begin best-path selection as quickly as possible, without waiting for the timer to expire.3
Once the Restarting Router has received all available routes from each peer, it can conduct best-path selection, and send any updates to its Peer Routers. The Restarting Router will also use the EOR to indicate the completion of this process.
Figure 2 provides a graphical representation of this process.
Figure 2
BGP Graceful Restart Procedures
Consider the step-by-step protocol exchange to clarify the implementation of End of RIB (EOR) and Graceful Restart (GR). The goal is to restart a BGP session without the Restarting Router's peers redirecting traffic around the Restarting Router.
1. The BGP process of Router A (RTR_A) BGP begins, and it establishes a peering relationship with router B (RTR_B). It sends an OPEN message to router B, and the OPEN message includes the Graceful Restart Capability (Code 64) and Address Family of IPv4, Subsequent Address Family of unicast. Because router B also supports GR, it also sends an acknowledgement via its own OPEN Message, which contains GR=64 and AF=IPv4.
2. An RP switchover occurs and Router A's BGP process restarts on the newly active RP. Router A does not have a routing information base on this RP, and must reacquire it from its Peer Routers. Router A will continue to forward IP packets destined for (or through) Router B using the last updated FIB and CEF table.
3. When the Receiving router (Router B) detects that the TCP session between it and Restarting Router is cleared, it immediately marks routes learned from the Restarting Router as STALE. Router B only marks routes learned from Router A as STALE; If B had other peers, then the routes learned from those peers would remain in the UP state. Router B also initializes a Restart-timer for the Restarting Router. The default setting for this timer is 120 seconds. The Restart-Timer is the amount of time that a Receiving router will wait for an OPEN message from the Restarting Router. A Receiving router will remove all STALE routes unless it receives an OPEN message from the Restarting Router within the specified Restart-time. Once router B receives router A's OPEN message, the Restart-timer is reset. During this time, Routers A and B continue to forward traffic using the last updated CEF table.
4. Router A's BGP process has initialized. It will now attempt to re-establish a BGP session with router B. It first establishes a new TCP session, and then it sends an OPEN message (Restart State bit set, Restart Time= n, and Forwarding State= IPv4). By default, Restart time is 120 seconds and it is also configurable. When Router B receives this OPEN message, it resets its own Restart-timer and starts a Stale-path timer. Stale-path, by default, is 360 seconds and is configurable.
5. Both routers successfully re-establish their session. At this point, if Router B recognizes that the Forwarding State in Router A's OPEN message is not set for IPv4, it immediately removes any STALE routes, which it had learned from the Restarting Speaker, and re-computes its routing database. (Normally, the Forwarding State will be set for IPv4)
6. Router B will begin to send UPDATE messages to Router A. These messages contain IP prefix information, and Router A will process them accordingly. Until an EOR indication is received from all peers (or the bgp update-delay timer expires), Router A will not start the BGP Route Selection Process. A new routing information database is available after the Route Selection Process is finished and the CEF information is updated accordingly. Router A starts an update-delay timer and waits up to 120 seconds to receive EOR from all of its NSF-peers.
7. Once Router A has received EOR from all its peers, it will begin the BGP Route Selection Process. Once this process is complete, it will begin to send UPDATE messages, which contain prefix information, to router B. Router A concludes this process by sending an EOR indication to Router B so that B, in turn, can start its Route Selection Process. Once Router B receives an EOR from A, and it has completed its Route Selection Process, then any STALE entries in BGP will be refreshed with newer information or removed from the BGP RIB and FIB. Router B is now converged. While Router B waits for an EOR, it also monitors stalepath-time. If the timer expires, all STALE routes will be removed and "normal" BGP processes will be started.
4.0 Router Preparation and Network Configuration:
In order to ensure a successful migration to a Graceful Restart-capable router, there are a few important principles to consider.
The router must have compatible RPs installed. In addition, care should be taken when mixing RP types:
•Cisco 12000 Series Internet Router: GRP and GRP-B RPs can be used together. If using a PRP on this router, it must be paired with another PRP.
•Cisco 10000 Series Internet Router: two PRE-1s must be used. The original PRE for this router is not supported for purposes of Cisco NSF with SSO.
•Cisco 7500: RSP-2 and RSP-4 can be used in combination. RSP-8 and RSP-16 can also be used in combination. However, an RSP-8 or RSP-16 cannot be mixed with an RSP-2 or an RSP-4.
•For all RP types on all supported platforms, the active and standby RPs must have the same amount of memory
A wide variety of line cards support Cisco NSF with SSO, but—for optimum performance of BGP Graceful Restart—every card in the router chassis should support Cisco SSO. For a list of supported line cards, please visit: http://www.cisco.com/en/US/products/sw/iosswrel/ps1829/1221748
Cisco SSO may not be supported on any line card not specifically listed in the aforementioned document. In this case, that specific line card will operate in RPR+ mode. At the time of the RP switchover, the dCEF table on the card will be cleared. This will cause Cisco NSF to destinations reachable through that card to fail.
Subsequent releases of Cisco IOS Software provide additional hardware support for Cisco SSO on specific line cards. Please check the release notes for later releases of Cisco IOS Software to determine if support for a particular line card may be available.
The referenced document also supplies detailed instructions on enabling SSO on the platforms that support Cisco NSF. Cisco SSO is an absolute requirement for enabling Cisco NSF; it will not work unless both are concurrently enabled.
On the Cisco 12000 Series Internet Router, there is a method to validate whether all line cards within a chassis are supported. Load a software image enabled with Cisco NSF with SSO and then issue the command "show redundancy mode-supported". Each card in the chassis will be listed, and indicate the highest level of system redundancy it supports (RPR, RPR-Plus, Cisco SSO).
To achieve the full benefit of Cisco NSF with SSO, all line cards should support Cisco SSO. Furthermore, depending on platform, Distributed Cisco Express Forwarding (dCEF) must be enabled for the line cards in order for NSF to work.
The correct software image must be loaded on the flash disks of both route processors. Currently, mixing software versions between the active and standby router processors is not supported—even if both software images support Cisco NSF with SSO.
The software boot image in bootflash should also be upgraded and should correspond to the software image being loaded on the RP.
BGP Graceful Restart is configured under the global "router bgp" configuration command. The most basic configuration is "bgp graceful-restart"
Router(config-route)# [no] bgp graceful-restart
Router(config-route)# [no] bgp graceful-restart restart-time n
Router(config-route)# [no] bgp update-delay n
Router(config-route)# [no] bgp graceful-restart stalepath-time n
The "bgp graceful-restart" command must be entered on the Cisco NSF-capable router, and also must be entered on any NSF-aware peer that will be participating in Graceful Restart. Graceful Restart is not enabled by default, and must be explicitly configured on both the Restarting Router and all Peer Routers.
The "bgp graceful-restart restart-time n" is the maximum amount of time that a peer will wait for a reconnection of the TCP session and a new BGP OPEN message following the detection of a failure on the Restarting Router. If the TCP and BGP sessions are not re-established before this timer expires, the BGP session is deemed a failure, and normal BGP recovery procedures take effect. The default value for restart-time is 120 seconds.
The "bgp update-delay n" command may be entered on the Cisco NSF-capable router. The update-delay specifies the time interval- after the first peer has reconnected—during which the restarting router expects to receive all BGP updates and the EOR marker from all of its configured peers. The default value of n is 120 seconds, and n is always measured in seconds. If the restarting router has a large number of peers, each with a large number of updates to be sent, this value may need to be increased from its default value.
The "bgp graceful-restart stalepath-time n" command may be entered on the NSF-aware peer(s) of the restarting router. This timer sets an upper limit on how long the peer will continue to use stale routes for forwarding after it has re-established the BGP session with the restarting router. The default value is 360 seconds. While this should allow an adequate amount of time to allow for complete convergence, on very large networks it may be necessary to increase this value.
5.0 Deployment
There are many different variations on design and deployment of BGP networks. However, to simplify matters, it is easier to think about BGP design in terms of router functionality: What does a particular router need to accomplish given its particular placement within the network? There are three basic types of routers within a BGP network:
•Inter-AS routers run a combination of eBGP and iBGP to connect different autonomous systems. There are many variations to this: edge routers that connect Enterprise customers to the Service Provider network, Internet peering points that connect Service Provider autonomous systems together, edge routers that exist on the boundary of a BGP confederation sub-AS. (see RFC 3065). Yet, the functionality of each of these routers is identical from the Cisco NSF perspective.
•Intra-AS routers exist in the distribution layer or core of an individual AS. These routers run only iBGP and interact only with routers within their own autonomous system. Any knowledge they have of the world outside of their AS is communicated to them via Inter-AS routers
•Route Reflectors (RRs) act as aggregation and distribution points for BGP routing information. Intra-AS routers report BGP routing information to the RRs and receive information from them. RRs increase the scalability of a BGP network by removing the restriction that all iBGP peers must be fully-meshed. The two most common deployment scenarios for RRs are
–Centralized RRs: The Route Reflectors exist at the core of the BGP network, roughly equidistant from all the other routers in the AS. Each router in the AS forms a BGP session with this RR. Frequently, there will be redundant RRs in this configuration
–Distributed RRs: In this design, some subset of routers within an AS will be administratively grouped and have a local RR to which each router will form a BGP session. These RRs subsequently form BGP sessions to other RRs in other regions, or a meshed connection to other RR as well as Intra-AS routers in the core. A typical example of this type of configuration would be a Service Provider that has local RRs in each of its Points of Presence (PoPs). Figure 3 illustrates an eBGP deployment with peers in several different autonomous systems. R1, R2, R3, R4 and R5 are all eBGP peers. Furthermore, the connections R1-to-R4 are multi-homed and are peering to the loopback interface address. R1, R3, and R4 recognize each other as NSF-aware peers because each sends and acknowledges each other's GR capability. On the other hand, the BGP sessions between R1-R2 and R1-R5 are non-NSF, meaning they do not signal or acknowledge the GR capability. The following occurs when R1 goes through a BGP restart:
•R2 detects failure of the TCP/BGP session established over the R1-R2 link and will attempt to route around it. As a result of this re-computation, R2 will take the R2-R3/R3-R1 links to access AS100.
•R1 (the Restarting Router) continues to forward packets destined to AS200 along the R1-to-R2 link, because its CEF table remains intact after switchover. We now have an example of asymmetric routing that can occur when there is a mixture of NSF-aware and non-NSF-aware peers. Although asymmetric routing is not a desirable condition and may result in some packet loss, it is still preferable to the network disruption that would have ensued had R1 completely reinitialized.
•R3 and R4 will not flush routes that they had previously learned from R1. R1 should continue to forward IP packets between R3 and R4, using its last updated CEF Table.
•R3 and R4 should continue to forward IP packets to R1 using their last updated CEF table.
•R5 is non-NSF-aware; as such, it will lose the BGP session to R1 and initialize the BGP session from scratch. R1 will continue to forward packets bound for AS500 through R5, but there is no return path for the traffic. There will be packet loss until R5 successfully reconverges with R1.
•There is an exception to this rule, if R5 has a static default route pointing to R1 as the next-hop. In this case, R5 was only using BGP so it could advertise its routes into the R1 BGP table. The R5 routes are preserved at R1 and R5 only needs a default route, so there should be no packet loss.
Figure 4
NSF—Inter-AS peers
Figure 4 illustrates that R1, RC1 and RC3 are all BGP NSF-aware. Additionally, R1 is NSF-capable—meaning that it supports BGP restartability. Should R1 restart BGP, there should be little-to-no packet loss within AS100. Because R1 is also an Inter-AS router, there may be some packet loss before R2 reconverges to route traffic destined to AS100 via R3. This process was described in the previous section.
Note that there is an important deployment consideration in this scenario. In this topology, it is very common to be running an IGP protocol (i.e.: OSPF or IS-IS) to provide next-hop reachability within AS100. There is interdependence between BGP and the selected IGP protocol. During best-path calculation, BGP knows the IP address of the router advertising certain destination prefixes. However, it relies on the information from the IGP to determine the next-hop to reach that advertising router.
Because BGP Graceful Restart can alter the timing of BGP convergence, situations can potentially occur where BGP is ready to conduct best-path selection, but the IGP has not yet converged. Therefore, some destination prefixes could exist in BGP, but cannot be added to the CEF table because a path to the advertising router has not been calculated by the IGP yet. This could result in packet loss. See section 6.5 of this paper for an example of this situation.
Therefore, it is strongly recommended that NSF for IS-IS or OSPF should be configured in addition to BGP Graceful Restart. To learn more about NSF for both OSPF and IS-IS, review Globally Resilient IP deployment papers: http://www.cisco.com/warp/public/732/Tech/grip/tech.shtml
Figure 5
Cisco NSF—Inter-AS with Route Reflectors
This topology demonstrates that R1, RC1, RC2 and the Route Reflector (RR) are NSF-aware. RR is deployed as a control plane to reduce the requirements for a full iBGP mesh. Thus, it is not in the forwarding path, but forms iBGP peering arrangements with R1, RC1 and RC2. It is also assumed that a flavor of IGP NSF is implemented in this topology.
•When R1 restarts its BGP, it relies on the existing CEF table and continues to forward packets destined to (or through) RC1 and RC2.
•Meanwhile, the only peering arrangement that R1 has is with the Route Reflector. It has no direct peering with RC1 or RC2.
•Because RR is NSF-aware, it masks the fact that R1 has restarted BGP from propagating to RC1 or RC2. RC1 and RC2 continue to forward through R1.
A more interesting variation on this occurs if the Route Reflector is actually NSF-capable and restarted its BGP process.
•When the Route Reflector restarts BGP, all the clients will keep routing information, which was reflected by RR. None of the clients will switch to a backup RR (assuming a backup RR is available).
•Some special considerations must be made when using an NSF-capable Route Reflector
–It is anticipated that a Route Reflector will have more BGP peers and a larger aggregate collection of BGP data than other routers in the AS; due to these conditions, best-path selection may take longer to complete during a switchover.
–Users must balance the requirement to provide uninterrupted packet forwarding and routing stability to the network versus the likelihood of a significant routing change before convergence is complete. During this interim period, Cisco NSF uses the CEF table and not BGP routing information to forward packets.
–Assuming that the decision has been made to use Cisco NSF on the Route Reflector, another configuration adjustment may be required. If it is anticipated that the entire process of reconvergence will exceed 360 seconds, then the default value of the "bgp graceful-restart stalepath-time xxx" may need to be adjusted on all of the peers of the route reflector. The value for stale-path should be adjusted to equal the expected convergence time (in seconds) plus an additional buffer zone of 30-60 seconds to account for variances in convergence time based on changing network conditions.
The decision of whether to use BGP Graceful Restart on a Route Reflector is a complex one and depends largely on network operations. Users must consider the key trade-offs in this decision:
•Is there an alternate availability strategy (i.e.: backup RR); if so, does it provide acceptable failover time?
•How long does it take for the restarting Route Reflector to reconverge, so its Peer Routers can begin to base forwarding decisions on fresh information?
•What is the likelihood that there will be other significant BGP routing changes that occur while the Route Reflector is reconverging?
While these questions are posed in the context of a decision to use Cisco NSF with SSO on a Route Reflector, they are also good general questions that should help in determining where and how to deploy Cisco NSF with SSO.
6.0 Troubleshooting
Because the protocol changes associated with Nonstop Forwarding for BGP are well defined and documented, troubleshooting Cisco NSF with SSO becomes a matter of assuring that the correct protocol exchanges occur at the correct times. The various Cisco IOS Software "show" and "debug" commands have been modified to provide this type of information.
6.1 Validating BGP Graceful Restart
As described in the "Protocol Enhancements" section of this paper, Cisco NSF / BGP begins with, and depends upon, an agreement between the BGP peers that they have implemented the protocol extensions in the IETF Graceful-Restart draft. Examining the output of the show ip bgp neighbor X.X.X.X command can validate this negotiation.
Figure 6
sh ip bgp neighbor
Figure 6 illustrates that the show ip bgp neighbor command now documents the successful negotiation of the Graceful Restart Capability. In order for Cisco NSF to work, the Restarting Router and its peer must each have implemented the BGP protocol enhancements. It is thus vital that the capability be both "advertised and received". This output should be the same, regardless of whether it is queried on the NSF-capable router or its NSF-aware peer. Note that the advertisement of the BGP graceful-restart capability does not specify that a particular router is NSF-capable; it only signifies that the router understands the protocol changes specified in the IETF draft.
The value of the Remote Restart timer is listed. Although the local and remote BGP peers are permitted to have different restart timers, it is wise to be suspicious of a significant variance between the values. The restart timer sets an upper bound on the amount of time it will take to re-establish the TCP connection and renegotiate the graceful-restart capability in a new BGP OPEN message. Setting this value too low on either of the peers makes it unlikely that the OPEN can be re-established in time, and will lead to the failure of the Cisco NSF process.4 BGP will revert to "normal" recovery procedures in this type of case.
Refer to the following line, "Address Families preserved by peer" for information about what Address Families (i.e.: types of traffic) for which the Peer Router can maintain forwarding state during the progression of an RP switchover. IPv4 is currently the only supported Address Family. Future extensions to Cisco NSF will add support for other Address Families (or Subsequent Address Families), including Multicast, IPv6 and MPLS when these protocols are made NSF-capable.
6.2 Validating Stale Routes on an NSF-aware Peer
One of the agreed-upon capabilities of the NSF-aware Peer Router is the use of "stale" routes (the last known good routes before the restart) to continue packet forwarding while the Restarting Router is dynamically reacquiring routing information in the background (see previous section for details). This capability can be queried with the show ip bgp command.
Figure 7
sh ip bgp on NSF-aware peer
In Figure 7, the NSF-aware router has marked the routes for the 170, 180 and 190 networks as "stale" (as signified by the capital "S" beside the route itself). This behavior is expected. Only the routes reachable through the Restarting Router will be "stale". All other routes should appear as normal.
If the specific routes through the Restarting Router are not marked "S", one of several explanations might apply:
•If the route is missing from sh ip bgp, then Cisco NSF has failed for some reason and "normal" BGP reconvergence is taking place. Possible causes:
–The Peer Router is not NSF-aware. See 6.1, above
–The restart timer has expired
–The stale-path timer has expired
•If the route is present, but is not marked "S", then one of the following might apply:
–The Cisco NSF-aware router has not detected the restart on the NSF-capable router yet
–The Cisco NSF process is complete, and the NSF-aware router has replaced the stale route with a fresh one
–If the next-hop is not pointing to the Restarting Router, then Cisco NSF has failed and "normal" BGP recovery procedures have occurred, and the network has reconverged around the Restarting Router
6.3 Debugging Output on the NSF-aware Peer
As with almost all Cisco-supported protocols, BGP NSF supports debugging commands that can monitor the progression of events as a route processor switchover occurs. Table 2 illustrates the output of the commands debug ip bgp events and debug ip bgp updates on the NSF-aware peer of a Restarting Router at the time of a switchover.
...(略)
5.1 Deployment Scenarios
Figure 3
NSF Deployment —Inter-AS Peers
Cisco Nonstop Forwarding with Stateful Switchover (NSF with SSO) is a Cisco innovation for routers with dual route processors. Cisco NSF with SSO allows a router that has experienced hardware or software failure of an active route processor to maintain data link layer connections and continue forwarding packets during the switchover to the standby route processor. This forwarding can continue despite lost routing protocol peering arrangements with other routers. Routing information is recovered dynamically, in the background, while packet forwarding proceeds uninterrupted.
Cisco NSF for BGP is a combination of internal system modifications to the various NSF-capable hardware platforms, and external enhancements to the BGP-4 protocol. The modifications to the BGP protocol (BGP Graceful Restart) have been submitted to the Internet Engineering Task Force (IETF):
http://www.ietf.org/internet-drafts/draft-ietf-idr-restart-06.txt1
This document will detail specific changes to the BGP protocol. In addition, it will explore common deployment scenarios for BGP NSF and the basic troubleshooting techniques available to analyze the functionality of this new technology.
2.0 Benefits of Cisco Nonstop Forwarding
In pursuit of higher revenues and profitability, Service Providers and Enterprises are increasingly putting more mission-critical, time sensitive services on their IP infrastructure. One of the key challenges in this business model is achieving and delivering high network availability. This network availability is measured and billed appropriately via a Service Level Agreement (SLA); therefore, users must address the following issues:
•Increase network and node availability during planned or unplanned software restarts, peer resets, and/or hardware (e.g., Route Processor [RP]) changes. In other words, negatively impacting events must be minimized, and maintenance windows decreased.
•Minimize topology changes seen in the network. Topological changes could cause route flapping, consuming expensive CPU cycles on the router, and producing packet jitter and undesirable traffic patterns through sub-optimal routing.
•Reduce Capital Expenditures (CapEx) and Operational Expenditures (OpEx) in deploying a highly available network.
Cisco NSF addresses these issues by:
•Providing transparent RP switchover during a hardware or software fault on a router
•Masking the impact of any failure by localizing the associated topology changes so they do not cascade throughout the entire network
•Reducing CapEx and OpEx by providing intelligent unattended failover within a single router.
•Maintaining original capital investment by enabling features on existing Cisco platforms and route processors
3.0 Protocol Enhancements
This section provides an examination of how BGP Graceful Restart works. For a more complete discussion, refer to the IETF Internet draft at http://www.ietf.org/internet-drafts/draft-ietf-idr-restart-06.txt.
Cisco routers that support dual RPs and Cisco NSF with SSO can maintain Layer 2 data link connections and sufficient forwarding information to continue processing packets during a RP switchover. The ability to maintain Layer 2 connections during such an event is referred to as Cisco SSO.2 The ability to continue forwarding packets is implemented by using the existing information in the Forwarding Information Base (FIB) and the Cisco Express Forwarding (CEF) tables until the routing protocols can reconverge. This document will refer to a router going through an RP switchover as the Restarting Router.
However, all of these innovations would be for naught if routers that were peered with the router that performs the switchover (hereafter, the Peer Routers) did not continue to forward packets to it. In order for a Peer Router to continue packet forwarding, several conditions must be met:
•The Restarting Router and the Peer Router must each agree to support BGP Graceful Restart
•The Peer Router must not prematurely declare the Restarting Router as unavailable
•The Peer Router must not communicate any state change in the Restarting Router to any of its peers. This avoids the network-wide detrimental effect on performance associated with the sudden failure of a router
•The Peer Router must send BGP updates to help the restarting NSF router reacquire its BGP Routing Information Base (RIB)
• The Peer Router must signal the completion of the initial routing update by sending the End-of-RIB marker (discussed below)
•In the interim (before the Restarting Router has reacquired the routing information), the Peer Router must mark any routes associated with the Restarting Router as "stale", but continue to use those routes for packet forwarding
The protocol modifications begin when the initial BGP connection is established. Both the NSF-capable router and its peer indicate their understanding of the BGP Graceful Restart mechanism by exchanging a new BGP capability (#64) during the initial BGP OPEN that establishes the session.
Note that the Peer Router will send Capability 64, regardless of whether it is capable of restartability. Capability 64 does not alone indicate restartability. It can indicate that the router in question has implemented the BGP enhancements specified in the IETF draft. Thus, a Cisco 7200 Series Router that is BGP graceful-restart configured will still advertise Capability 64 to its peers, even though it does not support dual RPs and cannot restart BGP.
Additionally, the NSF-capable router will provide a list of Address Family Identifiers (AFI) and Subsequent Address Family Identifiers (SAFI) for which it has the capability to maintain forwarding state across a BGP Restart. The AFI and SAFI indicate different types of protocols for which BGP can carry information. This would include protocol support, including IPv4, IPv6, MPLS, and Unicast/Multicast routing.
Figure 1 illustrates the significant fields within the new Capabilities 64 exchange, and provides a brief discussion of their usage. Table 1 offers more detailed information about each field.
Figure 1
Format of BGP Graceful Restart Capability 64
When the NSF-capable router performs a route processor switchover, the TCP connection to the Peer Router is cleared; a Peer Router that does not support BGP restart then clears all routes associated with the Restarting Router and no longer forwards packets to it. With BGP Graceful Restart, the Peer Router marks all routes to the Restarting Router as stale, but continues to use them for packet forwarding, based upon the knowledge that the Restarting Router will re-establish the BGP session shortly and that it maintains the capability to forward packets in the interim.
When the Restarting Router's newly active RP opens the new BGP session, it will again send the Graceful Restart capability (#64). However, this time, the restart bit in the Restart Flags portion of the capability exchange will be set. This notifies the Peer Routers that the restart of the BGP process on the Restarting Router caused the disconnect/reconnect.
While continuing to forward packets, the Peer Router refreshes the Restarting Router with any relevant BGP updates. The Peer Router indicates completion of this process by sending an End-of-RIB (EOR) marker. The EOR marker for IPv4 is a BGP update message that is of the minimum length—23 bytes. The EOR does not contain any routes to be added or withdrawn. Essentially, it is an "empty" update, whose sole purpose is to indicate that all available routes have been sent. The EOR marker helps speed convergence, because it allows the router to begin best-path selection as quickly as possible, without waiting for the timer to expire.3
Once the Restarting Router has received all available routes from each peer, it can conduct best-path selection, and send any updates to its Peer Routers. The Restarting Router will also use the EOR to indicate the completion of this process.
Figure 2 provides a graphical representation of this process.
Figure 2
BGP Graceful Restart Procedures
Consider the step-by-step protocol exchange to clarify the implementation of End of RIB (EOR) and Graceful Restart (GR). The goal is to restart a BGP session without the Restarting Router's peers redirecting traffic around the Restarting Router.
1. The BGP process of Router A (RTR_A) BGP begins, and it establishes a peering relationship with router B (RTR_B). It sends an OPEN message to router B, and the OPEN message includes the Graceful Restart Capability (Code 64) and Address Family of IPv4, Subsequent Address Family of unicast. Because router B also supports GR, it also sends an acknowledgement via its own OPEN Message, which contains GR=64 and AF=IPv4.
2. An RP switchover occurs and Router A's BGP process restarts on the newly active RP. Router A does not have a routing information base on this RP, and must reacquire it from its Peer Routers. Router A will continue to forward IP packets destined for (or through) Router B using the last updated FIB and CEF table.
3. When the Receiving router (Router B) detects that the TCP session between it and Restarting Router is cleared, it immediately marks routes learned from the Restarting Router as STALE. Router B only marks routes learned from Router A as STALE; If B had other peers, then the routes learned from those peers would remain in the UP state. Router B also initializes a Restart-timer for the Restarting Router. The default setting for this timer is 120 seconds. The Restart-Timer is the amount of time that a Receiving router will wait for an OPEN message from the Restarting Router. A Receiving router will remove all STALE routes unless it receives an OPEN message from the Restarting Router within the specified Restart-time. Once router B receives router A's OPEN message, the Restart-timer is reset. During this time, Routers A and B continue to forward traffic using the last updated CEF table.
4. Router A's BGP process has initialized. It will now attempt to re-establish a BGP session with router B. It first establishes a new TCP session, and then it sends an OPEN message (Restart State bit set, Restart Time= n, and Forwarding State= IPv4). By default, Restart time is 120 seconds and it is also configurable. When Router B receives this OPEN message, it resets its own Restart-timer and starts a Stale-path timer. Stale-path, by default, is 360 seconds and is configurable.
5. Both routers successfully re-establish their session. At this point, if Router B recognizes that the Forwarding State in Router A's OPEN message is not set for IPv4, it immediately removes any STALE routes, which it had learned from the Restarting Speaker, and re-computes its routing database. (Normally, the Forwarding State will be set for IPv4)
6. Router B will begin to send UPDATE messages to Router A. These messages contain IP prefix information, and Router A will process them accordingly. Until an EOR indication is received from all peers (or the bgp update-delay timer expires), Router A will not start the BGP Route Selection Process. A new routing information database is available after the Route Selection Process is finished and the CEF information is updated accordingly. Router A starts an update-delay timer and waits up to 120 seconds to receive EOR from all of its NSF-peers.
7. Once Router A has received EOR from all its peers, it will begin the BGP Route Selection Process. Once this process is complete, it will begin to send UPDATE messages, which contain prefix information, to router B. Router A concludes this process by sending an EOR indication to Router B so that B, in turn, can start its Route Selection Process. Once Router B receives an EOR from A, and it has completed its Route Selection Process, then any STALE entries in BGP will be refreshed with newer information or removed from the BGP RIB and FIB. Router B is now converged. While Router B waits for an EOR, it also monitors stalepath-time. If the timer expires, all STALE routes will be removed and "normal" BGP processes will be started.
4.0 Router Preparation and Network Configuration:
In order to ensure a successful migration to a Graceful Restart-capable router, there are a few important principles to consider.
The router must have compatible RPs installed. In addition, care should be taken when mixing RP types:
•Cisco 12000 Series Internet Router: GRP and GRP-B RPs can be used together. If using a PRP on this router, it must be paired with another PRP.
•Cisco 10000 Series Internet Router: two PRE-1s must be used. The original PRE for this router is not supported for purposes of Cisco NSF with SSO.
•Cisco 7500: RSP-2 and RSP-4 can be used in combination. RSP-8 and RSP-16 can also be used in combination. However, an RSP-8 or RSP-16 cannot be mixed with an RSP-2 or an RSP-4.
•For all RP types on all supported platforms, the active and standby RPs must have the same amount of memory
A wide variety of line cards support Cisco NSF with SSO, but—for optimum performance of BGP Graceful Restart—every card in the router chassis should support Cisco SSO. For a list of supported line cards, please visit: http://www.cisco.com/en/US/products/sw/iosswrel/ps1829/1221748
Cisco SSO may not be supported on any line card not specifically listed in the aforementioned document. In this case, that specific line card will operate in RPR+ mode. At the time of the RP switchover, the dCEF table on the card will be cleared. This will cause Cisco NSF to destinations reachable through that card to fail.
Subsequent releases of Cisco IOS Software provide additional hardware support for Cisco SSO on specific line cards. Please check the release notes for later releases of Cisco IOS Software to determine if support for a particular line card may be available.
The referenced document also supplies detailed instructions on enabling SSO on the platforms that support Cisco NSF. Cisco SSO is an absolute requirement for enabling Cisco NSF; it will not work unless both are concurrently enabled.
On the Cisco 12000 Series Internet Router, there is a method to validate whether all line cards within a chassis are supported. Load a software image enabled with Cisco NSF with SSO and then issue the command "show redundancy mode-supported". Each card in the chassis will be listed, and indicate the highest level of system redundancy it supports (RPR, RPR-Plus, Cisco SSO).
To achieve the full benefit of Cisco NSF with SSO, all line cards should support Cisco SSO. Furthermore, depending on platform, Distributed Cisco Express Forwarding (dCEF) must be enabled for the line cards in order for NSF to work.
The correct software image must be loaded on the flash disks of both route processors. Currently, mixing software versions between the active and standby router processors is not supported—even if both software images support Cisco NSF with SSO.
The software boot image in bootflash should also be upgraded and should correspond to the software image being loaded on the RP.
BGP Graceful Restart is configured under the global "router bgp" configuration command. The most basic configuration is "bgp graceful-restart"
Router(config-route)# [no] bgp graceful-restart
Router(config-route)# [no] bgp graceful-restart restart-time n
Router(config-route)# [no] bgp update-delay n
Router(config-route)# [no] bgp graceful-restart stalepath-time n
The "bgp graceful-restart" command must be entered on the Cisco NSF-capable router, and also must be entered on any NSF-aware peer that will be participating in Graceful Restart. Graceful Restart is not enabled by default, and must be explicitly configured on both the Restarting Router and all Peer Routers.
The "bgp graceful-restart restart-time n" is the maximum amount of time that a peer will wait for a reconnection of the TCP session and a new BGP OPEN message following the detection of a failure on the Restarting Router. If the TCP and BGP sessions are not re-established before this timer expires, the BGP session is deemed a failure, and normal BGP recovery procedures take effect. The default value for restart-time is 120 seconds.
The "bgp update-delay n" command may be entered on the Cisco NSF-capable router. The update-delay specifies the time interval- after the first peer has reconnected—during which the restarting router expects to receive all BGP updates and the EOR marker from all of its configured peers. The default value of n is 120 seconds, and n is always measured in seconds. If the restarting router has a large number of peers, each with a large number of updates to be sent, this value may need to be increased from its default value.
The "bgp graceful-restart stalepath-time n" command may be entered on the NSF-aware peer(s) of the restarting router. This timer sets an upper limit on how long the peer will continue to use stale routes for forwarding after it has re-established the BGP session with the restarting router. The default value is 360 seconds. While this should allow an adequate amount of time to allow for complete convergence, on very large networks it may be necessary to increase this value.
5.0 Deployment
There are many different variations on design and deployment of BGP networks. However, to simplify matters, it is easier to think about BGP design in terms of router functionality: What does a particular router need to accomplish given its particular placement within the network? There are three basic types of routers within a BGP network:
•Inter-AS routers run a combination of eBGP and iBGP to connect different autonomous systems. There are many variations to this: edge routers that connect Enterprise customers to the Service Provider network, Internet peering points that connect Service Provider autonomous systems together, edge routers that exist on the boundary of a BGP confederation sub-AS. (see RFC 3065). Yet, the functionality of each of these routers is identical from the Cisco NSF perspective.
•Intra-AS routers exist in the distribution layer or core of an individual AS. These routers run only iBGP and interact only with routers within their own autonomous system. Any knowledge they have of the world outside of their AS is communicated to them via Inter-AS routers
•Route Reflectors (RRs) act as aggregation and distribution points for BGP routing information. Intra-AS routers report BGP routing information to the RRs and receive information from them. RRs increase the scalability of a BGP network by removing the restriction that all iBGP peers must be fully-meshed. The two most common deployment scenarios for RRs are
–Centralized RRs: The Route Reflectors exist at the core of the BGP network, roughly equidistant from all the other routers in the AS. Each router in the AS forms a BGP session with this RR. Frequently, there will be redundant RRs in this configuration
–Distributed RRs: In this design, some subset of routers within an AS will be administratively grouped and have a local RR to which each router will form a BGP session. These RRs subsequently form BGP sessions to other RRs in other regions, or a meshed connection to other RR as well as Intra-AS routers in the core. A typical example of this type of configuration would be a Service Provider that has local RRs in each of its Points of Presence (PoPs). Figure 3 illustrates an eBGP deployment with peers in several different autonomous systems. R1, R2, R3, R4 and R5 are all eBGP peers. Furthermore, the connections R1-to-R4 are multi-homed and are peering to the loopback interface address. R1, R3, and R4 recognize each other as NSF-aware peers because each sends and acknowledges each other's GR capability. On the other hand, the BGP sessions between R1-R2 and R1-R5 are non-NSF, meaning they do not signal or acknowledge the GR capability. The following occurs when R1 goes through a BGP restart:
•R2 detects failure of the TCP/BGP session established over the R1-R2 link and will attempt to route around it. As a result of this re-computation, R2 will take the R2-R3/R3-R1 links to access AS100.
•R1 (the Restarting Router) continues to forward packets destined to AS200 along the R1-to-R2 link, because its CEF table remains intact after switchover. We now have an example of asymmetric routing that can occur when there is a mixture of NSF-aware and non-NSF-aware peers. Although asymmetric routing is not a desirable condition and may result in some packet loss, it is still preferable to the network disruption that would have ensued had R1 completely reinitialized.
•R3 and R4 will not flush routes that they had previously learned from R1. R1 should continue to forward IP packets between R3 and R4, using its last updated CEF Table.
•R3 and R4 should continue to forward IP packets to R1 using their last updated CEF table.
•R5 is non-NSF-aware; as such, it will lose the BGP session to R1 and initialize the BGP session from scratch. R1 will continue to forward packets bound for AS500 through R5, but there is no return path for the traffic. There will be packet loss until R5 successfully reconverges with R1.
•There is an exception to this rule, if R5 has a static default route pointing to R1 as the next-hop. In this case, R5 was only using BGP so it could advertise its routes into the R1 BGP table. The R5 routes are preserved at R1 and R5 only needs a default route, so there should be no packet loss.
Figure 4
NSF—Inter-AS peers
Figure 4 illustrates that R1, RC1 and RC3 are all BGP NSF-aware. Additionally, R1 is NSF-capable—meaning that it supports BGP restartability. Should R1 restart BGP, there should be little-to-no packet loss within AS100. Because R1 is also an Inter-AS router, there may be some packet loss before R2 reconverges to route traffic destined to AS100 via R3. This process was described in the previous section.
Note that there is an important deployment consideration in this scenario. In this topology, it is very common to be running an IGP protocol (i.e.: OSPF or IS-IS) to provide next-hop reachability within AS100. There is interdependence between BGP and the selected IGP protocol. During best-path calculation, BGP knows the IP address of the router advertising certain destination prefixes. However, it relies on the information from the IGP to determine the next-hop to reach that advertising router.
Because BGP Graceful Restart can alter the timing of BGP convergence, situations can potentially occur where BGP is ready to conduct best-path selection, but the IGP has not yet converged. Therefore, some destination prefixes could exist in BGP, but cannot be added to the CEF table because a path to the advertising router has not been calculated by the IGP yet. This could result in packet loss. See section 6.5 of this paper for an example of this situation.
Therefore, it is strongly recommended that NSF for IS-IS or OSPF should be configured in addition to BGP Graceful Restart. To learn more about NSF for both OSPF and IS-IS, review Globally Resilient IP deployment papers: http://www.cisco.com/warp/public/732/Tech/grip/tech.shtml
Figure 5
Cisco NSF—Inter-AS with Route Reflectors
This topology demonstrates that R1, RC1, RC2 and the Route Reflector (RR) are NSF-aware. RR is deployed as a control plane to reduce the requirements for a full iBGP mesh. Thus, it is not in the forwarding path, but forms iBGP peering arrangements with R1, RC1 and RC2. It is also assumed that a flavor of IGP NSF is implemented in this topology.
•When R1 restarts its BGP, it relies on the existing CEF table and continues to forward packets destined to (or through) RC1 and RC2.
•Meanwhile, the only peering arrangement that R1 has is with the Route Reflector. It has no direct peering with RC1 or RC2.
•Because RR is NSF-aware, it masks the fact that R1 has restarted BGP from propagating to RC1 or RC2. RC1 and RC2 continue to forward through R1.
A more interesting variation on this occurs if the Route Reflector is actually NSF-capable and restarted its BGP process.
•When the Route Reflector restarts BGP, all the clients will keep routing information, which was reflected by RR. None of the clients will switch to a backup RR (assuming a backup RR is available).
•Some special considerations must be made when using an NSF-capable Route Reflector
–It is anticipated that a Route Reflector will have more BGP peers and a larger aggregate collection of BGP data than other routers in the AS; due to these conditions, best-path selection may take longer to complete during a switchover.
–Users must balance the requirement to provide uninterrupted packet forwarding and routing stability to the network versus the likelihood of a significant routing change before convergence is complete. During this interim period, Cisco NSF uses the CEF table and not BGP routing information to forward packets.
–Assuming that the decision has been made to use Cisco NSF on the Route Reflector, another configuration adjustment may be required. If it is anticipated that the entire process of reconvergence will exceed 360 seconds, then the default value of the "bgp graceful-restart stalepath-time xxx" may need to be adjusted on all of the peers of the route reflector. The value for stale-path should be adjusted to equal the expected convergence time (in seconds) plus an additional buffer zone of 30-60 seconds to account for variances in convergence time based on changing network conditions.
The decision of whether to use BGP Graceful Restart on a Route Reflector is a complex one and depends largely on network operations. Users must consider the key trade-offs in this decision:
•Is there an alternate availability strategy (i.e.: backup RR); if so, does it provide acceptable failover time?
•How long does it take for the restarting Route Reflector to reconverge, so its Peer Routers can begin to base forwarding decisions on fresh information?
•What is the likelihood that there will be other significant BGP routing changes that occur while the Route Reflector is reconverging?
While these questions are posed in the context of a decision to use Cisco NSF with SSO on a Route Reflector, they are also good general questions that should help in determining where and how to deploy Cisco NSF with SSO.
6.0 Troubleshooting
Because the protocol changes associated with Nonstop Forwarding for BGP are well defined and documented, troubleshooting Cisco NSF with SSO becomes a matter of assuring that the correct protocol exchanges occur at the correct times. The various Cisco IOS Software "show" and "debug" commands have been modified to provide this type of information.
6.1 Validating BGP Graceful Restart
As described in the "Protocol Enhancements" section of this paper, Cisco NSF / BGP begins with, and depends upon, an agreement between the BGP peers that they have implemented the protocol extensions in the IETF Graceful-Restart draft. Examining the output of the show ip bgp neighbor X.X.X.X command can validate this negotiation.
Figure 6
sh ip bgp neighbor
Figure 6 illustrates that the show ip bgp neighbor command now documents the successful negotiation of the Graceful Restart Capability. In order for Cisco NSF to work, the Restarting Router and its peer must each have implemented the BGP protocol enhancements. It is thus vital that the capability be both "advertised and received". This output should be the same, regardless of whether it is queried on the NSF-capable router or its NSF-aware peer. Note that the advertisement of the BGP graceful-restart capability does not specify that a particular router is NSF-capable; it only signifies that the router understands the protocol changes specified in the IETF draft.
The value of the Remote Restart timer is listed. Although the local and remote BGP peers are permitted to have different restart timers, it is wise to be suspicious of a significant variance between the values. The restart timer sets an upper bound on the amount of time it will take to re-establish the TCP connection and renegotiate the graceful-restart capability in a new BGP OPEN message. Setting this value too low on either of the peers makes it unlikely that the OPEN can be re-established in time, and will lead to the failure of the Cisco NSF process.4 BGP will revert to "normal" recovery procedures in this type of case.
Refer to the following line, "Address Families preserved by peer" for information about what Address Families (i.e.: types of traffic) for which the Peer Router can maintain forwarding state during the progression of an RP switchover. IPv4 is currently the only supported Address Family. Future extensions to Cisco NSF will add support for other Address Families (or Subsequent Address Families), including Multicast, IPv6 and MPLS when these protocols are made NSF-capable.
6.2 Validating Stale Routes on an NSF-aware Peer
One of the agreed-upon capabilities of the NSF-aware Peer Router is the use of "stale" routes (the last known good routes before the restart) to continue packet forwarding while the Restarting Router is dynamically reacquiring routing information in the background (see previous section for details). This capability can be queried with the show ip bgp command.
Figure 7
sh ip bgp on NSF-aware peer
In Figure 7, the NSF-aware router has marked the routes for the 170, 180 and 190 networks as "stale" (as signified by the capital "S" beside the route itself). This behavior is expected. Only the routes reachable through the Restarting Router will be "stale". All other routes should appear as normal.
If the specific routes through the Restarting Router are not marked "S", one of several explanations might apply:
•If the route is missing from sh ip bgp, then Cisco NSF has failed for some reason and "normal" BGP reconvergence is taking place. Possible causes:
–The Peer Router is not NSF-aware. See 6.1, above
–The restart timer has expired
–The stale-path timer has expired
•If the route is present, but is not marked "S", then one of the following might apply:
–The Cisco NSF-aware router has not detected the restart on the NSF-capable router yet
–The Cisco NSF process is complete, and the NSF-aware router has replaced the stale route with a fresh one
–If the next-hop is not pointing to the Restarting Router, then Cisco NSF has failed and "normal" BGP recovery procedures have occurred, and the network has reconverged around the Restarting Router
6.3 Debugging Output on the NSF-aware Peer
As with almost all Cisco-supported protocols, BGP NSF supports debugging commands that can monitor the progression of events as a route processor switchover occurs. Table 2 illustrates the output of the commands debug ip bgp events and debug ip bgp updates on the NSF-aware peer of a Restarting Router at the time of a switchover.
...(略)
5.1 Deployment Scenarios
Figure 3
NSF Deployment —Inter-AS Peers
Comments