In Part 1 of this 2 part blog series, we went into details about the environment, setup, and control plane. In this blog we are going to continue our advanced routing and DMVPN fun but will be focusing on the forwarding plane side of things.
The forwarding plane is pretty where things are going to start getting cool and neat with DMVPN, NHRP shortcut switching and Next-Hop-Override. Gets even cooler and neater with 2547oDMVPN, NHRP shortcut switching and Next-Hop-Override.
Prior to continuing to read this, if you have not reviewed Part 1 I would recommended just going thru it to get your bearings on the environment we will now be assuming has already been explained and we can move past that point. The previous post is https://www.networkingwithfish.com/advanced-dmvpn-routing-fun-in-the-lab-part-1-the-control-plane/
One additional comment before we begin – I’m not going to, in this article go into the details of which commands we had configured on the tunnel interfaces in the previous post to enable all this fun stuff. So if NHRP short cutting is new to you and you would like to read more on it I would suggest maybe something like http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_nhrp/configuration/xe-3s/nhrp-xe-3s-book/nhrp-switch-enhancemts-dmvpn.html
Ready to play in the forwarding plane world with DMVPN, NHRP shortcuts and NHO? Well instead of jumping right into the 2547oDMVPN world and seeing the “more fun” there….. Let’s begin in the tunnel 11 world where there is no MPLS.
DMVPN & NHRP Shortcuts in Tunnel 11 Environment – No MPLS
So here we are, our nice simple DMVPN world with no MPLS.
Let’s go to deer and do a traceroute between the LANs of the 2 spokes.
So our path from deer’s 11.11.92.1 is thru thewall (10.99.11.2) and then down to charlie10 (10.99.11.91). So a “hairpin” up to the hub in order to talk ‘spoke-to-spoke’. This may be what you want – which is great cause this is what happens by default. But this may not be what you want. You might want those “NHRP shortcuts” I was referring to.
Hmmmmm…. wait. I’m already all configured for those shortcuts. Let’s try that traceroute again. Up arrow and press return.
Wait? What?
Question: Why didn’t it go 2 hops this time? Why didn’t it go thru thewall (10.99.11.2) again?
Answer: Because we have a shortcut now.
Got the general idea? Let’s go over to the 2547oDMVPN environment see what happens.
DMVPN & NHRP Shortcuts in Tunnel 2 (2547oDMVPN) Environment
As a reminder the Tunnel 2 environment is 2547oDMVPN also known as MPLS VPN over DMVPN. So the NHRP short cutting is going to actually not just involve IP next hops like we saw in the tunnel 11 environment, but ALSO the BGP vpnv4 labels. To try to “see” this more clearly …. to “see” what is happening let’s
- Narrow the view to just 2 of the Spokes (sonic and parrot) and to just the VRF HR subnets they have
- Turn OFF crypto so we can sniff in the middle!
Before we send ANY traffic between the 2 spokes… which will, as we saw in the above with tunnel 11, trigger our shortcuts….. Let us FIRST look at “steady state” in the control plane. Ready? This may hurt a little if you aren’t used to following labels.
- Sonic is originating the BGP NLRI for 10.38.11.0. Running show bgp vpnv4 unicast vrf HR 10.38.11.0 on sonic shows that it is going to advertise it’s eBGP vpnv4 peer, thewall, to use label 142 to get to 10.38.11.0
- Parrot is originating the BGP NLRI for 10.31.11.0. Running show bgp vpnv4 unicast vrf HR 10.31.11.0 on parrot shows that it is going to advertise it’s eBGP vpnv4 peer, thewall, to use label 21 to get to 10.31.11.0
To put this another way let’s add these to our diagram.
To help “see” things better we are going to change our coloring scheme.
As we can see from the above
- sonic is originating and advertising its local LAN subnet (10.38.11.0 /24) + a VPNv4 label (142 ) to its eBGP vpnv4 peer, thewall at 10.99.2.2.
- parrot is originating and advertising its local LAN subnet (10.31.11.0 /24) + a VPNv4 label (21) to its eBGP vpnv4 peer, thewall at 10.99.2.2.
Let’s just stop right there…. with just those two things happening and talk about what thewall should see.
So we see that sonic is sending OUT the label of 142 for others to use when they come IN to sonic wanting to get to that subnet. Sonic is sending “out” instructions to it’s eBGP vpnv4 peer, thewall, for what label thewall should put on a packet destined to 10.38.11.0.
So, based on the “instructions” received from sonic and parrot, thewall should view that if it wants to send traffic
- out to 10.38.11.0 then add label 142 to the packet
- out to 10.31.11.0 then add label 21 to the packet
NEXT: TheWall Advertises the Prefixes out via eBGP VPNv4
Following labels, when you are not used to it, can be a little tricky at first. So before we to into the vpnv4 addl complexity of this… Let’s review just good old fashion eBGP IPv4.
eBGP IPv4 Review
If thewall was eBGP IPv4 peered with sonic and it learned 10.38.11.0 via next hop 10.99.2.38. When thewall went to advertise this prefix out to parrot via eBGP, thewall would modify the 10.38.11.0 advertisement. Agreed? The most obvious modifications are that it would, of course, modify the next-hop to itself (10.99.2.2) as well as it would also add it’s AS (100).
With that in mind let’s now return to this all being VPNv4 prefixes we are dealing with. So now thewall is going to modify a little more. It is also going to be creating its own labels which help tell it, thewall, what to with a packet when it comes in to it.
I have gone into thewall and looked at the output to see what labels it generated. They are as below.
I decided to just go with one picture and just 2 arrows. We will use this as an example and how you can see the other way around. So sonic advertises with the green arrow. thewall receives this and notes that if it has packets to send to 10.38.11.0 to use label 142 going out towards sonic. In the red arrow you can see that when thewall is advertising the vpnv4 prefix it learned from sonic, it modifies it some. Next hop changed and set to itself, add its AS, plus create its own label (43). parrot’s “instructions” if it wants to send a packet to 10.38.11.0 in VRF HR, is to take the packet, push a MPLS label of 43 onto it and send it out to next-hop 10.99.2.2 (thewall).
Time for Shortcuts and Sniffing!
We are going to go to parrot and initiate an extended ping between the 2 spoke LAN interfaces.
Ready?
1) On parrot do a show ip route vrf HR 10.38.11.0 and see what the routing table is instructing parrot to do it it wants to get to 10.38.11.0
I have outlined the instructions in red as these are the instructions that were modified and sent by thewall.
- Next-hop: thewall
- Label: 43
2) On parrot do a show ip nhrp to make sure no shortcuts are currently installed.
Nope… no shortcuts to get to 10.38.11.0.
3) On parrot extended ping between spokes’ LAN interfaces.
4) On parrot type show ip nhrp to see what the shortcuts look like.
Two shortcuts were created in relationship to sonic.
- 10.38.11.0 via 10.99.2.38 w/ NBMA address 10.24.111.1 and flags: router, rib, nho
- 10.99.2.38 w/ NBMA address 10.24.11.1 and flags: router, nhop, rib.
5) On parrot look at the RIB for VRF HR
But wait? Why is 10.38.11.0 still showing up as via 10.99.2.2 (thewall)? Where is the shortcut? Hint: See that “%”?
6) On parrot type the command as above but type “next-hop-overrides” at the end (show ip route vrf HR next-hop-override)
Notice 3 things
- % flag indicates that this RIB entry has a next hop override (nho)
- Control plane entry for 10.38.11.0 via 10.99.2.2
- NHO (next hop override) entry for 10.38.11.0 via 10.99.2.38
So it looks like parrot has switched from hair pinning thru thewall to get to 10.38.11.0 behind sonic…. to going directly to sonic via NHO. But what about the BGP vpnv4 labels?
Have we really gotten to the shortcut we want for the forwarding table? The one that doesn’t have the BGP vpnv4 labels that were inserted by thewall? Remember? We didn’t want all this where parrot pushes a label of 43 on a packet to 10.38.11.0.
We just want this. Where parrot pushes a label of 142 on a packet to 10.38.11.0.
7) On parrot type the command show ip route vrf HR next-hop-override 10.38.11.0
Looks good. Looks promising. What does the CEF look like?
8) On parrot type the command show ip cef vrf HR 10.38.11.0
Nice! Looking nice. Everything seems to point to parrot using label 142 with next hop 10.99.2.38 to get to 10.38.11.0.
9) Final test! With the shortcuts in place let’s so a sniffer capture of the extended ping.
We should only see labels 142 and 21 in the capture if the 2547oDMVPN shortcuts have been setup on both sides.
Eh, Voila!
Hope you had fun in the lab again. 🙂
One additional note. If you end up playing with this in your environment and you try to ping with a /32 and you see an issue you might be hitting.
CSCut61985: MPLSoDMVPN; Ping to spoke int. fails if dest. mask is /32
Test using an interface with a mask other than /32 or from nodes that are behind the spokes.
Categories: BGP, DMVPN, Fun in the Lab, Routing
Dear Fish – I really loved this 2547oDMVPN dive. I haven’t found anything out there that shows phase 4 yet with dynamic spoke to spoke tunnels. This was a great find. Anyhow, I was wondering if there were configuration examples you would share?
Thanks Fish, always a pleasure to read you going step by step and dissecting the technology like this!
Thanks for the article!
Small correction:
“eBGP IPv4 review:
If thewall was eBGP IPv4 peered with sonic and it learned 10.38.11.0 via next hop 10.99.2.38. When thewall went to advertise this prefix out to parrot via eBGP, thewall would modify the 10.38.11.0 advertisement. Agreed? The most obvious modifications are that it would, of course, modify the next-hop to itself (10.99.2.2) as well as it would also add it’s AS (100).”
To be 100% correct, this is not exactly true. By default, if there was eBGP for IPv4 thewall wouldn’t modify next hop to itself, because sonic 10.99.2.38/24 is on the same network as parrot 10.99.2.31. This is called third-party next hop and this is why you don’t need manual next-hop modification to make Phase 2 work if eBGP is used for routing over DMVPN.