Time for some serious geeky fun in the lab! In this blog we are going to have a little advanced routing and DMVPN fun and focus and follow the control plane side of things. In the next blog we will follow the forwarding plane.
Ready?
Play Time!
What we are going to do is to connect VRF end points together so the devices in those VRFs can communicate with each other via IP. The end points will be connected to our network via one of 3 ways
- HeadQuarters/Hub:
- VRF HR (in green circle) is connected to an interface on a Cat6K named Pulse. This interface is configured to be in VRF HR.
- VRF PC (in peach circle) is connected to an interface on Pulse also. This interface is configured to be in VRF PC.
- thewall and pulse are iBGP vpnv4 peers with each other.
- Branches off of Tunnel 2:
- VRF HR (in green circles) are connected to interfaces on each of the 3 branches (sonic, parrot, whiteduck). These interfaces are configured to be in VRF HR on each of these routers.
- VRF PC (in peach circles) are connected to interfaces on each of the 3 branches (sonic, parrot, whiteduck). These interfaces are configured to be in VRF PC on each of these routers.
- Branches off of Tunnel 11: The branches themselves are not VRF away. In their configurations, all of the interfaces are in the global routing table. The VRF definition for VRF HR is actually up at the DMVPN head end (thewall) on the tunnel 11 interface.
Tunnel #2 Environment
This is a DMVPN tunnel with the head end up on the thewall. The subnet for this tunnel is 10.99.2.0/24. thewall is 4th octet .2, sonic .38, parrot .31, and whiteduck .32 as the diagram shows. The DMVPN head end is in BGP AS 100 and each of the 3 spokes are in different BGP AS as the diagram illustrates. Each spoke will have an eBGP VPNv4 neighbor with thewall. These eBGP neighbors will be using their Tunnel2 interface IP addresses for their BGP endpoints. In other words sonic and thewall will have an eBGP peering relationship between 10.99.2.38 and 10.99.2.2.
BGP vpnv4 means that our blue boxes are going to be sharing vpn “labels” with each other over their BGP vpnv4 peer connection. But this is 2547oDMVPN also known as MPLS VPN over DMVPN. The DMVPN tunnel will get us thru the middle internet part. We don’t need additional labels in front of the VPNv4 BGP labels as we might in a traditional MPLS L3VPN environment with the traditional “P”s in the middle. Traffic that goes out thewall destined for a green circle behind sonic will do the lookup for the BGP vpnv4 label sonic told thewall to use to get to that IP in the green circle behind sonic. thewall will push this label on but then toss the mGRE (and crypto in our environment) on and send it out. So no need for another label cause once it goes into the tunnel destined for sonic as the endpoint… that is all the “internet” needs to know. Once the packet gets to sonic it will look at the vpn label that thewall put into the packet. This will be used to instruct sonic as to where this packet should go.
Tunnel #11 Environment
This is a DMVPN tunnel with the head end up on the thewall. The subnet for this tunnel is 10.99.11.0/24. thewall is 4th octet .2, deer .92, and charlie10 .91 as the diagram shows. The DMVPN head end is in BGP AS 100 and each of the 2 spokes are in different BGP AS as the diagram illustrates. Each spoke will have an eBGP IPv4 neighbor with thewall. These eBGP neighbors will be using their Tunnel11 interface IP addresses for their BGP endpoints. In other words deer and thewall will have an eBGP peering relationship between 10.99.11.92 and 10.99.11.2.
Deer and charlie10 are NOT aware they are viewed as in VRF HR by thewall via the configuration of the Tunnel11 interface on the wall as being defined to be in VRF HR. As such, on the spokes, the eBGP IPv4 peer will be viewed as in the global routing table. However, on thewall these eBGP peers will be found under the BGP address-family ipv4 vrf HR subsection as a typical CE is configured.
TheWall: Config Snippets
Let’s look at some of the configuration snippets from thewall.
Above we see the Tunnel 2 configuration up on thewall. Pretty much probably looks to you like a basic DMVPN hub side tunnel definition except for 2 things: the 2 mpls commands and the 3 nhrp map group commands. I’m not going to go into detail on these. Suffice it to say that we are doing 2547oDMVPN so it probably isn’t a surprise that we need some indication somewhere that we would maybe like to be able to process packets with mpls labels on the tunnel interface. The 3 nhrp map group commands are to support what I think is a wicked cool thing with DMVPN – per tunnel QoS.
Below we see the Tunnel 11 configuration on thewall. This one looks a lot more like the typical DMVPN head end tunnel I’m used to creating. Well, okay… except for being part of a VRF.
The BGP configs in thewall show the eBGP vpnv4 peering in the global routing table to the spokes off of Tunnel2 as well as the iBGP IPv4 peering in VRF HR to the spokes off of Tunnel 11.
VPNv4 Peers (PE-PE)
VPNv4 BGP peering definitions, like this, are what you would typically see in PE (provider edge) type boxes in an MPLS L3VPN environment. If you have played with PE to PE VPNv4 peering in a “basic level” MPLS L3VPN lab environment you may notice two things that are probably different than “your norm”.
- These are eBGP vpnv4 vs iBGP vpnv4
- Update-source is the Tunnel interface instead of a Loopback
eBGP vs iBGP – your choice, really, if you own all the routers. In this lab play environment I’m having it that the customers own all of these routers (well of course except for our 2 internet clouds). At that point this is really just a design choice for you.
update-source tunnel instead of loopback – I want the BGP peering to be with the tunnel endpoints.
IPv4 eBGP Peering within a VRF (PE-CE)
The configs that you see in thewall for the spokes connected via Tunnel 11 are really just the “traditional” PE:CE (provider edge:customer edge) eBGP IPv4 peering. On the PE side of things (thewall) the neighbors are defined under the “address-family ipv4 vrf” section for that specific VRF. On the CE side of things (e.g. deer / charlie10) the BGP peering is not done under the VRF section but in the “global” section. Why? Remember – the CE doesn’t need to have any VRF definitions in it.
Tunnel 2 Spoke/PE Config Snippets (Sonic)
As you can see from the tunnel 2 configs below taken from sonic, the 2 mpls commands show up again. Same exact reasons for why they are on this end of the tunnel interface for why they are at the head end. So shouldn’t really be a surprise. Also notice the lack of an nhrp map group styled command on the tunnel 2 interface. That is because sonic is not configured to request of the DMVPN headend (thewall) to have a per-tunnel qos policy applied on traffic to it.
Notice anything else? Something new and different than either of the two configs we looked at up on thewall? “service-insertion waas” During this “lab playing” waas was in the mix so I just left the config in there.
Let’s take a look at a tunnel 2 spoke BGP config.
Well the VPNv4 looks pretty identical to thewall. Notice anything different in the blue section? 🙂
…. Yup, sonic has an IPv4 eBGP peer neighbor statement for thewall and in the snippet above thewall didn’t even have a non VRF IPv4 address-family section. Well… that is only missing up in thewall config because I snipped it out to make things clearer. For what we are doing in this blog, the IPv4 eBGP peering over tunnel 2 is not relevant.
Notice those network commands under VRF HR and VRF PC IPv4 address-families? Yes, even in the lab I practice what I preach – use network statements even in a lab environment as simple as this instead of giving into the temptation of using redistribute connected instead.
Tunnel 11 Spoke/CE Config Snippets (Deer)
Just to be complete here is a sample of the tunnel 11 interface of one of those spokes. Pretty much the basic DMVPN tunnel config I tend to use in the lab. Well – except I tend not to add tunnel protection cause I just love sniffer tracing the middle to learn and see what is happening.
As I mentioned earlier the spokes in the tunnel 11 are not “VRF HR”-aware.
Checking the Control Plane
Prior to checking BGP tables and routing tables let’s first step thru what we believe we should see prior to going to the CLI.
TheWall should have a total of 6 BGP peers and 10 learned prefixes.
- 1 iBGP vpnv4 neighbor with Pulse (BGP peer IP address 123.123.123.14) where it should be learning 2 vpnv4 prefixes. 10.11.11.0/24 for VRF HR and 10.12.12.0/24 for VRF PC.
- 3 eBGP vpnv4 neighbors over tunnel 2.
- 10.99.2.38 (sonic) advertising 2 vpnv4 prefixes 10.38.11.0/24 for VRF HR and 10.38.12.0/24 for VRF PC.
- 10.99.2.31 (parrot) advertising 2 vpnv4 prefixes. 10.31.11.0/24 for VRF HR and 10.31.12.0/24 for VRF PC.
- 10.99.2.32 (whiteduck) advertising 2 vpnv4 prefixes. 10.32.11.0/24 for VRF HR and 10.32.12.0/24 for VRF PC.
- 2 eBGP ipv4 neighbors over tunnel 11 in VRF HR.
- 10.99.11.92 (deer) advertising 1 ipv4 prefix 11.11.92.0/24
- 10.99.11.91 (charlie10) advertising 1 ipv4 prefix 11.11.91.0/24
With that in mind lets go to the CLI and see what we see on thewall via the command show bgp vpnv4 unicast all summary
We expected to see 6 BGP peers over on thewall and a total of 10 prefixes. Eh, voila!
Let’s drill down deeper a little in the VRF HR world. I expect to see 6 of those prefixes in the vrf HR BGP table. So let’s run the command show bgp vpnv4 unicast vrf HR. So we see our 6 that are all green and in VRF HR.
Ready to drill down deeper? I’m going to run a command here and I want you to look at it and see if you notice a pattern with some of the prefixes versus the others.
Notice anything? Look over to the far right column. The one labeled “In label/out label”. Focus on the out label.
- All 4 of the prefixes learned via VPNv4 BGP peers have an “out label”
- Both of the prefixes learned via the IPv4 eBGP peers over tunnel 11 have no “out label”
Question #1: What is the “out label” in this situation?
It is the label that the peer created and sent to you to tell you what label to use if you want to get to that prefix in that VRF. Let’s take our friend Sonic (prefix 10.38.11.0/24 sent from peer 10.99.2.38) for example.
The above is a detailed look at sonic’s BGP table for the prefix 10.38.11.0 in VRF HR. sonic is the originator of this BGP advertisement that it created and is advertising to its eBGP vpnv4 peer thewall. So that means sonic is the one who created that label 142. sonic is “telling” thewall that if it has a packet to send to 10.38.11.0 in VRF HR put an mpls label 142 on the packet and send it.
Why all the fuss with the labels? Lots of reasons. Simplest thing to ask you back is what if sonic actually had the following to advertise to thewall – 10.38.11.0 for VRF HR, 10.38.11.0 for VRF PC ? Yes. What if the same subnet lived in both VRF HR and VRF PC?
Question #2: Why aren’t the DMVPN spokes off of tunnel 11 advertising labels?
They are not vpnv4 peers so they are not configured to send labels.
Control Plane All Done!
Yup! Control plane portion is all done. Everyone can ping around within their own VRFs and all is good with the network. Next blog – the forwarding plane. 🙂 This gets neato with DMVPN and NHO (next-hop-override) so that the spokes can talk with each other via a shortcut instead of always having to go to the hub just to talk with another spoke.
One additional note. If you end up playing with this in your environment and you try to ping with a /32 and you see an issue you might be hitting.
CSCut61985: MPLSoDMVPN; Ping to spoke int. fails if dest. mask is /32
Test using an interface with a mask other than /32 or from nodes that are behind the spokes.
Categories: BGP, DMVPN, Fun in the Lab
Nice article! We run something similar but since we own the network all the way out to the WAN site (well we do have providers giving us L2 metro-e service) we run all our WAN routers as PE’s that just peer directly with our RR’s. Nice to see it from a PE/CE perspective as well.
Eric, do you use DMVPN?
Yeah. We have a dual hub, encrypted 2547oDMVPN architecture. The hubs are asr 1006’s and the spokes are currently 3945’s with new sites moving to the 4400 series routers. We actually labbed it up at the CPOC about 2 years ago and we scaled it out to about 600 sites with no issues. We currently have only 30 sites or so on the DMVPN but we are ramping up migrating our other 150 or so sites over this year.
Denise – Can you post the configuration examples that get you to dynamic spoke to spoke directly?
Is it a requirement for 2547oDMVPN to have the same device for the hub and route reflector?
Don’t know if it is a requirement. It’s what I have done ever since 2547oDMVPN support came out. I’m at CiscoLive Berlin and happen to have a session with “Mr. DMVPN” (Mike Sullenberger) tomorrow. I’ll check in with him 🙂
After some more research and testing I’ve discovered that the route reflector does not have to be a hub. I have successfully gotten spoke-to-spoke working with a separate RR, however, the NHRP and NHO’s don’t happen because the ‘next-hop-self all’ command is not used on the RR.
I’ve been trying to test using iBGP where the source interface is NOT the tunnel interface and so far I have not gotten this to work. Perhaps that is a requirement?
Woot! What URL did you find that at?