Time for a little more fun in the Network Detective series! Today’s “Network Detective Ride-Along” will bring us into the config setup of someone very new to MPLS L3VPNs.
As we go along remember the Network Detective Mantra “Be Prepared, Find the Suspects, Question the Suspects, Improve”
Ready to begin? Grab your Network Detective badge and something to take notes with. In this part 1 blog we will focus on the BGP and the routing. 🙂
The Symptom
So the symptom is pretty basic. IP addresses that exist on the PEs inside VRF Customer1 cannot ping each other.
As a Network Detective we
- Gather the Facts
- Collect the Clues
- Interview the Witnesses
- Question the Suspects
- Find out Who Done It
What is the difference between a “Fact” and a “Clue”? Knowledge. Knowledge that you need to build in the “Be Prepared: Knowledge is Key” portion. Obviously, we have to have knowledge of how this is all supposed to work, so we can figure out what is not “right”.
First Pass: Pick a PE and Check the Basics
Like any good Detective, we need to start with the basics. I usually pick one device and start the “interview process” there. Once I have finished the “first pass” interviews with that device (witness) I will make a decision based on what I learned from the interview to decide where to go next. Letting the evidence guide me.
We will start with R1 and I already know the basics I’ll be starting with.
Routing Basics:
- Existence: First we are going to check on the existence (or lack thereof) of 14.14.14.1 and 14.14.14.3 in the routing table for the VRF Customer1
- Advertise: Is R1 configured to even advertise 14.14.14.1 to R3 via BGP vpnv4? If so with what route-target is R1 exporting to R3?
- Receive: Is R1 receiving 14.14.14.3 via its BGP vpnv4 peer with R3?
Let’s go interview our first witness, R1 and check first on just the existence question
R1: Do Both Routes Exist in the VRF Routing Table?
R1#sh ip route vrf Customer1
Routing Table: Customer1
<snip>
14.0.0.0/32 is subnetted, 1 subnets
C 14.14.14.1 is directly connected, Loopback14
R1#
Interview Result: 14.14.14.1 is in the VRF for Customer1 but not 14.14.14.3
Now let’s be honest about something here. Admittedly we tend to find ourselves sometimes following the breadcrumbs and forgetting some of the questions we were originally planning to ask. Right now you are probably wondering if R1 is even receiving 14.14.14.3 via BGP from R3. This happens to us. At least it happens to me all the time. I assume I have found the answer and I start following that path before collecting more facts and clues. What do I mean by this? Basically I’ll tell you now that R1 is NOT advertising 14.14.14.1. But likely we were about to skip over that because we think that the missing 14.14.14.3 is the only issue. I’m not saying this is “bad”. It just “is”. So just be aware is all I’m saying. Admittedly that is why I find taking notes helpful while I’m troubleshooting a “who done it”
R1: Is 14.14.14.1 Being Advertised Via BGP vpnv4?
R1#sh ip bgp vpnv4 vrf Customer1 R1#
Interview Result: 14.14.14.1 is NOT being advertised in vrf Customer1 to R3 via BGP vpnv4. 🙂 For that matter 14.14.14.3 is not being received by R1 from R3 either. But let’s address one at a time.
What usually happens now? It’s possible I would have gone into R1 and found nothing. So I would still be in the “first pass” phase and “hunting for clues” But once I have found some clues to follow… I can start the move to the Second Pass – deepening the interview questions based on the initial interview responses.
Second Pass: Follow The Clues
At this point we have 2 clues as to why the ping may be failing.
- R1 is not advertising 14.14.14.1 to R3 via BGP vpnv4
- R1 is not showing it is receiving 14.14.14.3 via R3 in BGP vpnv4
Where do I usually begin? For me everything is in a non production environment but time is still of the essence. So I tend to stay with the first “witness” that caused me to move out of the “hunting for clues” 1st pass to the “follow the clues” 2nd pass. Admittedly if I find something that looks to be “easy” and “obvious” (like R1 not advertising to R3) I will dig into that and likely try to fix that first. Of course looks CAN be deceiving. LOL. And I have been burned quite often by this. This is also why taking notes as you go along is so crucial. Seriously. It really is.
Let’s start with R1 since we know that 14.14.14.1 while in the routing table for vrf Customer1 is not making it to the BGP table to get advertised.
Why isn’t R1 Advertising 14.14.14.1?
Knowledge is key. We need to know that for R1 to originate a prefix and advertise it via BGP it must be configured to do so.
vrf definition Customer1 rd 14:14 ! address-family ipv4 route-target export 14:14 route-target import 14:14 exit-address-family ! router bgp 100 bgp log-neighbor-changes neighbor 10.100.100.3 remote-as 100 neighbor 10.100.100.3 update-source Loopback0 ! address-family vpnv4 neighbor 10.100.100.3 activate exit-address-family !
R1 is not originating the prefix 14.14.14.1 because it is not configure to do so. Let’s add some configs to fix that.
R1#config t Enter configuration commands, one per line. End with CNTL/Z. R1(config)#router bgp 100 R1(config-router)#address-family ipv4 vrf Customer1 R1(config-router-af)#redistribute connected R1(config-router-af)#end R1#
Did that fix it? Let’s check.
R1#sh ip bgp vpnv4 vrf Customer1
<snip>
Route Distinguisher: 14:14 (default for vrf Customer1)
*> 14.14.14.1/32 0.0.0.0 0 32768 ?
R1#
Advertise – R1 now looks to be advertising the subnet.
Why isn’t R1 Receiving 14.14.14.3?
Okay… so I’ll be honest. I planted a harder to find one here. Let’s go to the sniffer trace of the BGP coming up between R1 and R3.
File: bgp_startup.pcap
Line 38… looks like an update message from R3 to R1. If you download the pcap and look you will see R3 actually IS advertising 14.14.14.3 to R1. If we take an even closer look and focus just on the path attributes associated with R3‘s advertisement of 14.14.14.3 to R1 we will notice something very important.
Again… knowledge is key. What you need to know when troubleshooting this is that the route-targets (RTs), so critical to how VPNv4 works, is sent as an extended community. And R3 is not sending extended communities. Note that the sending of communities – whether standard, extended, or both, is not enabled by default. In fact not only is R3 not configured correctly… neither is R1. Look at the sniffer trace yourself and you will see.
For BGP vpnv4 to work each peer must be configured to send extended community information to its peer. It is inside of that the very important route-targets are advertised.
The correct BGP config for R1 is as below. R3 has also been updated with the equivalent for it.
Sniffer Trace After Config Change: bgp_more.pcap
router bgp 100
bgp log-neighbor-changes
neighbor 10.100.100.3 remote-as 100
neighbor 10.100.100.3 update-source Loopback0
!
address-family vpnv4
neighbor 10.100.100.3 activate
neighbor 10.100.100.3 send-community extended
exit-address-family
!
address-family ipv4 vrf Customer1
redistribute connected
exit-address-family
We update both routers …. and now let’s go thru our checklist again.
Back to the Checklist
- Existence: First we are going to check on the existence (or lack thereof) of 14.14.14.1 and 14.14.14.3 in the routing table for the VRF Customer1?
- Advertise: Is R1 configured to even advertise 14.14.14.1 to R3 via BGP vpnv4? If so with what route-target is R1 exporting to R3?
- Receive: Is R1 receiving 14.14.14.3 via its BGP vpnv4 peer with R3?
Existence? – Yes
R1#show ip route vrf Customer1
<snip>
C 14.14.14.1 is directly connected, Loopback14
B 14.14.14.3 [200/0] via 10.100.100.3, 00:07:50
R1#
Advertise? – Yes
R1#sh ip bgp vpnv4 all 14.14.14.1 BGP routing table entry for 14:14:14.14.14.1/32, version 2 Paths: (1 available, best #1, table Customer1) Advertised to update-groups: 10 <snip> 0.0.0.0 (via vrf Customer1) from 0.0.0.0 (10.100.100.1) Origin incomplete, metric 0, localpref 100, weight 32768, valid, sourced, best Extended Community: RT:14:14 mpls labels in/out 16/nolabel(Customer1) rx pathid: 0, tx pathid: 0x0
Receive? – Yes
R1#sh ip bgp vpnv4 all 14.14.14.3 BGP routing table entry for 14:14:14.14.14.3/32, version 7 Paths: (1 available, best #1, table Customer1) <snip> 10.100.100.3 (metric 3) (via default) from 10.100.100.3 (10.100.100.3) Origin incomplete, metric 0, localpref 100, valid, internal, best Extended Community: RT:14:14 mpls labels in/out nolabel/19 rx pathid: 0, tx pathid: 0x0
Test The Ping
So can we ping? Okay… we can’t.
HOWEVER… I promise you the BGP is perfect now. Thoughts? 🙂
Time for part 2!
Troubleshooting Basic MPLS L3VPN – Part 2 – MPLS
Categories: BGP, Fun in the Lab, MPLS, Network Detective, Troubleshooting
“Admittedly we tend to find ourselves sometimes following the breadcrumbs and forgetting some of the questions we were originally planning to ask.”
This is SO true, it happens to me all the time, once you fix the problem you thought and it doesn’t work you have to go back and repeat the process, but its part of the fun, sometimes you fix the problem right away and sometimes you stumble and you learn in the process
Could NOT agree more. 🙂