Cart
This is yet another post in our interactive troubleshooting series. I have a suspicion this one will be quite fun. Please, use the comments section of the post to post troubleshooting questions and I will update the post as we go on.
R2 and R6 Reachability (3 points)
R2 and R6 are unable to communicate. Re-establish the communication using the following guidelines and restrictions.
- No new IP addresses are allowed to be added.
- Pre-configured OSPF area membership cannot be changed.
- Introduction of any new routing protocols or routing protocol instances is not allowed.
- No static routes are allowed.
- No Policy-based routing is allowed.
Verification
To successfully solve the ticket, the following output must match:
R2:
R2#ping 192.168.46.6 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.46.6, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms R2#show ip route ospf O 192.168.46.0/24 [110/129] via 192.168.45.5, 00:15:25, Serial0/2/0 192.168.45.0/24 is variably subnetted, 2 subnets, 2 masks O 192.168.45.0/24 [110/128] via 192.168.45.5, 00:29:34, Serial0/2/0
R6:
R6#ping 192.168.2.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.2.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms R6#show ip route ospf O IA 192.168.45.0/24 [110/65] via 192.168.46.4, 2d03h, FastEthernet0/1 O 192.168.2.0/24 [110/1066] via 192.168.46.4, 00:16:17, FastEthernet0/1
Student Troubleshooting
Ball is in your hands. I will update the post as we receive the comments with your instructions… Please, refrain from asking for full “show running-config” outputs. I will provide them if asked, but try to be more specific. Good luck!
R2:
R2#show ip ospf interface brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Gi0/1 1 0 192.168.2.2/24 1 DR 0/0
Se0/2/0 1 254 Unnumbered Gi0/1 64 P2P 1/1
R4:
R4#show ip route ospf
R4:
R4#show ip ospf database
OSPF Router with ID (0.0.0.4) (Process ID 1)
Router Link States (Area 0)
Link ID ADV Router Age Seq# Checksum Link count
0.0.0.4 0.0.0.4 974 0x80000003 0x006B73 2
0.0.0.6 0.0.0.6 973 0x80000002 0x00528A 2
Summary Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
192.168.45.0 0.0.0.4 974 0x80000002 0x004F12
Router Link States (Area 254)
Link ID ADV Router Age Seq# Checksum Link count
0.0.0.2 0.0.0.2 1114 0x80000004 0x00CE11 1
0.0.0.4 0.0.0.4 974 0x80000003 0x001B48 2
0.0.0.5 0.0.0.5 979 0x80000004 0x001DE9 3
Summary Net Link States (Area 254)
Link ID ADV Router Age Seq# Checksum
192.168.2.0 0.0.0.2 1114 0x80000002 0x00BD10
192.168.46.0 0.0.0.4 976 0x80000002 0x00CBD3
R5:
R5#show ip ospf int br Interface PID Area IP Address/Mask Cost State Nbrs F/C Se0/2/0 1 254 Unnumbered Se0/0/0 64 P2P 1/1 Se0/0/0 1 254 192.168.45.5/24 64 P2P 1/1 R5#show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 0.0.0.2 0 FULL/ - 00:00:33 192.168.2.2 Serial0/2/0 0.0.0.4 0 FULL/ - 00:00:31 192.168.45.4 Serial0/0/0
R2:
R2#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
0.0.0.5 0 FULL/ - 00:00:32 192.168.45.5 Serial0/2/0
R4:
R4#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
0.0.0.6 0 FULL/ - 00:00:37 192.168.46.6 FastEthernet0/1
0.0.0.5 0 FULL/ - 00:00:36 192.168.45.5 Serial0/1/0
R6:
R6#show ip interface brief
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 unassigned YES unset administratively down down
FastEthernet0/1 192.168.46.6 YES manual up up
Serial0/1/0 unassigned YES unset administratively down down
Serial0/2/0 unassigned YES unset administratively down down
Serial0/2/1 unassigned YES unset administratively down down
R2:
R2#show ip ospf interface Serial0/2/0
Serial0/2/0 is up, line protocol is up
Interface is unnumbered. Using address of GigabitEthernet0/1 (192.168.2.2), Area 254
Process ID 1, Router ID 0.0.0.2, Network Type POINT_TO_POINT, Cost: 64
Enabled by interface config, including secondary ip addresses
Transmit Delay is 1 sec, State POINT_TO_POINT
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
Hello due in 00:00:06
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 1/2, flood queue length 0
Next 0x0(0)/0x0(0)
Last flood scan length is 1, maximum is 1
Last flood scan time is 0 msec, maximum is 0 msec
Neighbor Count is 1, Adjacent neighbor count is 1
Adjacent with neighbor 0.0.0.5
Suppress hello for 0 neighbor(s)
R5:
R5#show ip ospf interface Serial0/2/0
Serial0/2/0 is up, line protocol is up
Interface is unnumbered. Using address of Serial0/0/0 (192.168.45.5), Area 254
Process ID 1, Router ID 0.0.0.5, Network Type POINT_TO_POINT, Cost: 64
Enabled by interface config, including secondary ip addresses
Transmit Delay is 1 sec, State POINT_TO_POINT
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
Hello due in 00:00:05
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 2/2, flood queue length 0
Next 0x0(0)/0x0(0)
Last flood scan length is 1, maximum is 1
Last flood scan time is 0 msec, maximum is 0 msec
Neighbor Count is 1, Adjacent neighbor count is 1
Adjacent with neighbor 0.0.0.2
Suppress hello for 0 neighbor(s)
Here comes the solution attempt:
R2:
router ospf 1 area 254 virtual-link 0.0.0.5 !
R5:
router ospf 1 area 254 virtual-link 0.0.0.2 area 254 virtual-link 0.0.0.4
R4:
router ospf 1 area 254 virtual-link 0.0.0.5
R2:
R2#show ip ospf virtual-links
Virtual Link OSPF_VL0 to router 0.0.0.5 is up
Run as demand circuit
DoNotAge LSA allowed.
Transit area 254, via interface Serial0/2/0, Cost of using 64
Transmit Delay is 1 sec, State POINT_TO_POINT,
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Hello due in 00:00:01
R5:
R5#show ip ospf virtual-links
Virtual Link OSPF_VL1 to router 0.0.0.4 is up
Run as demand circuit
DoNotAge LSA allowed.
Transit area 254, via interface Serial0/0/0, Cost of using 64
Transmit Delay is 1 sec, State POINT_TO_POINT,
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Hello due in 00:00:03
Adjacency State FULL (Hello suppressed)
Index 1/3, retransmission queue length 0, number of retransmission 0
First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)
Last retransmission scan length is 0, maximum is 0
Last retransmission scan time is 0 msec, maximum is 0 msec
Virtual Link OSPF_VL0 to router 0.0.0.2 is down
Run as demand circuit
DoNotAge LSA allowed.
Transit area 254, Cost of using 65535
Transmit Delay is 1 sec, State DOWN,
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
R4:
R4#show ip ospf virtual-links
Virtual Link OSPF_VL0 to router 0.0.0.5 is up
Run as demand circuit
DoNotAge LSA allowed.
Transit area 254, via interface Serial0/1/0, Cost of using 64
Transmit Delay is 1 sec, State POINT_TO_POINT,
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Hello due in 00:00:00
Adjacency State FULL (Hello suppressed)
Index 2/3, retransmission queue length 0, number of retransmission 0
First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)
Last retransmission scan length is 0, maximum is 0
Last retransmission scan time is 0 msec, maximum is 0 msec
R2:
R2#show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 0.0.0.5 0 FULL/ - 00:00:33 192.168.45.5 Serial0/2/0 R2#show ip route ospf O IA 192.168.46.0/24 [110/129] via 192.168.45.5, 01:38:18, Serial0/2/0 O 192.168.45.0/24 [110/128] via 192.168.45.5, 01:38:18, Serial0/2/0
R5:
R5#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
0.0.0.4 0 FULL/ - - 192.168.45.4 OSPF_VL1
0.0.0.2 0 FULL/ - 00:00:39 192.168.2.2 Serial0/2/0
0.0.0.4 0 FULL/ - 00:00:35 192.168.45.4 Serial0/0/0
R4:
R4#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
0.0.0.5 0 FULL/ - - 192.168.45.5 OSPF_VL0
0.0.0.6 0 FULL/ - 00:00:33 192.168.46.6 FastEthernet0/1
0.0.0.5 0 FULL/ - 00:00:31 192.168.45.5 Serial0/1/0
R6:
R6#show ip route ospf
O IA 192.168.45.0/24 [110/65] via 192.168.46.4, 01:36:28, FastEthernet0/1
R2:
R2#show ip ospf interface brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
VL0 1 0 0.0.0.0/0 64 P2P 0/0
Gi0/1 1 0 192.168.2.2/24 1 DR 0/0
Se0/2/0 1 254 Unnumbered Gi0/1 64 P2P 1/1
R5:
R5#show ip ospf interface brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
VL1 1 0 192.168.45.5/24 64 P2P 1/1
VL0 1 0 0.0.0.0/0 65535 DOWN 0/0
Se0/2/0 1 254 Unnumbered Se0/0/0 64 P2P 1/1
Se0/0/0 1 254 192.168.45.5/24 64 P2P 1/1
R4:
R4#show ip ospf interface brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
VL0 1 0 192.168.45.4/24 64 P2P 1/1
Fa0/1 1 0 192.168.46.4/24 1 P2P 1/1
Se0/1/0 1 254 192.168.45.4/24 64 P2P 1/1
Here’s the added configuration – virtual links between R2 and R4, on top of the R2-R5-R4 that were already built.
R2:
router ospf 1 area 254 virtual-link 0.0.0.4 !
R4:
router ospf 1 area 254 virtual-link 0.0.0.2 !
R2:
R2#show ip ospf virtual-links Virtual Link OSPF_VL1 to router 0.0.0.4 is up Run as demand circuit DoNotAge LSA allowed. Transit area 254, via interface Serial0/2/0, Cost of using 128 Transmit Delay is 1 sec, State POINT_TO_POINT, Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 Hello due in 00:00:04 Virtual Link OSPF_VL0 to router 0.0.0.5 is up Run as demand circuit DoNotAge LSA allowed. Transit area 254, via interface Serial0/2/0, Cost of using 64 Transmit Delay is 1 sec, State POINT_TO_POINT, Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 Hello due in 00:00:05 R2#show ip ospf interface brief Interface PID Area IP Address/Mask Cost State Nbrs F/C VL1 1 0 0.0.0.0/0 128 P2P 0/0 VL0 1 0 0.0.0.0/0 64 P2P 0/0 Gi0/1 1 0 192.168.2.2/24 1 DR 0/0 Se0/2/0 1 254 Unnumbered Gi0/1 64 P2P 1/1
R4:
R4#show ip ospf virtual-links Virtual Link OSPF_VL1 to router 0.0.0.2 is down Run as demand circuit DoNotAge LSA allowed. Transit area 254, Cost of using 65535 Transmit Delay is 1 sec, State DOWN, Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 Virtual Link OSPF_VL0 to router 0.0.0.5 is up Run as demand circuit DoNotAge LSA allowed. Transit area 254, via interface Serial0/1/0, Cost of using 64 Transmit Delay is 1 sec, State POINT_TO_POINT, Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 Hello due in 00:00:05 Adjacency State FULL (Hello suppressed) Index 2/3, retransmission queue length 0, number of retransmission 0 First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0) Last retransmission scan length is 0, maximum is 0 Last retransmission scan time is 0 msec, maximum is 0 msec R4#show ip ospf interface brief Interface PID Area IP Address/Mask Cost State Nbrs F/C VL1 1 0 0.0.0.0/0 65535 DOWN 0/0 VL0 1 0 192.168.45.4/24 64 P2P 1/1 Fa0/1 1 0 192.168.46.4/24 1 P2P 1/1 Se0/1/0 1 254 192.168.45.4/24 64 P2P 1/1
R5:
R5#ping 192.168.2.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.2.2, timeout is 2 seconds: ..... Success rate is 0 percent (0/5) R5#show ip route 192.168.2.0 % Network not in table
R2:
R2#show interfaces Serial0/2/0
Serial0/2/0 is up, line protocol is up
Hardware is GT96K Serial
Interface is unnumbered. Using address of GigabitEthernet0/1 (192.168.2.2)
MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
CRC checking enabled
Last input 00:00:05, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/1/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 1158 kilobits/sec
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1735 packets input, 130740 bytes, 0 no buffer
Received 897 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
2083 packets output, 158695 bytes, 0 underruns
0 output errors, 0 collisions, 6 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
R5:
R5#show interfaces Serial0/2/0
Serial0/2/0 is up, line protocol is up
Hardware is GT96K Serial
Interface is unnumbered. Using address of Serial0/0/0 (192.168.45.5)
MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
CRC checking enabled
Last input 00:00:00, output 00:00:03, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/1/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 1158 kilobits/sec
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
2099 packets input, 159767 bytes, 0 no buffer
Received 905 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
1745 packets output, 131522 bytes, 0 underruns
0 output errors, 0 collisions, 6 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
1 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
Follow-up to the configured virtual links is to change the encapsulation on the serial interfaces between R2 and R5 to PPP.
R2:
interface Serial0/2/0 encapsulation ppp !
R5:
interface Serial0/2/0 encapsulation ppp !
Here’s what we have at this point:
R6:
R6#show ip route ospf
O IA 192.168.45.0/24 [110/65] via 192.168.46.4, 02:15:26, FastEthernet0/1
R2:
R2#show ip route ospf
O IA 192.168.46.0/24 [110/129] via 192.168.45.5, 00:04:10, Serial0/2/0
192.168.45.0/24 is variably subnetted, 2 subnets, 2 masks
O 192.168.45.0/24 [110/128] via 192.168.45.5, 00:04:10, Serial0/2/0
R5:
R5#show ip ospf virtual-links
Virtual Link OSPF_VL1 to router 0.0.0.4 is up
Run as demand circuit
DoNotAge LSA allowed.
Transit area 254, via interface Serial0/0/0, Cost of using 64
Transmit Delay is 1 sec, State POINT_TO_POINT,
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Hello due in 00:00:08
Adjacency State FULL (Hello suppressed)
Index 1/3, retransmission queue length 0, number of retransmission 0
First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)
Last retransmission scan length is 0, maximum is 0
Last retransmission scan time is 0 msec, maximum is 0 msec
Virtual Link OSPF_VL0 to router 0.0.0.2 is down
Run as demand circuit
DoNotAge LSA allowed.
Transit area 254, Cost of using 65535
Transmit Delay is 1 sec, State DOWN,
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Let’s take a look what happens if we remove the virtual links between R2-R5 and R5-R4, and just leave the virtual link between R2 and R4 in place.
R2:
router ospf 1 no area 254 virtual-link 0.0.0.5 area 254 virtual-link 0.0.0.4 !
R5:
router ospf 1 no area 254 virtual-link 0.0.0.2 no area 254 virtual-link 0.0.0.4 !
R4:
router ospf 1 no area 254 virtual-link 0.0.0.5 area 254 virtual-link 0.0.0.2 !
R2:
R2#show ip ospf virtual-links
Virtual Link OSPF_VL1 to router 0.0.0.4 is up
Run as demand circuit
DoNotAge LSA allowed.
Transit area 254, via interface Serial0/2/0, Cost of using 128
Transmit Delay is 1 sec, State POINT_TO_POINT,
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Hello due in 00:00:02
R4:
R4#show ip ospf virtual-links
Virtual Link OSPF_VL1 to router 0.0.0.2 is down
Run as demand circuit
DoNotAge LSA allowed.
Transit area 254, Cost of using 65535
Transmit Delay is 1 sec, State DOWN,
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Let me give you a little push.
R4:
R4#show ip route ospf
R5:
router ospf 1 redistribute connected subnets !
R4:
R4#show ip route ospf 192.168.2.0/32 is subnetted, 1 subnets O E2 192.168.2.2 [110/20] via 192.168.45.5, 00:00:00, Serial0/1/0 R4#show ip ospf virtual-links Virtual Link OSPF_VL1 to router 0.0.0.2 is down Run as demand circuit DoNotAge LSA allowed. Transit area 254, Cost of using 65535 Transmit Delay is 1 sec, State DOWN, Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
R2:
R2#ping 192.168.45.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.45.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R4:
R4#ping 192.168.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
We’ve had an alternate proposal for the solution: Remove all virtual links and apply GRE tunnels from R2 to R5 and from R5 to R4. Here’s that configuration.
R2:
interface Tunnel0 ip unnumbered GigabitEthernet0/1 tunnel source GigabitEthernet0/1 tunnel destination 192.168.45.5 ip ospf 1 area 0 ! router ospf 1 no area 254 virtual-link 0.0.0.4 !
R5:
interface Tunnel0 ip unnumbered Serial0/0/0 tunnel source Serial0/0/0 tunnel destination 192.168.45.4 ip ospf 1 area 0 ! interface Tunnel1 ip unnumbered Serial0/0/0 tunnel source Serial0/0/0 tunnel destination 192.168.2.2 ip ospf 1 area 0 !
R4:
interface Tunnel0 ip unnumbered Serial0/1/0 tunnel source Serial0/1/0 tunnel destination 192.168.45.5 ip ospf 1 area 0 ! router ospf 1 no area 254 virtual-link 0.0.0.2 !
At this point, we have the reachability between R2 and R6! However… there was a requirement for a particular output to match,. Does it? :-)
R2:
R2#show ip route ospf O 192.168.46.0/24 [110/2001] via 192.168.45.5, 00:07:42, Tunnel0 192.168.45.0/24 is variably subnetted, 2 subnets, 2 masks O 192.168.45.0/24 [110/128] via 192.168.45.5, 02:33:30, Serial0/2/0 R2#ping 192.168.46.6 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.46.6, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R6:
R6#ping 192.168.2.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.2.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms R6#show ip route ospf O IA 192.168.45.0/24 [110/65] via 192.168.46.4, 04:47:25, FastEthernet0/1 192.168.2.0/24 is variably subnetted, 2 subnets, 2 masks O E2 192.168.2.2/32 [110/20] via 192.168.46.4, 01:53:36, FastEthernet0/1 O 192.168.2.0/24 [110/2002] via 192.168.46.4, 00:09:07, FastEthernet0/1
Let’s remove the configured redistribution on R5.
R5:
router ospf 1 no redistribute connected !
R6#show ip route ospf O IA 192.168.45.0/24 [110/65] via 192.168.46.4, 04:54:47, FastEthernet0/1 O 192.168.2.0/24 [110/2002] via 192.168.46.4, 00:16:29, FastEthernet0/1
We’re 99% there! Something still doesn’t match.
As we identified in the comments below, the metrics of the routes on R2 and R6 don’t match. We can manipulate the interface costs and get the desired results, but there is a simpler solution. One of the proposals was to build a GRE tunnel between R2 and R5 only. That means at this point removing the tunnel between R4 and R5. Here it is.
R5:
no interface Tunnel0
R4:
no interface Tunnel0
R6:
R6#show ip route ospf
O IA 192.168.45.0/24 [110/65] via 192.168.46.4, 05:40:42, FastEthernet0/1
Back to the drawing board… Or at least half-way there… :-)
R5:
R5#show ip route | begin ^Gateway
Gateway of last resort is not set
C 192.168.45.0/24 is directly connected, Serial0/0/0
192.168.2.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.2.2/32 is directly connected, Serial0/2/0
O 192.168.2.0/24 [110/1001] via 192.168.2.2, 00:28:05, Tunnel1
At this point, if we add the virtual link between R5 and R4…
R5:
router ospf 1 area 254 virtual-link 0.0.0.4 !
R4:
router ospf 1 area 254 virtual-link 0.0.0.5 !
R2:
R2#ping 192.168.46.6 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.46.6, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms R2#show ip route ospf O 192.168.46.0/24 [110/129] via 192.168.45.5, 00:00:56, Serial0/2/0 192.168.45.0/24 is variably subnetted, 2 subnets, 2 masks O 192.168.45.0/24 [110/128] via 192.168.45.5, 04:04:36, Serial0/2/0
R6:
R6#ping 192.168.2.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.2.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms R6#show ip route ospf O IA 192.168.45.0/24 [110/65] via 192.168.46.4, 06:17:37, FastEthernet0/1 O 192.168.2.0/24 [110/1066] via 192.168.46.4, 00:01:28, FastEthernet0/1
The connectivity is successful and our command outputs match!
I should really repeat that we’ve had a connectivity solution long time ago using the two GRE tunnels. The exercise after that was just to match the metrics, which was again solved using the cost manipulation (not shown in the article, but mentioned in the comments). In the end, we were after “the simplest” solution, even though… there was nothing simple with this problem.
Thank you all for participating in this interactive troubleshooting session. You were all fantastic!
Solution
I’ll post the detailed explanation of the problem and the solution I think to be the most appropriate in the following post, regardless of how we solve this one today! That article will be called “OSPF Virtual Links and IP Unnumbered Interfaces” and will not rely on any troubleshooting performed here, as I will use this problem for a foundation to explain the root causes of the problem, which are the unnumbered link between R2 and R5, and the split backbone.
Happy studies!
–
Marko Milivojevic – CCIE #18427
Senior CCIE Instructor – IPexpert
Join our Online Study List
Tags: OSPF, OSPF Backbone, Troubleshooting
Hi Marko!
could you please paste output for
“show ip ospf interface brief” from R2?
I would like to know if there is any tunnel or virtual link.
Please refresh :-)
Can you do show ip ospf int on R2 and R5 as well as show ip ospf nei on R2 and R4.
Obviously we need a virtual link over area 254, but I assume that might be tricky due to the ip unnumbered on R2.
Sorry Marc, I missed your instructions. I ran “sh ip ospf int br” and “sh ip ospf nei” on R5. Is that OK, or you want the full output for interfaces?
R4:
show ip route ospf
show ip ospf database
Done. Hit the refresh button… :-)
R6:
show ip int brief
This is now in.
Thanks. Realized after I asked for it the end-state RTs basically invalidate the need to look at that. ;-)
The size of the network suggests it is OK to have all routers in area 0. What is still broken if you do this?
That’s a very good observation. I should’ve made it clear that you cannot change interface area membership. I updated the post with those restrictions.
-Marko
sh ip ospf interface Se0/2/0 on R2 and R5
Absolutely… :-)
I think that since R2 sees 192.168.46.0/24 network as “O” route then there should be a virtual link between R2 and R5 and then R5 to R4. That should get us the required output.
That sounds reasonable. Can you please provide me with the commands to configure and verify the solution? :-)
Sure marko..
R2
R5
R4
Thanks :)
This is now in place. How would you like me to verify what it did? :-)
R2
router ospf 1
area 256 virtual-link 192.168.45.5 (Assuming 192.168.45.5 is the routerID for R4)
R4
router ospf 1
area 256 virtual-link 192.168.2.x (what ever the routerID is for R2)
Router IDs are available from the output we ran earlier and the virtual links are now in place. How should we verify the configuration and our solution?
“show ip ospf virt” on R2,R5,R4 . Please check if links come up. I doubt that now.haha.lets see.
Excellent. The post has been updated with these commands :-)
Nicely done. I have updated the output. Check it out.
show ip ospf neighbor to verify the new adjacency and show ip route to see if the routing table matches your desired output.
Addition to above comment :
If you see virtual links to be up, then check if we are learning ospf routes on R5 and R4 apart from R2.
once we have routes then you can ping as per the above requirement in the task.
Please refresh the post to check the output of the commands provided in the meantime. I can ping, but it won’t work :-)
The virtual link needs to be between R2 and R4 and not involve R5. This is why it does not work.
OK, good observation, would you like me to configure it as such and remove the existing virtual links?
Try to verify with:
R2,R4,R5:
show ip ospf int bri
R2,R6:
show ip ro ospf
Output of “show ip route” on R2 and R6 is already there. The rest of the output is provided, so please refresh.
If we create a virtual link between R2 and R4 directly apart from the current configuration I think O routes will be will be preferred.
R2
area 254 virtual-link 0.0.0.4
R4
area 254 virtual-link 0.0.0.2
Please check if the links come up and then try again with the “show ip route ospf” to see if we get O routes.
Would you like me to remove the existing virtual links before I try this?
Please keep them for now. We will check if this works.
OK, I will add the virtual link between R2 and R4 then followed by “show ip ospf virtual-links” and “show ip ospf interface brief” from R2 and R4. Please, refresh.
How do we change it to IA? ping works everything is fine just its a IA route and we want O :O
Leaving it at that :) will see the solution. It was fun.
Thanks.
Well, that’s the thing, isn’t it? How do we make it work? :-)
Can you just ping/trace 192.168.2.2 from R5?
Very good test. The ping doesn’t work and I provided the output of “show ip route 192.168.2.0″ for a clue.
I should point out that the difficulty level of this problem on a scale from 1 to 10 is probably 9.99. This is a very complex problem.
Hang on…. Please “show int s0/2/0″ on R2, R5.
Post has been updated with those commands.
Both R2, R5:
int s0/2/0
encap ppp
Let’s get the peer routes in there so we might have addressable unicast across that link between R2/R5
Very good. It’s done… Now what? :-)
This is now a breakthrough thinking. It’s done, but can you explain why you wanted this? :-)
Yep. Across a serial link, all OSPF traffic is link-local multicast. You don’t need working unicast to get an adjacency and even exchange routes. But VL traffic is always unicasted between the endpoints, so you need to have addressable unicast between them.
Can you “show ip ospf virtual” on R5 again to check the VL status?
And of course by switching to PPP, each peer to inform the other of it’s assigned IP address via IPCP, and the peer will (by default) install a /32 route to that peer IP pointed out the interface. So it will allow the unnumbered neighbors to unicast to each other because they now have a connected route to the other peer’s assigned IP.
Sorry, should have included that bit.
Again, very good, but read my previous response for a hidden hint :-)
There will always be unicast even on point to point links after the neighbors move to exchange (start) phase, but for virtual links you absolutely need endpoint addresses in OSPF. Well observed and done. Requested output is in.
Ahh, thank you. I thought unicast was only used on broadcast interfaces. Good clarification.
I updated the post with the routing tables on R2 and R6 after configuring all the virtual links and changing R2-R5 link to PPP. Still no-go…
Ahhh….
How about this:
R5:
router ospf 1
redist conn sub
That should get the /32 for 192.168.2.2 into OSPF area 254.
Now perhaps you can get just a single VL up between R4 and R2.
I can do it, but let me save you the time. Since all links on R5 are already advertised as OSPF internal routes, redistribution will have no effect.
Fair enough.
Can R4 ping 192.168.2.2, then? If so, can you move to a single VL between R2 and R4 to finally fix Area 0?
Very well. I have removed the virtual links between R2 and R5, and between R5 and R4. I added “show ip ospf virtual-links” output as well.
You didn’t verify whether R4 could ping 192.168.2.2 first! ;-)
The ‘show ip ospf virtual’ on R4 suggests he still doesn’t actually have the route to 192.168.2.2.
‘show ip ospf data’ on R4, please.
Quite correct. I added the requested redistribution from the previous step you suggested. R4 now has the route.
Just to add a thing or two:
I added couple of show commands at the end now to give you a little bit of a push. Let’s see if it helps.
-Marko.
I’m going to take a guess that R4 won’t build a VL to an endpoint IP that it knows via an external (make sense, since the external is describing a destination outside of the OSPF domain).
Can R2 and R4 ping one another at this point?
This wouldn’t match up with your end-state RT output, but now how about replacing the VL with a GRE tunnel, if we can’t build a VL to an externally learned IP.
We shouldn’t get a recursive tunnel issue, since the GRE endpoint IP would actually be learned as a /32 and not the /24 that R2 should be advertising 192.168.2.0 into area 0 as.
Bah, never mind. Can’t add IPs.
And making an unnumbered GRE tunnel just sounds like we’re starting over.
Don’t give up.
Well done! Virtual links won’t form for the externally known destinations! Since virtual links don’t seem to want to play, we can always replace them with GRE tunnels – nice. However, let’s try those pings you asked for. Post has been updated.
What would you have me do next?
OK, now I’m asking and not telling: Can we do an IP unnumbered GRE tunnel between R2 and R4? Maybe it’s not as crazy as it sounds. I am concerned about recursive tunneling issue in the R2–>R4 direction, though. R4 has the /32 it’s learning via R5, but if we put up a GRE tunnel, R2 will learn 192.168.45.0 via it.
Could you put that GRE tunnel in another new area and filter 192.168.45.0 from being advertised into that area from R4?
I’m now putting my Lab Proctor hat on. I will provide the answers and it is up to you what you want to do:
If the path cost for the virtual-link goes higher than 65535, the link won’t operate. I’m thinking this has something to do with it but not sure how to approach it. Maybe auto-cost reference-bandwidth?
That’s an interesting angle. I like the thought process. How would you have me verify if that’s an issue?
Also, please note that the cost you are referencing is the cost for the link that is DOWN. Virtual-link that’s up has the cost of 128.
The network type of both the routers are different is it possible to make it similar on both sides?
Which routers are you referring to? I can fix network types…
OK, so how about a GRE tunnel R4R2, unnumbered, then use a distribute-list on R2 to prevent the tunnel endpoint learned over the tunnel from making it into the IP routing table when learned from the RID of R4.
On my phone now, so please don’t ask me for exact commands ;-)
Looks like Spiridon is already headed in roughly the same direction, but I was going to try just one virtual-link between R2 and R4 like this:
Remove all VLs.
R2:
interface Tunnel0
ip unnumbered FastEthernet0/0
tunnel source Serial0/2/0
tunnel destination 192.168.45.4
ip ospf 1 area 0
exit
!
access-list 45 permit 192.168.45.0 0.0.0.255
!
router ospf 1
distance 255 0.0.0.4 0.0.0.0 45
R4:
interface Tunnel0
ip unnumbered FastEthernet0/1
tunnel source Serial0/1/0
tunnel destination 192.168.2.2
ip ospf 1 area 0
exit
That was my first thought as well, but I doubt R4 would have a route to 192.168.2.2 . In fact, according to the post, it doesn’t have any ospf routes , which makes sense since the only not directly connected route is in the OTHER area 0.
Once Marko added “redistribute connected subnets” to the R5 OSPF process, it added the PPP peer route in to OSPF as an E2 and R4 picked it up. R2 and R4 have IP reachability to one another now.
Excellent thinking.
I somehow managed to missed that. Still, according to the verification text that we’re supposed to match, 192.168.2.0/24 is an intra-area route on R6 . Would redistribution be valid since it will mark it as external ?
Fantastic observation, Spiridon! Paying close attention to the details is extremely important.
Also, I did, of course, mean GRE tunnel and not virtual-link up above!
If we already have ppp with neighbour routes installed, we can just use 2 unnumbered gre tunnels in area0. One using the link between R4 and R5 as endpoints, the other using the addresses of the interfaces we’ve unnumbered the link between R5 and R2 to.
That’s interesting idea. Can you give me the commands to run, please? :-)
On R4:
int Tun0
ip unn s0/1/0
tun source s0/1/0
tun dest 192.168.45.5
ip ospf 1 a 0
On R5:
int Tun0
ip unn s0/0/0
tun source s0/0/0
tun dest 192.168.45.4
ip ospf 1 a 0
int Tun1
ip unn s0/0/0
tun source s0/0/0
tun dest 192.168.2.2
ip ospf 1 a 0
on R2:
int Tun0
ip unn g0/1
tun source g0/1
tun dest 192.168.45.5
ip ospf 1 a 0
Do you want me to keep the configured virtual links or remove them?
Remove them :) We don’t need any further area0 adjacencies since we’ll have the tunnels and they won’t work over unnumbered anyway.
The only change from the intial config that you should keep is setting the encapsulation on the segment between R2 and R5 to ppp .
OK, the configuration has been applied. What next? :-)
sh ip rou ospf on R2 and R6.
ping 192.168.2.2 on R6
Very nicely done, Spiridon. We have the connectivity, but does the output match?
Devil is in the details…
Heheh, I was just posting that on http://blog.ipexpert.com/2012/04/30/interactive-troubleshooting-blog-ospf-routing/comment-page-1/#comment-115084 . Since we have 2 tunnels, we don’t need the redistribution.
Should I remove it then? :-)
Yes please :)
Done. Please check the updated output. Take a look at the metrics.
So close….
The E2 for 192.168.2.2/32 could be filtered on R6, but we also have the problem of the egress interface on R2 being a tunnel. And the metrics don’t match.
Almost like the tunnel needs to be from R5 rather than R2.
Oh no… We don’t have to tunnel R4 to R5, and then *bridge* the R2-R4 serial link to the tunnel, so we?!!!?
Good thinking, but really adding bridging to this problem would be a little bit too evil even by my standards. It may work (I haven’t tried), but since we solved the problem and are just looking for a “cleaner” solution, let’s try and find one! :-)
The virtual-links must be on ABR´s routers R2 and R4, and the subnet 192.168.2.0/24 must be known on area 0.
The config i propose is:
on R2
router ospf 1
passive-interface gi0/1
network 192.168.2.0 0.0.0.255 area 0
area 254 virtual-link 0.0.0.4
on R4
area 254 virtual-link 0.0.0.2
Alberto,
Area 0 is already activated on R2′s 192.168.2.0/24 using the interface-level configuration. We have also already tried to build the link between R2 and R4 and that didn’t work.
Spiridon K posted a working configuration that establishes the connection between R2 and R6. In most cases, this would be an end to it, but OSPF costs on the output we need to match don’t match. Let’s get that sorted out and we’re done!
R4:
int tun 0
ip ospf cost 1064
i mean
R4:
int tun 0
ip ospf cost 1065
Ok, how about this ?
On R4:
int tun0
ip ospf cost 532
On R5
int tun0
ip ospf cost 532
int tun1
ip ospf cost 64
On R2
int tun0
ip ospf cost 64
Includes a bit of guesswork ( sh ip ospf int on the tunnels would help ) but it should fix the cost issues, I can’t however for the life of me think of a way to have R2 list s0/2/0 instead of the tunnel as the egress interface.
R2
interface Tunnel0
ip ospf cost 266
R5
int tun0
ip ospf cost 266
int tun1
ip ospf cost 266
R4
int tun0
ip ospf cost 266
Many proposals to fix the problem by tweaking the metrics. All of them would work, but… Are there simpler solutions that can fix the problem?
All routers:
router ospf 1
auto-cost reference-bandwidth 1000
How will that affect GRE tunnels that have the default bandwidth of 100 Kb/s and Serial interfaces with the bandwidth of 1544 Kb/s?
As I previously said – cost manipulation at this stage is a done deal. We can manipulate costs on all routers and get them to match on both sides, but I’m looking for an alternate solution for the original problem – not only the costs.
Is there a solution that will solve the connectivity and match the output? We have a hybrid problem – is there a hybrid solution… so to speak :-)
Change the bandwidth of each tunnel to 187kbps?
That may work, but it’s still just metric manipulation. Let’s find an alternate to that!
Just one GRE tunnel needed between R2 and R5!!!!
OK, let’s try that. I have removed the tunnel between R5 and R4.
What if I said that the key is in virtual links? How would that sit with everyone at this point? :-)
This is weird, because on R4 the subnet 192.168.2.0/24 is in area 254 in the ospf database, but the router is not installing the route in the routing table…
Ahhh – you noticed! Well done. It’s actually not weird at all. It’s an expected OSPF behavior in multi-area operation. It’s part of the loop prevention mechanism. The solution article (to be published later this week) will explain that, but nicely spotted! Good job!
If I may add to what I wrote before. What you discovered here is the root cause of the problem. The reason why there is no communication between R2 and R6, so again – well done!
Alberto is on to something. GRE tunnel between R2 and R5 and then a virtual link between ?? and R4.
That’s very good. We only need to find what to replace those ??s with. Let me help you – a virtual link between R4 and R? ;-)
The last comment is forgetting the GRE tunnels Clay…i noticed that at start but i didn´t know how to solve it without changing the areas of the routers…
That’s quite right. We can observe 192.168.2.0/24 in the OSPF database before we add any GRE tunnels and/or virtual links.
Can u add the routing table and ospf table of R5 before virtual-links and GRE tunnels?
Let’s keep the GRE tunnel between R2 and R5 in place. Output of the routing table added.
The final solution is GRE tunnel between R2 and R5 and virtual-link between R5 and R4?…
Bingo! To complete your answer:
The result is posted in the article.