DMVPN Phase 3 (spoke-to-spoke) Point-to-Multipoint OSPF

Lab Introduction

This lab tests point-to-multipoint (P2MP) OPSF with DMVPN phase 3 to enable spoke-to-spoke traffic and understand how OSPF metric/cost works in point-to-multipoint scenario.

I use C7200 15.2(4) M6 (IOS: c7200-adventerprisek9-mz.152-4.M6.bin) as router on GNS3. Please note:

  • Not all C7200 IOS support DMVPN Phase 3 with GNS3. For example, when I used later version 15.2(4) S7 (IOS: c7200-adventerprisek9-mz.152-4.S7.bin), it says ‘ip nhrp redirect’ failed to initialize.
  • CSR1000v (IOS-XE) supports DMVPN Phase 3 with GNS3; however, it shows some different results in terms of troubleshooting. It should be ideal to use CSR1000v if RAM is not an issue.
  • Crypto is not configured to protect tunnels in this lab for simplicity.

Lab topology is as below:
dmvpn_ospf_p2mp.jpg

HUB1_1

G1/0 : 200.1.1.101/24 (global routing instance)

Tunnel 101: 172.16.1.101/24 (global routing instance), OSPF 1 area 0

Lo 1: 10.66.1.101/24 (global routing instance), OSPF 1 area 1

HUB1_1 Tu101 configuration:
interface Tunnel 101
ip address 172.16.1.101 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication GREEN
ip nhrp map multicast dynamic
ip nhrp network-id 101
ip nhrp holdtime 300
ip nhrp redirect
ip ospf network point-to-multipoint
ip ospf 1 area 0
tunnel source GigabitEthernet1/0
tunnel mode gre multipoint
tunnel key 101

HUB1_2

G1/0 : 200.1.1.102/24 (global routing instance)

Tunnel 101: 172.16.1.102/24 (GREEN_IVRF), OSPF 1 area 0, cost 1001

Lo 1: 10.66.2.102/24 (GREEN_IVRF), OSPF 1 area 1

HUB1_2 Tu101 configuration:
interface Tunnel 101
vrf forwarding GREEN_IVRF
ip address 172.16.1.102 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication GREEN
ip nhrp map multicast dynamic
ip nhrp network-id 101
ip nhrp holdtime 300
ip nhrp redirect
ip ospf network point-to-multipoint
ip ospf 1 area 0
ip ospf cost 1001
tunnel source GigabitEthernet1/0
tunnel mode gre multipoint
tunnel key 101

 

SPOKE1_1

G1/0 : 200.1.1.103/24 (global routing instance)

Tunnel 101: 172.16.1.103/24 (GREEN_IVRF), OPSF 1 area 0

Lo 1: 10.66.3.103/24 (GREEN_IVRF), OSPF 1 area 1

SPOKE1_1 Tu101 configuration:
interface Tunnel 101
vrf forwarding GREEN_IVRF
ip address 172.16.1.103 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication GREEN
ip nhrp map multicast 200.1.1.101
ip nhrp map multicast 200.1.1.102
ip nhrp map 172.16.1.101 200.1.1.101
ip nhrp map 172.16.1.102 200.1.1.102
ip nhrp network-id 101
ip nhrp holdtime 300
ip nhrp nhs 172.16.1.101 cluster 1
ip nhrp nhs 172.16.1.102 priority 2 cluster 1
ip nhrp nhs cluster 1 max-connections 2
ip nhrp nhs fallback 25
ip nhrp shortcut
ip ospf network point-to-multipoint
ip ospf 1 area 0
tunnel source GigabitEthernet1/0
tunnel mode gre multipoint
tunnel key 101

 

SPOKE1_2

G1/0 : 200.1.1.104/24 (global routing instance)

Tunnel 101: 172.16.1.104/24 (GREEN_IVRF), OSPF 1 area 0

Lo 1: 10.66.4.104/24 (GREEN_IVRF), OSPF 1 area 1

SPOKE1_2 Tu101 configuration:
interface Tunnel 101
vrf forwarding GREEN_IVRF
ip address 172.16.1.104 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication GREEN
ip nhrp map multicast 200.1.1.101
ip nhrp map multicast 200.1.1.102
ip nhrp map 172.16.1.101 200.1.1.101
ip nhrp map 172.16.1.102 200.1.1.102
ip nhrp network-id 101
ip nhrp holdtime 300
ip nhrp nhs 172.16.1.101 cluster 1
ip nhrp nhs 172.16.1.102 priority 2 cluster 1
ip nhrp nhs cluster 1 max-connections 2
ip nhrp nhs fallback 25
ip nhrp shortcut
ip ospf network point-to-multipoint
ip ospf 1 area 0
tunnel source GigabitEthernet1/0
tunnel mode gre multipoint
tunnel key 101

 

Verification 1 – show ip route

If we ‘show ip route’ on SPOKE1_1 and SPOKE1_2, spoke-to-spoke route appears traversing via HUB and with dual-trip metric (SPOKE1_1 to HUB1_1 to SPOKE1_2). Please note ‘10.66.4.104/32 [110/2001]’. Is the spoke-to-spoke direct tunnel really working?

SPOKE1_1#show ip route vrf GREEN_IVRF
10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O IA     10.66.1.101/32 [110/1001] via 172.16.1.101, 01:11:28, Tunnel101
O IA     10.66.2.102/32 [110/1001] via 172.16.1.102, 00:14:50, Tunnel101
C        10.66.3.0/24 is directly connected, Loopback1
L        10.66.3.103/32 is directly connected, Loopback1
O IA     10.66.4.104/32 [110/2001] via 172.16.1.101, 00:02:49, Tunnel101
172.16.0.0/16 is variably subnetted, 5 subnets, 2 masks
C        172.16.1.0/24 is directly connected, Tunnel101
O        172.16.1.101/32 [110/1000] via 172.16.1.101, 01:15:38, Tunnel101
O        172.16.1.102/32 [110/1000] via 172.16.1.102, 00:14:50, Tunnel101
L        172.16.1.103/32 is directly connected, Tunnel101
O        172.16.1.104/32 [110/2000] via 172.16.1.101, 00:02:49, Tunnel101

 

Verification 2 – ‘show ip nhrp’ and ‘show dmvpn’

Before issuing any traffic, we execute ‘show ip nhrp’ on SPOKEs. It only shows nhrp entries to hubs.

SPOKE1_1#    show ip nhrp
172.16.1.101/32 via 172.16.1.101
Tunnel101 created 02:30:32, never expire
Type: static, Flags: used
NBMA address: 200.1.1.101
172.16.1.102/32 via 172.16.1.102
Tunnel101 created 02:30:32, never expire
Type: static, Flags: used
NBMA address: 200.1.1.102

However, when we ping from SPOKE1_2 to SPOKE1_1 lo1 address ‘ping vrf GREEN_IVRF 10.66.3.103’; SPOKE1_1 receives NHRP entries to SPOKE1_2 as dynamic type (172.16.1.104/32 via 172.16.1.104). Please also note ‘flags: router used’, which indicates spoke-to-spoke tunnel as temporary not ‘never expire’.

SPOKE1_1#    show ip nhrp
10.66.3.0/24 via 172.16.1.103
Tunnel101 created 00:00:07, expire 00:04:52
Type: dynamic, Flags: router unique local
NBMA address: 200.1.1.103
(no-socket)
172.16.1.101/32 via 172.16.1.101
Tunnel101 created 02:32:31, never expire
Type: static, Flags: used
NBMA address: 200.1.1.101
172.16.1.102/32 via 172.16.1.102
Tunnel101 created 02:32:31, never expire
Type: static, Flags: used
NBMA address: 200.1.1.102
172.16.1.104/32 via 172.16.1.104
Tunnel101 created 00:00:07, expire 00:04:51
Type: dynamic, Flags: router used
NBMA address: 200.1.1.104

SPOKE1_2 also receives NHRP entries to SPOKE1_1 as dynamic type (172.16.1.103/32 via 172.16.1.103).

SPOKE1_2#show ip nhrp
172.16.1.101/32 via 172.16.1.101
Tunnel101 created 01:34:28, never expire
Type: static, Flags: used
NBMA address: 200.1.1.101
172.16.1.102/32 via 172.16.1.102
Tunnel101 created 01:34:28, never expire
Type: static, Flags: used
NBMA address: 200.1.1.102
172.16.1.103/32 via 172.16.1.103
Tunnel101 created 00:10:01, expire 00:02:35
Type: dynamic, Flags: router implicit
NBMA address: 200.1.1.103
172.16.1.104/32 via 172.16.1.104
Tunnel101 created 00:10:01, expire 00:02:35
Type: dynamic, Flags: router unique local
NBMA address: 200.1.1.104

‘show dmvpn detail’ also demonstrates spoke-to-spoke dynamic tunnel.

SPOKE1_1#show dmvpn detail
Legend: Attrb –> S – Static, D – Dynamic, I – Incomplete
N – NATed, L – Local, X – No Socket
# Ent –> Number of NHRP entries with same NBMA peer
NHS Status: E –> Expecting Replies, R –> Responding, W –> Waiting
UpDn Time –> Up or Down Time for a Tunnel
=======================================================
Interface Tunnel101 is up/up, Addr. is 172.16.1.103, VRF “GREEN_IVRF”
Tunnel Src./Dest. addr: 200.1.1.103/MGRE, Tunnel VRF “”
Protocol/Transport: “multi-GRE/IP”, Protect “”
Interface State Control: Disabled
nhrp event-publisher : Disabled
IPv4 NHS:
172.16.1.101  RE priority = 0 cluster = 1
172.16.1.102  RE priority = 2 cluster = 1
Type:Spoke, Total NBMA Peers (v4/v6): 4# Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb    Target Network
—– ————— ————— —– ——– —– —————–
1 200.1.1.103        172.16.1.103    UP 01:26:03  DLX       10.66.3.0/24
1 200.1.1.101        172.16.1.101    UP 00:50:08    S    172.16.1.101/32
1 200.1.1.102        172.16.1.102    UP 01:38:28    S    172.16.1.102/32
1 200.1.1.104        172.16.1.104    UP 00:07:02    D    172.16.1.104/32

One HUB1_1, we also see nhrp redirect information as below:

HUB1_1#show ip nhrp redirect
I/F      NBMA address     Destination    Drop Count    ExpiryTunnel101  200.1.1.104     10.66.3.103          36        00:00:05

 

Verification 3 – debug nhrp

Let’s have a detailed look on what exactly happened on HUB1_1, SPOKE1_1 and SPOKE1_2 by ‘debug nhrp’.

HUB1_1: provides SPOKE physical WAN-facing IP to allow dynamic tunnel to be established between SPOKEs.

*May  4 20:34:41.415: NHRP: Tunnels gave us remote_nbma: 200.1.1.104 for Redirect
*May  4 20:34:41.415: NHRP: Attempting to Redirect, remote_nbma: 200.1.1.104

 

SPOKE1_2 adds SPOKE1_1 tunnel endpoint (172.16.1.103) as NHRP entry and sends packet via 172.16.1.103 directly. When no traffic traversing SPOKE1_1 and SPOKE1_2, the dynamic spoke-to-spoke tunnel will be deleted.

*May  4 20:32:55.131: NHRP-ATTR: In nhrp_cache_pak LINE: 1422
*May  4 20:32:55.131: NHRP: Adding Tunnel Endpoints (VPN: 172.16.1.103, NBMA: 200.1.1.103)
*May  4 20:32:55.131: NHRP: NHRP subblock already exists for Tunnel Endpoints (VPN: 172.16.1.103, NBMA: 200.1.1.103)
*May  4 20:32:55.131: NHRP: Cache Delete: Converting prefix: ‘10.66.3.0’ in cache: 0x6B0072B8

 

SPOKE 1_1 also adds SPOKE1_2 tunnel endpoint (172.16.1.104) as NHRP entry and sends packet via 172.16.1.104 directly.

*May  4 20:31:42.631: NHRP: Adding Tunnel Endpoints (VPN: 172.16.1.104, NBMA: 200.1.1.104)
*May  4 20:31:42.639: NHRP: Successfully attached NHRP subblock for Tunnel Endpoints (VPN: 172.16.1.104, NBMA: 200.1.1.104)
*May  4 20:31:42.655: NHRP: Attempting to send packet via DEST 172.16.1.104
*May  4 20:31:42.655: NHRP: Encapsulation succeeded.  Sending NHRP Control Packet  NBMA Address: 200.1.1.101
*May  4 20:31:42.655: NHRP: Send Resolution Reply via Tunnel101 vrf 1, packet size: 133
*May  4 20:31:42.655:       src: 172.16.1.103, dst: 172.16.1.104

Verification 4 – traceroute

Traceroute result is interesting. When first time traceroute, the traffic traverses via HUB; however, traffic directly goes through SPOKE when tried second time after the dynamic spoke-to-spoke tunnel established.

SPOKE1_2#traceroute vrf GREEN_IVRF 10.66.3.103
Type escape sequence to abort.
Tracing the route to 10.66.3.103
VRF info: (vrf in name/id, vrf out name/id)
1 172.16.1.101 5 msec
172.16.1.102 5 msec
172.16.1.101 5 msec

 

SPOKE1_2#traceroute vrf GREEN_IVRF 10.66.3.103
Type escape sequence to abort.
Tracing the route to 10.66.3.103
VRF info: (vrf in name/id, vrf out name/id)
1 172.16.1.103 5 msec 5 msec *

 

When traffic going through spoke-to-spoke directly. Do a quick ‘show ip route’. 2 records are marked with % now. % means ‘next hop override’. Traffic to SPOKE1_2’s loopback 10.66.4.104 had HUB1_1’s tunnel IP (172.16.1.101) as next-hop according to the routing table; however, the next-hop is now overridden by SPOKE1_2’s tunnel IP (172.16.1.104). The traffic will now take 172.16.1.104 as next hop to reach 10.66.4.104 with OSPF metric 2001.

SPOKE1_1#show ip route vrf GREEN_IVRF
10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O IA     10.66.1.101/32 [110/1001] via 172.16.1.101, 01:11:28, Tunnel101
O IA     10.66.2.102/32 [110/1001] via 172.16.1.102, 00:14:50, Tunnel101
C        10.66.3.0/24 is directly connected, Loopback1
L        10.66.3.103/32 is directly connected, Loopback1
O IA%     10.66.4.104/32 [110/2001] via 172.16.1.101, 00:02:49, Tunnel101
172.16.0.0/16 is variably subnetted, 5 subnets, 2 masks
C        172.16.1.0/24 is directly connected, Tunnel101
O        172.16.1.101/32 [110/1000] via 172.16.1.101, 01:15:38, Tunnel101
O        172.16.1.102/32 [110/1000] via 172.16.1.102, 00:14:50, Tunnel101
L        172.16.1.103/32 is directly connected, Tunnel101
O  %  172.16.1.104/32 [110/2000] via 172.16.1.101, 00:02:49, Tunnel101

Verification 5 – OSPF neighbor

SPOKEs only form OPSF neighbor with HUBs, but not other SPOKEs.

SPOKE1_2#show ip ospf neighbour
Neighbor ID     Pri   State           Dead Time   Address         Interface
200.1.1.101       0   FULL/  –        00:01:51    172.16.1.101    Tunnel101
200.1.1.102       0   FULL/  –        00:01:47    172.16.1.102    Tunnel101

Other Useful Commands

Clear ip nhrp

Show ip nhrp nhs

Show ip nhrp nhs redundancy

Conclusion

1) SPOKEs only form routing neighbors with HUBs, but not other SPOKEs.

2) SPOKE to SPOKE OSPF routing metric is calculated as if traffic traversing HUB; i.e. SPOKE1_1 to SPOKE1_2 metric = 1000+1000 = 2000. When spoke-to-spoke traffic initiates, routing metric stays same but next-hop is overriden to spoke.

3) When spoke-to-spoke traffic initiates, HUB will redirect traffic to spoke and provide spoke WAN-facing physical interface IP. Dynamic tunnel between SPOKE1_1 and SPOKE1_2 will established based on information provided by HUB. When spoke-to-spoke traffic ends, the dynamic tunnel will be terminated.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s