path: root/docs/configexamples/dual-hub-dmvpn.rst

########################
Dual-Hub DMVPN with VyOS
########################

DMVPN is a Dynamic Multipoint VPN technology that provides the capability 
for creating a dynamic-mesh VPN network without having to pre-configure 
(static) all possible tunnel end-point peers those simplifying deployment 
and management of the newly added remote sites. There are 3 main protocols 
primarily used to implement DMVPN:

* NHRP - provides the dynamic tunnel endpoint discovery mechanism (endpoint 
  registration, and endpoint discovery/lookup) 
* mGRE - provides the tunnel encapsulation itself 
* IPSec - protocols handle the key exchange, and crypto mechanism

For this example we are using the following devices:

* 2 x Hubs
* 3 x Spokes
* 1 x Client device (VPC)
* 1 x ISP router

The following software was used in the creation of this document:

* Operating system: VyOS
* Version: 1.3-beta-202112090443
* Image name: vyos-1.3-beta-202112090443-amd64.iso



********
Topology
********
.. image:: /_static/images/VyOS_Dual-Hub_DMVPN.png
   :width: 80%
   :align: center
   :alt: Network Topology Diagram



******************************************
Network Addressing and Protocol Parameters
******************************************

The following ip addressing schema used for the devices IPv4 connectivity:

+-----------------------------------------------------------------------------+
|10.X1.0.0/30 - p2p Hubs to ISP networks, where X is Hub site number          |
+-----------------------------------------------------------------------------+
|10.Y1.1.0/24 - p2p Spokes to ISP networks(DHCP), where Y is Spoke site number|
+-----------------------------------------------------------------------------+
|172.16.253.0/29 - tunnels addressing for Hub-1 connections                   |
+-----------------------------------------------------------------------------+
|172.16.254.0/29 - tunnels addressing for Hub-2 connections                   |
+-----------------------------------------------------------------------------+
|192.168.0.0/24 - HQ site local network                                       |
+-----------------------------------------------------------------------------+
|192.168.Z.0/24 - remote sites local network, where Z is Spoke site number    |
+-----------------------------------------------------------------------------+

eBGP parameters for the routers:

+----------------------------------------------+
|AS65000 - HQ (Hub-1 and Hub-2)                |
+----------------------------------------------+
|AS6500X - Spokes, where X is Spoke site number|
+----------------------------------------------+



*************
Configuration
*************



Step-1: Basic connectivity configuration
========================================

- Hub-1:

.. code-block:: none
   
    set interfaces ethernet eth0 address '10.11.0.1/30'
    set interfaces ethernet eth1 address '192.168.0.1/24'
    set protocols static route 0.0.0.0/0 next-hop 10.11.0.2
    set system host-name 'Hub-1'

- Hub-2:

.. code-block:: none
   
    set interfaces ethernet eth0 address '10.21.0.1/30'
    set interfaces ethernet eth1 address '192.168.0.2/24'
    set protocols static route 0.0.0.0/0 next-hop 10.21.0.2
    set system host-name 'Hub-2'

- Spoke-1:

.. code-block:: none
   
    set interfaces ethernet eth0 address 'dhcp'
    set interfaces ethernet eth1 address '192.168.1.1/24'
    set system host-name 'Spoke-1'

- Spoke-2:

.. code-block:: none
   
    set interfaces ethernet eth0 address 'dhcp'
    set interfaces ethernet eth1 address '192.168.2.1/24'
    set system host-name 'Spoke-2'
    
- Spoke-3:

.. code-block:: none
   
    set interfaces ethernet eth0 address 'dhcp'
    set interfaces ethernet eth1 address '192.168.3.1/24'
    set system host-name 'Spoke-3'
    
- ISP-1:

.. code-block:: none
   
    set interfaces ethernet eth0 address '10.11.0.2/30'
    set interfaces ethernet eth1 address '10.21.0.2/30'
    set interfaces ethernet eth2 address '10.31.1.1/24'
    set interfaces ethernet eth3 address '10.21.1.1/24'
    set interfaces ethernet eth4 address '10.11.1.1/24'
    set service dhcp-server shared-network-name SPK-1 authoritative
    set service dhcp-server shared-network-name SPK-1 subnet 10.11.1.0/24 default-router '10.11.1.1'
    set service dhcp-server shared-network-name SPK-1 subnet 10.11.1.0/24 range 1 start '10.11.1.10'
    set service dhcp-server shared-network-name SPK-1 subnet 10.11.1.0/24 range 1 stop '10.11.1.100'
    set service dhcp-server shared-network-name SPK-2 authoritative
    set service dhcp-server shared-network-name SPK-2 subnet 10.21.1.0/24 default-router '10.21.1.1'
    set service dhcp-server shared-network-name SPK-2 subnet 10.21.1.0/24 range 1 start '10.21.1.10'
    set service dhcp-server shared-network-name SPK-2 subnet 10.21.1.0/24 range 1 stop '10.21.1.100'
    set service dhcp-server shared-network-name SPK-3 authoritative
    set service dhcp-server shared-network-name SPK-3 subnet 10.31.1.0/24 default-router '10.31.1.1'
    set service dhcp-server shared-network-name SPK-3 subnet 10.31.1.0/24 range 1 start '10.31.1.10'
    set service dhcp-server shared-network-name SPK-3 subnet 10.31.1.0/24 range 1 stop '10.31.1.100'
    set system host-name 'ISP1'



Step-2: VRRP configuration for HQ Local network redundancy
==========================================================

Here we are using VRRP as a local redundancy protocol between Hub-1 and Hub-2.
Initially, Hub-1 operates as an Active and Hub-2 as a Standby router.
Additionally, health-check and script are used to track uplinks and properly 
switch mastership between Hub nodes based on the upstream router 
reachability (ISP-1). **Note, that before adding local paths to the scripts into 
configuration, you have to create and make them executable first**.

Hub-1 and Hub-2 VRRP health-check script:
_________________________________________

* /config/scripts/vrrp-check.sh

.. code-block:: none
    
    #!/bin/bash

    eth0status="$(cat /sys/class/net/eth0/operstate | grep 'up')"
    
    if [[ ! -z ${eth0status} ]]; then
     eth0gw="$(ip -j r show 0.0.0.0/0 dev eth0 | awk 'match($0, /\"gateway":\"([[:digit:]\.]+)/, gw) {print gw[1]}')"
     if [[ ! -z $eth0gw ]]; then
      /bin/ping -I eth0 -c 1 -W 1 $eth0gw && exit 0 || exit 1
     else
      exit 1
     fi
    else
     #Exit 0 because eth0 down is handled by vrrp transition
     exit 0
    fi


**Note**: some parts of the script might be dependent on your network topology 
and connectivity. Be careful before using it on your own devices.


Hub-1 and Hub-2 VRRP configuration:
___________________________________

* Hub-1

.. code-block:: none
   
    set high-availability vrrp group HQ health-check failure-count '3'
    set high-availability vrrp group HQ health-check interval '1'
    set high-availability vrrp group HQ health-check script '/config/scripts/vrrp-check.sh'
    set high-availability vrrp group HQ interface 'eth1'
    set high-availability vrrp group HQ no-preempt
    set high-availability vrrp group HQ priority '200'
    set high-availability vrrp group HQ rfc3768-compatibility
    set high-availability vrrp group HQ virtual-address '192.168.0.254/24'
    set high-availability vrrp group HQ vrid '1'

* Hub-2:

.. code-block:: none
    
    set high-availability vrrp group HQ health-check failure-count '3'
    set high-availability vrrp group HQ health-check interval '1'
    set high-availability vrrp group HQ health-check script '/config/scripts/vrrp-check.sh'
    set high-availability vrrp group HQ interface 'eth1'
    set high-availability vrrp group HQ no-preempt
    set high-availability vrrp group HQ priority '100'
    set high-availability vrrp group HQ rfc3768-compatibility
    set high-availability vrrp group HQ virtual-address '192.168.0.254/24'
    set high-availability vrrp group HQ vrid '1'



Step-3: DMVPN configuration between Hub and Spoke devices
=========================================================

This section provides an example configuration of the DMVPN enabled devices. 
Hub devices are configured with static IPv4 addresses on the uplink interfaces 
while Spoke devices receive addresses dynamically from a pre-defined DHCP 
pool configured on ISP router. For redundancy purposes, we use 1 tunnel 
interface on each Hub device and 2 tunnel interfaces on Spoke devices 
destined to each of the Hubs. For the optimal tunnel operation timers are 
significantly decreased and set to the following values:

**NHRP** tunnel holding time - 30 seconds

**IKE DPD** enabled with "restart" action set, interval 3 and timeout 
30 seconds

**Note**: these values are used only for the lab demonstration and may not 
suit exclusive production networks.

- Hub-1:

.. code-block:: none
   
    set interfaces tunnel tun100 address '172.16.253.134/29'
    set interfaces tunnel tun100 encapsulation 'gre'
    set interfaces tunnel tun100 multicast 'enable'
    set interfaces tunnel tun100 parameters ip key '1'
    set interfaces tunnel tun100 source-address '10.11.0.1'
    
    set protocols nhrp tunnel tun100 cisco-authentication 'secret'
    set protocols nhrp tunnel tun100 holding-time '30'
    set protocols nhrp tunnel tun100 multicast 'dynamic'
    set protocols nhrp tunnel tun100 redirect
    set protocols nhrp tunnel tun100 shortcut
    
    set vpn ipsec esp-group ESP-HUB compression 'disable'
    set vpn ipsec esp-group ESP-HUB lifetime '1800'
    set vpn ipsec esp-group ESP-HUB mode 'transport'
    set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
    set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
    set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
    set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
    set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
    set vpn ipsec ike-group IKE-HUB close-action 'none'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
    set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
    set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
    set vpn ipsec ike-group IKE-HUB lifetime '3600'
    set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
    set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
    set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
    set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
    set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
    set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
    set vpn ipsec ipsec-interfaces interface 'eth0'
    set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
    set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
    set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
    set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
    set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'

- Hub-2:

.. code-block:: none
   
    set interfaces tunnel tun100 address '172.16.254.134/29'
    set interfaces tunnel tun100 encapsulation 'gre'
    set interfaces tunnel tun100 multicast 'enable'
    set interfaces tunnel tun100 parameters ip key '2'
    set interfaces tunnel tun100 source-address '10.21.0.1'
    
    set protocols nhrp tunnel tun100 cisco-authentication 'secret'
    set protocols nhrp tunnel tun100 holding-time '30'
    set protocols nhrp tunnel tun100 multicast 'dynamic'
    set protocols nhrp tunnel tun100 redirect
    set protocols nhrp tunnel tun100 shortcut
    
    set vpn ipsec esp-group ESP-HUB compression 'disable'
    set vpn ipsec esp-group ESP-HUB lifetime '1800'
    set vpn ipsec esp-group ESP-HUB mode 'transport'
    set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
    set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
    set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
    set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
    set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
    set vpn ipsec ike-group IKE-HUB close-action 'none'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
    set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
    set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
    set vpn ipsec ike-group IKE-HUB lifetime '3600'
    set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
    set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
    set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
    set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
    set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
    set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
    set vpn ipsec ipsec-interfaces interface 'eth0'
    set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
    set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
    set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
    set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
    set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
    
- Spoke-1:

.. code-block:: none
   
    set interfaces tunnel tun100 address '172.16.253.131/29'
    set interfaces tunnel tun100 encapsulation 'gre'
    set interfaces tunnel tun100 multicast 'enable'
    set interfaces tunnel tun100 parameters ip key '1'
    set interfaces tunnel tun100 source-address '0.0.0.0'
    set interfaces tunnel tun200 address '172.16.254.131/29'
    set interfaces tunnel tun200 encapsulation 'gre'
    set interfaces tunnel tun200 multicast 'enable'
    set interfaces tunnel tun200 parameters ip key '2'
    set interfaces tunnel tun200 source-address '0.0.0.0'
    
    set protocols nhrp tunnel tun100 cisco-authentication 'secret'
    set protocols nhrp tunnel tun100 holding-time '30'
    set protocols nhrp tunnel tun100 map 172.16.253.134/29 nbma-address '10.11.0.1'
    set protocols nhrp tunnel tun100 map 172.16.253.134/29 register
    set protocols nhrp tunnel tun100 multicast 'nhs'
    set protocols nhrp tunnel tun100 redirect
    set protocols nhrp tunnel tun100 shortcut
    set protocols nhrp tunnel tun200 cisco-authentication 'secret'
    set protocols nhrp tunnel tun200 holding-time '30'
    set protocols nhrp tunnel tun200 map 172.16.254.134/29 nbma-address '10.21.0.1'
    set protocols nhrp tunnel tun200 map 172.16.254.134/29 register
    set protocols nhrp tunnel tun200 multicast 'nhs'
    set protocols nhrp tunnel tun200 redirect
    set protocols nhrp tunnel tun200 shortcut
    
    set vpn ipsec esp-group ESP-HUB compression 'disable'
    set vpn ipsec esp-group ESP-HUB lifetime '1800'
    set vpn ipsec esp-group ESP-HUB mode 'transport'
    set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
    set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
    set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
    set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
    set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
    set vpn ipsec ike-group IKE-HUB close-action 'none'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
    set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
    set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
    set vpn ipsec ike-group IKE-HUB lifetime '3600'
    set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
    set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
    set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
    set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
    set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
    set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
    set vpn ipsec ipsec-interfaces interface 'eth0'
    set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
    set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
    set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
    set vpn ipsec profile NHRPVPN bind tunnel 'tun200'
    set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
    set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
    
- Spoke-2:

.. code-block:: none
   
    set interfaces tunnel tun100 address '172.16.253.132/29'
    set interfaces tunnel tun100 encapsulation 'gre'
    set interfaces tunnel tun100 multicast 'enable'
    set interfaces tunnel tun100 parameters ip key '1'
    set interfaces tunnel tun100 source-address '0.0.0.0'
    set interfaces tunnel tun200 address '172.16.254.132/29'
    set interfaces tunnel tun200 encapsulation 'gre'
    set interfaces tunnel tun200 multicast 'enable'
    set interfaces tunnel tun200 parameters ip key '2'
    set interfaces tunnel tun200 source-address '0.0.0.0'
    
    set protocols nhrp tunnel tun100 cisco-authentication 'secret'
    set protocols nhrp tunnel tun100 holding-time '30'
    set protocols nhrp tunnel tun100 map 172.16.253.134/29 nbma-address '10.11.0.1'
    set protocols nhrp tunnel tun100 map 172.16.253.134/29 register
    set protocols nhrp tunnel tun100 multicast 'nhs'
    set protocols nhrp tunnel tun100 redirect
    set protocols nhrp tunnel tun100 shortcut
    set protocols nhrp tunnel tun200 cisco-authentication 'secret'
    set protocols nhrp tunnel tun200 holding-time '30'
    set protocols nhrp tunnel tun200 map 172.16.254.134/29 nbma-address '10.21.0.1'
    set protocols nhrp tunnel tun200 map 172.16.254.134/29 register
    set protocols nhrp tunnel tun200 multicast 'nhs'
    set protocols nhrp tunnel tun200 redirect
    set protocols nhrp tunnel tun200 shortcut
    
    set vpn ipsec esp-group ESP-HUB compression 'disable'
    set vpn ipsec esp-group ESP-HUB lifetime '1800'
    set vpn ipsec esp-group ESP-HUB mode 'transport'
    set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
    set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
    set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
    set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
    set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
    set vpn ipsec ike-group IKE-HUB close-action 'none'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
    set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
    set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
    set vpn ipsec ike-group IKE-HUB lifetime '3600'
    set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
    set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
    set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
    set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
    set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
    set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
    set vpn ipsec ipsec-interfaces interface 'eth0'
    set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
    set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
    set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
    set vpn ipsec profile NHRPVPN bind tunnel 'tun200'
    set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
    set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
    
- Spoke-3:

.. code-block:: none
   
    set interfaces tunnel tun100 address '172.16.253.133/29'
    set interfaces tunnel tun100 encapsulation 'gre'
    set interfaces tunnel tun100 multicast 'enable'
    set interfaces tunnel tun100 parameters ip key '1'
    set interfaces tunnel tun100 source-address '0.0.0.0'
    set interfaces tunnel tun200 address '172.16.254.133/29'
    set interfaces tunnel tun200 encapsulation 'gre'
    set interfaces tunnel tun200 multicast 'enable'
    set interfaces tunnel tun200 parameters ip key '2'
    set interfaces tunnel tun200 source-address '0.0.0.0'
    
    set protocols nhrp tunnel tun100 cisco-authentication 'secret'
    set protocols nhrp tunnel tun100 holding-time '30'
    set protocols nhrp tunnel tun100 map 172.16.253.134/29 nbma-address '10.11.0.1'
    set protocols nhrp tunnel tun100 map 172.16.253.134/29 register
    set protocols nhrp tunnel tun100 multicast 'nhs'
    set protocols nhrp tunnel tun100 redirect
    set protocols nhrp tunnel tun100 shortcut
    set protocols nhrp tunnel tun200 cisco-authentication 'secret'
    set protocols nhrp tunnel tun200 holding-time '30'
    set protocols nhrp tunnel tun200 map 172.16.254.134/29 nbma-address '10.21.0.1'
    set protocols nhrp tunnel tun200 map 172.16.254.134/29 register
    set protocols nhrp tunnel tun200 multicast 'nhs'
    set protocols nhrp tunnel tun200 redirect
    set protocols nhrp tunnel tun200 shortcut
    
    set vpn ipsec esp-group ESP-HUB compression 'disable'
    set vpn ipsec esp-group ESP-HUB lifetime '1800'
    set vpn ipsec esp-group ESP-HUB mode 'transport'
    set vpn ipsec esp-group ESP-HUB pfs 'dh-group2'
    set vpn ipsec esp-group ESP-HUB proposal 1 encryption 'aes256'
    set vpn ipsec esp-group ESP-HUB proposal 1 hash 'sha1'
    set vpn ipsec esp-group ESP-HUB proposal 2 encryption '3des'
    set vpn ipsec esp-group ESP-HUB proposal 2 hash 'md5'
    set vpn ipsec ike-group IKE-HUB close-action 'none'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection action 'restart'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection interval '3'
    set vpn ipsec ike-group IKE-HUB dead-peer-detection timeout '30'
    set vpn ipsec ike-group IKE-HUB ikev2-reauth 'no'
    set vpn ipsec ike-group IKE-HUB key-exchange 'ikev2'
    set vpn ipsec ike-group IKE-HUB lifetime '3600'
    set vpn ipsec ike-group IKE-HUB proposal 1 dh-group '2'
    set vpn ipsec ike-group IKE-HUB proposal 1 encryption 'aes256'
    set vpn ipsec ike-group IKE-HUB proposal 1 hash 'sha1'
    set vpn ipsec ike-group IKE-HUB proposal 2 dh-group '2'
    set vpn ipsec ike-group IKE-HUB proposal 2 encryption 'aes128'
    set vpn ipsec ike-group IKE-HUB proposal 2 hash 'sha1'
    set vpn ipsec ipsec-interfaces interface 'eth0'
    set vpn ipsec profile NHRPVPN authentication mode 'pre-shared-secret'
    set vpn ipsec profile NHRPVPN authentication pre-shared-secret 'secret'
    set vpn ipsec profile NHRPVPN bind tunnel 'tun100'
    set vpn ipsec profile NHRPVPN bind tunnel 'tun200'
    set vpn ipsec profile NHRPVPN esp-group 'ESP-HUB'
    set vpn ipsec profile NHRPVPN ike-group 'IKE-HUB'
    


Step-4: Enabling eBGP as a Dynamic Routing Protocol between Hubs and Spokes
===========================================================================

For the simplified and better network management we're using eBGP for routing 
information exchange between devices. As we're using Active-Standby mode in 
this example, Hub-2 is configured with AS-prepand as an export route-policy 
and VRRP transition scripts are used for switching mastership based on the 
current link/device state. Also, we use multihop BFD for faster eBGP failure 
detection.

Hub-1 and Hub-2 VRRP transition scripts:
________________________________________

* /config/scripts/vrrp-master.sh

.. code-block:: none
    
    #!/bin/vbash

    if [ $(id -gn) != vyattacfg ]; then
        exec sg vyattacfg "$0 $*"
    fi
    
    source /opt/vyatta/etc/functions/script-template
    
    configure
    delete protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export AS65000-PREP
    commit
    
    exit


* /config/scripts/vrrp-fail.sh

.. code-block:: none
    
    #!/bin/vbash

    if [ $(id -gn) != vyattacfg ]; then
        exec sg vyattacfg "$0 $*"
    fi
    
    source /opt/vyatta/etc/functions/script-template
    
    configure
    set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export AS65000-PREP
    commit
    
    exit


**Note**: some parts of the script might be dependent on your network topology 
and connectivity. Be careful before using it on your own devices.


Hub devices configuration:
__________________________

- Hub-1:

.. code-block:: none
   
    set high-availability vrrp group HQ transition-script backup '/config/scripts/vrrp-fail.sh'
    set high-availability vrrp group HQ transition-script fault '/config/scripts/vrrp-fail.sh'
    set high-availability vrrp group HQ transition-script master '/config/scripts/vrrp-master.sh'
    set high-availability vrrp group HQ transition-script stop '/config/scripts/vrrp-fail.sh'
    
    set policy route-map AS65000-PREP rule 1 action 'permit'
    set policy route-map AS65000-PREP rule 1 set as-path-prepend '65000 65000 65000'
    
    set protocols bfd peer 172.16.253.131 interval multiplier '3'
    set protocols bfd peer 172.16.253.131 interval receive '300'
    set protocols bfd peer 172.16.253.131 interval transmit '300'
    set protocols bfd peer 172.16.253.131 multihop
    set protocols bfd peer 172.16.253.131 source address '172.16.253.134'
    set protocols bfd peer 172.16.253.132 interval multiplier '3'
    set protocols bfd peer 172.16.253.132 interval receive '300'
    set protocols bfd peer 172.16.253.132 interval transmit '300'
    set protocols bfd peer 172.16.253.132 multihop
    set protocols bfd peer 172.16.253.132 source address '172.16.253.134'
    set protocols bfd peer 172.16.253.133 interval multiplier '3'
    set protocols bfd peer 172.16.253.133 interval receive '300'
    set protocols bfd peer 172.16.253.133 interval transmit '300'
    set protocols bfd peer 172.16.253.133 multihop
    set protocols bfd peer 172.16.253.133 source address '172.16.253.134'
    
    set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
    set protocols bgp 65000 neighbor 172.16.253.131 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.253.131 remote-as '65001'
    set protocols bgp 65000 neighbor 172.16.253.132 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.253.132 remote-as '65002'
    set protocols bgp 65000 neighbor 172.16.253.133 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.253.133 remote-as '65003'
    set protocols bgp 65000 parameters log-neighbor-changes
    set protocols bgp 65000 parameters network-import-check
    set protocols bgp 65000 peer-group DMVPN bfd

- Hub-2:

.. code-block:: none
   
    set high-availability vrrp group HQ transition-script backup '/config/scripts/vrrp-fail.sh'
    set high-availability vrrp group HQ transition-script fault '/config/scripts/vrrp-fail.sh'
    set high-availability vrrp group HQ transition-script master '/config/scripts/vrrp-master.sh'
    set high-availability vrrp group HQ transition-script stop '/config/scripts/vrrp-fail.sh'
    
    set policy route-map AS65000-PREP rule 1 action 'permit'
    set policy route-map AS65000-PREP rule 1 set as-path-prepend '65000 65000 65000'
    
    set protocols bfd peer 172.16.254.131 interval multiplier '3'
    set protocols bfd peer 172.16.254.131 interval receive '300'
    set protocols bfd peer 172.16.254.131 interval transmit '300'
    set protocols bfd peer 172.16.254.131 multihop
    set protocols bfd peer 172.16.254.131 source address '172.16.254.134'
    set protocols bfd peer 172.16.254.132 interval multiplier '3'
    set protocols bfd peer 172.16.254.132 interval receive '300'
    set protocols bfd peer 172.16.254.132 interval transmit '300'
    set protocols bfd peer 172.16.254.132 multihop
    set protocols bfd peer 172.16.254.132 source address '172.16.254.134'
    set protocols bfd peer 172.16.254.133 interval multiplier '3'
    set protocols bfd peer 172.16.254.133 interval receive '300'
    set protocols bfd peer 172.16.254.133 interval transmit '300'
    set protocols bfd peer 172.16.254.133 multihop
    set protocols bfd peer 172.16.254.133 source address '172.16.254.134'
    
    set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
    set protocols bgp 65000 neighbor 172.16.254.131 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.254.131 remote-as '65001'
    set protocols bgp 65000 neighbor 172.16.254.132 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.254.132 remote-as '65002'
    set protocols bgp 65000 neighbor 172.16.254.133 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.254.133 remote-as '65003'
    set protocols bgp 65000 parameters log-neighbor-changes
    set protocols bgp 65000 parameters network-import-check
    set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export 'AS65000-PREP'
    set protocols bgp 65000 peer-group DMVPN bfd
    
Spoke devices configuration:
____________________________

- Spoke-1:

.. code-block:: none
   
    set protocols bfd peer 172.16.253.134 interval multiplier '3'
    set protocols bfd peer 172.16.253.134 interval receive '300'
    set protocols bfd peer 172.16.253.134 interval transmit '300'
    set protocols bfd peer 172.16.253.134 multihop
    set protocols bfd peer 172.16.253.134 source address '172.16.253.131'
    set protocols bfd peer 172.16.254.134 interval multiplier '3'
    set protocols bfd peer 172.16.254.134 interval receive '300'
    set protocols bfd peer 172.16.254.134 interval transmit '300'
    set protocols bfd peer 172.16.254.134 multihop
    set protocols bfd peer 172.16.254.134 source address '172.16.254.131'
    
    set protocols bgp 65001 address-family ipv4-unicast network 192.168.1.0/24
    set protocols bgp 65001 neighbor 172.16.253.134 address-family ipv4-unicast
    set protocols bgp 65001 neighbor 172.16.253.134 bfd
    set protocols bgp 65001 neighbor 172.16.253.134 remote-as '65000'
    set protocols bgp 65001 neighbor 172.16.254.134 address-family ipv4-unicast
    set protocols bgp 65001 neighbor 172.16.254.134 bfd
    set protocols bgp 65001 neighbor 172.16.254.134 remote-as '65000'
    set protocols bgp 65001 parameters log-neighbor-changes
    
- Spoke-2:

.. code-block:: none
   
    set protocols bfd peer 172.16.253.134 interval multiplier '3'
    set protocols bfd peer 172.16.253.134 interval receive '300'
    set protocols bfd peer 172.16.253.134 interval transmit '300'
    set protocols bfd peer 172.16.253.134 multihop
    set protocols bfd peer 172.16.253.134 source address '172.16.253.132'
    set protocols bfd peer 172.16.254.134 interval multiplier '3'
    set protocols bfd peer 172.16.254.134 interval receive '300'
    set protocols bfd peer 172.16.254.134 interval transmit '300'
    set protocols bfd peer 172.16.254.134 multihop
    set protocols bfd peer 172.16.254.134 source address '172.16.254.132'
    
    set protocols bgp 65002 address-family ipv4-unicast network 192.168.2.0/24
    set protocols bgp 65002 neighbor 172.16.253.134 address-family ipv4-unicast
    set protocols bgp 65002 neighbor 172.16.253.134 bfd
    set protocols bgp 65002 neighbor 172.16.253.134 remote-as '65000'
    set protocols bgp 65002 neighbor 172.16.254.134 address-family ipv4-unicast
    set protocols bgp 65002 neighbor 172.16.254.134 bfd
    set protocols bgp 65002 neighbor 172.16.254.134 remote-as '65000'
    set protocols bgp 65002 parameters log-neighbor-changes
    
- Spoke-3:

.. code-block:: none
   
    set protocols bfd peer 172.16.253.134 interval multiplier '3'
    set protocols bfd peer 172.16.253.134 interval receive '300'
    set protocols bfd peer 172.16.253.134 interval transmit '300'
    set protocols bfd peer 172.16.253.134 multihop
    set protocols bfd peer 172.16.253.134 source address '172.16.253.133'
    set protocols bfd peer 172.16.254.134 interval multiplier '3'
    set protocols bfd peer 172.16.254.134 interval receive '300'
    set protocols bfd peer 172.16.254.134 interval transmit '300'
    set protocols bfd peer 172.16.254.134 multihop
    set protocols bfd peer 172.16.254.134 source address '172.16.254.133'
    
    set protocols bgp 65003 address-family ipv4-unicast network 192.168.3.0/24
    set protocols bgp 65003 neighbor 172.16.253.134 address-family ipv4-unicast
    set protocols bgp 65003 neighbor 172.16.253.134 bfd
    set protocols bgp 65003 neighbor 172.16.253.134 remote-as '65000'
    set protocols bgp 65003 neighbor 172.16.254.134 address-family ipv4-unicast
    set protocols bgp 65003 neighbor 172.16.254.134 bfd
    set protocols bgp 65003 neighbor 172.16.254.134 remote-as '65000'
    set protocols bgp 65003 parameters log-neighbor-changes
    
**Note**: In case if you're using VyOS version that has a VRRP transition 
scripts issues after a device reboot, as a temporary solution you may add
postconfig-bootup script that reloads **keepalived** process additionally after 
the device booted.

- Hub devices /config/scripts/vyos-postconfig-bootup.script:

.. code-block:: none
   
    #!/bin/sh
    # This script is executed at boot time after VyOS configuration is fully applied.
    # Any modifications required to work around unfixed bugs
    # or use services not available through the VyOS CLI system can be placed here.
    
    echo "Reloading VRRP process"
    sudo systemctl restart keepalived.service
    echo "VRRP process reload completed"



Step-5: Verification
====================

Now, it's time to check that all protocols are working as expected and mastership 
during the failover switches correctly between Hub devices.

- Checking VRRP state between Hub-1 and Hub-2:

.. code-block:: none
   
    vyos@Hub-1:~$ show vrrp
    Name    Interface      VRID  State      Priority  Last Transition
    ------  -----------  ------  -------  ----------  -----------------
    HQ      eth1v1            1  MASTER          200  14s
    
    vyos@Hub-2:~$ show vrrp
    Name    Interface      VRID  State      Priority  Last Transition
    ------  -----------  ------  -------  ----------  -----------------
    HQ      eth1v1            1  BACKUP          100  29s

- Checking NHRP and eBGP sessions between Hub and Spoke devices:

.. code-block:: none
   
    vyos@Hub-1:~$ show nhrp tunnel
    Status: ok
    
    Interface: tun100
    Type: local
    Protocol-Address: 172.16.253.135/32
    Alias-Address: 172.16.253.134
    Flags: up
    
    Interface: tun100
    Type: local
    Protocol-Address: 172.16.253.134/32
    Flags: up
    
    Interface: tun100
    Type: dynamic
    Protocol-Address: 172.16.253.131/32
    NBMA-Address: 10.11.1.11
    Flags: up
    Expires-In: 0:23
    
    Interface: tun100
    Type: dynamic
    Protocol-Address: 172.16.253.133/32
    NBMA-Address: 10.31.1.11
    Flags: up
    Expires-In: 0:22
    
    Interface: tun100
    Type: dynamic
    Protocol-Address: 172.16.253.132/32
    NBMA-Address: 10.21.1.11
    Flags: up
    Expires-In: 0:21
    
    vyos@Hub-1:~$ show bgp summary
    
    IPv4 Unicast Summary:
    BGP router identifier 192.168.0.1, local AS number 65000 vrf-id 0
    BGP table version 20
    RIB entries 7, using 1344 bytes of memory
    Peers 3, using 64 KiB of memory
    Peer groups 1, using 64 bytes of memory
    
    Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
    172.16.253.131  4      65001     26519     26526        0    0    0 00:43:38            1        4
    172.16.253.132  4      65002     26545     26540        0    0    0 00:46:36            1        4
    172.16.253.133  4      65003     26528     26520        0    0    0 00:41:59            1        4
    
    Total number of neighbors 3
    
    
    vyos@Hub-2:~$ show nhrp tunnel
    Status: ok
    
    Interface: tun100
    Type: local
    Protocol-Address: 172.16.254.135/32
    Alias-Address: 172.16.254.134
    Flags: up
    
    Interface: tun100
    Type: local
    Protocol-Address: 172.16.254.134/32
    Flags: up
    
    Interface: tun100
    Type: dynamic
    Protocol-Address: 172.16.254.132/32
    NBMA-Address: 10.21.1.11
    Flags: up
    Expires-In: 0:28
    
    Interface: tun100
    Type: dynamic
    Protocol-Address: 172.16.254.131/32
    NBMA-Address: 10.11.1.11
    Flags: up
    Expires-In: 0:21
    
    Interface: tun100
    Type: dynamic
    Protocol-Address: 172.16.254.133/32
    NBMA-Address: 10.31.1.11
    Flags: up
    Expires-In: 0:20
    
    vyos@Hub-2:~$ show bgp summary
    
    IPv4 Unicast Summary:
    BGP router identifier 192.168.0.2, local AS number 65000 vrf-id 0
    BGP table version 14
    RIB entries 7, using 1344 bytes of memory
    Peers 3, using 64 KiB of memory
    Peer groups 1, using 64 bytes of memory
    
    Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
    172.16.254.131  4      65001     26516     26516        0    0    0 00:43:03            1        4
    172.16.254.132  4      65002     26563     26562        0    0    0 00:48:27            1        4
    172.16.254.133  4      65003     26518     26516        0    0    0 00:42:20            1        4
    
    Total number of neighbors 3
    
- Checking BFD sessions between Hub and Spoke devices:

.. code-block:: none
   
    vyos@Hub-1:~$ show protocols bfd peers
    Session count: 6
    SessionId  LocalAddress                             PeerAddress                             Status
    =========  ============                             ===========                             ======
    3600626867 172.16.253.134                           172.16.253.133                          up
    1123939978 172.16.253.134                           172.16.253.131                          up
    374394280  172.16.253.134                           172.16.253.132                          up
    1786735466 172.16.253.134                           172.16.253.132                          up
    1440522544 172.16.253.134                           172.16.253.131                          up
    1106910911 172.16.253.134                           172.16.253.133                          up
    
    
    vyos@Hub-2:~$ show protocols bfd peers
    Session count: 6
    SessionId  LocalAddress                             PeerAddress                             Status
    =========  ============                             ===========                             ======
    2442966178 172.16.254.134                           172.16.254.133                          up
    393258775  172.16.254.134                           172.16.254.131                          up
    2990308682 172.16.254.134                           172.16.254.133                          up
    2267910949 172.16.254.134                           172.16.254.132                          up
    3542474595 172.16.254.134                           172.16.254.131                          up
    4239538185 172.16.254.134                           172.16.254.132                          up

- Checking routing information and connectivity between Hub and Spoke devices:

.. code-block:: none
   
    vyos@Hub-1:~$ show ip bgp
    BGP table version is 20, local router ID is 192.168.0.1, vrf id 0
    Default local pref 100, local AS 65000
    Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                   i internal, r RIB-failure, S Stale, R Removed
    Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
    Origin codes:  i - IGP, e - EGP, ? - incomplete
    
       Network          Next Hop            Metric LocPrf Weight Path
    *> 192.168.0.0/24   0.0.0.0                  0         32768 i
    *> 192.168.1.0/24   172.16.253.131           0             0 65001 i
    *> 192.168.2.0/24   172.16.253.132           0             0 65002 i
    *> 192.168.3.0/24   172.16.253.133           0             0 65003 i
    
    Displayed  4 routes and 4 total paths


    vyos@Hub-2:~$ show ip bgp
    BGP table version is 14, local router ID is 192.168.0.2, vrf id 0
    Default local pref 100, local AS 65000
    Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                   i internal, r RIB-failure, S Stale, R Removed
    Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
    Origin codes:  i - IGP, e - EGP, ? - incomplete
    
       Network          Next Hop            Metric LocPrf Weight Path
    *> 192.168.0.0/24   0.0.0.0                  0         32768 i
    *> 192.168.1.0/24   172.16.254.131           0             0 65001 i
    *> 192.168.2.0/24   172.16.254.132           0             0 65002 i
    *> 192.168.3.0/24   172.16.254.133           0             0 65003 i
    
    Displayed  4 routes and 4 total paths


    vyos@Spoke-1:~$ show ip bgp
    BGP table version is 19, local router ID is 192.168.1.1, vrf id 0
    Default local pref 100, local AS 65001
    Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                   i internal, r RIB-failure, S Stale, R Removed
    Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
    Origin codes:  i - IGP, e - EGP, ? - incomplete
    
       Network          Next Hop            Metric LocPrf Weight Path
    *  192.168.0.0/24   172.16.254.134           0             0 65000 65000 65000 65000 i
    *>                  172.16.253.134           0             0 65000 i
    *> 192.168.1.0/24   0.0.0.0                  0         32768 i
    *  192.168.2.0/24   172.16.254.132                         0 65000 65000 65000 65000 65002 i
    *>                  172.16.253.132                         0 65000 65002 i
    *  192.168.3.0/24   172.16.254.133                         0 65000 65000 65000 65000 65003 i
    *>                  172.16.253.133                         0 65000 65003 i
    
    Displayed  4 routes and 7 total paths

As you can see, Hub-2 announces routes with longer(prepended) AS path as 
we've configured it previously, those, traffic towards HQ subnet will be 
forwarded over Hub-1 which is operating as an Active VRRP router. Let's 
check connectivity and the path from Spoke-1 to the HQ local network:

.. code-block:: none
   
    vyos@Spoke-1:~$ ping 192.168.0.10 count 5 interface 192.168.1.1
    PING 192.168.0.10 (192.168.0.10) from 192.168.1.1 : 56(84) bytes of data.
    64 bytes from 192.168.0.10: icmp_seq=1 ttl=63 time=3.50 ms
    64 bytes from 192.168.0.10: icmp_seq=2 ttl=63 time=2.45 ms
    64 bytes from 192.168.0.10: icmp_seq=3 ttl=63 time=2.34 ms
    64 bytes from 192.168.0.10: icmp_seq=4 ttl=63 time=2.20 ms
    64 bytes from 192.168.0.10: icmp_seq=5 ttl=63 time=2.44 ms
    
    --- 192.168.0.10 ping statistics ---
    5 packets transmitted, 5 received, 0% packet loss, time 11ms
    rtt min/avg/max/mdev = 2.195/2.583/3.496/0.465 ms
    
    vyos@Spoke-1:~$ traceroute 192.168.0.10
    traceroute to 192.168.0.10 (192.168.0.10), 30 hops max, 60 byte packets
     1  172.16.253.134 (172.16.253.134)  0.913 ms  0.884 ms  0.819 ms
     2  192.168.0.10 (192.168.0.10)  1.352 ms  1.446 ms  1.391 ms

From the output, we can confirm successful connectivity between Spoke-1 and HQ 
local networks. From the traceroute we see that the traffic pass through the 
Hub-1.

Now, let's check traffic between Spoke sites. Based on our configuration, Spoke 
sites are using shortcut for direct reachability between each other. First, let's 
check NHRP tunnels before passing the traffic between Spoke-1 and Spoke-2:

.. code-block:: none
   
    vyos@Spoke-1:~$ show nhrp tunnel
    Status: ok
    
    Interface: tun200
    Type: local
    Protocol-Address: 172.16.254.135/32
    Alias-Address: 172.16.254.131
    Flags: up
    
    Interface: tun200
    Type: local
    Protocol-Address: 172.16.254.131/32
    Flags: up
    
    Interface: tun100
    Type: local
    Protocol-Address: 172.16.253.135/32
    Alias-Address: 172.16.253.131
    Flags: up
    
    Interface: tun100
    Type: local
    Protocol-Address: 172.16.253.131/32
    Flags: up
    
    Interface: tun200
    Type: static
    Protocol-Address: 172.16.254.134/29
    NBMA-Address: 10.21.0.1
    Flags: used up
    
    Interface: tun100
    Type: static
    Protocol-Address: 172.16.253.134/29
    NBMA-Address: 10.11.0.1
    Flags: used up

    vyos@Spoke-2:~$ show nhrp tunnel
    Status: ok
    
    Interface: tun100
    Type: local
    Protocol-Address: 172.16.253.135/32
    Alias-Address: 172.16.253.132
    Flags: up
    
    Interface: tun100
    Type: local
    Protocol-Address: 172.16.253.132/32
    Flags: up
    
    Interface: tun200
    Type: local
    Protocol-Address: 172.16.254.135/32
    Alias-Address: 172.16.254.132
    Flags: up
    
    Interface: tun200
    Type: local
    Protocol-Address: 172.16.254.132/32
    Flags: up
    
    Interface: tun100
    Type: static
    Protocol-Address: 172.16.253.134/29
    NBMA-Address: 10.11.0.1
    Flags: used up
    
    Interface: tun200
    Type: static
    Protocol-Address: 172.16.254.134/29
    NBMA-Address: 10.21.0.1


After passing traffic we could see that there is additional shortcut tunnel 
created between Spoke-1 and Spoke-2 for the direct communication:

.. code-block:: none
   
    vyos@Spoke-1:~$ ping 192.168.2.1 count 5 interface 192.168.1.1
    PING 192.168.2.1 (192.168.2.1) from 192.168.1.1 : 56(84) bytes of data.
    64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=1.03 ms
    64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=0.820 ms
    64 bytes from 192.168.2.1: icmp_seq=3 ttl=64 time=1.13 ms
    64 bytes from 192.168.2.1: icmp_seq=4 ttl=63 time=1.41 ms
    64 bytes from 192.168.2.1: icmp_seq=5 ttl=64 time=0.988 ms
    
    --- 192.168.2.1 ping statistics ---
    5 packets transmitted, 5 received, 0% packet loss, time 10ms
    rtt min/avg/max/mdev = 0.820/1.075/1.412/0.197 ms
    
    vyos@Spoke-1:~$ traceroute 192.168.2.1
    traceroute to 192.168.2.1 (192.168.2.1), 30 hops max, 60 byte packets
     1  192.168.2.1 (192.168.2.1)  1.172 ms  1.109 ms  1.151 ms

    vyos@Spoke-1:~$ show nhrp tunnel
    Status: ok
    
    Interface: tun200
    Type: local
    Protocol-Address: 172.16.254.135/32
    Alias-Address: 172.16.254.131
    Flags: up
    
    Interface: tun200
    Type: local
    Protocol-Address: 172.16.254.131/32
    Flags: up
    
    Interface: tun100
    Type: local
    Protocol-Address: 172.16.253.135/32
    Alias-Address: 172.16.253.131
    Flags: up
    
    Interface: tun100
    Type: local
    Protocol-Address: 172.16.253.131/32
    Flags: up
    
    Interface: tun200
    Type: static
    Protocol-Address: 172.16.254.134/29
    NBMA-Address: 10.21.0.1
    Flags: used up
    
    ____________________________________
    Interface: tun100
    Type: cached
    Protocol-Address: 172.16.253.132/32
    NBMA-Address: 10.21.1.11
    Flags: used up
    Expires-In: 0:24
    ____________________________________
    
    Interface: tun100
    Type: static
    Protocol-Address: 172.16.253.134/29
    NBMA-Address: 10.11.0.1
    Flags: used up

The same applies to the rest of the devices and works with the same logic. 
As we've already confirmed successfull connectivity between Hub and Spoke 
devices, let's check failover process.

- Failover on the health-check failure on Hub-1:
  
.. code-block:: none
   
    # disabling interface towards Hub-1 on ISP router
    vyos@ISP1:~$ configure
    [edit]
    vyos@ISP1# set interfaces ethernet eth0 disable
    [edit]
    vyos@ISP1# commit
    [edit]
    vyos@ISP1#


    # checking VRRP state and eBGP configuration on Hub-1:
    vyos@Hub-1:~$ show vrrp
    Name    Interface      VRID  State      Priority  Last Transition
    ------  -----------  ------  -------  ----------  -----------------
    HQ      eth1v1            1  FAULT           200  1m15s
    
    vyos@Hub-1:~$ show configuration commands | match bgp
    set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
    set protocols bgp 65000 neighbor 172.16.253.131 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.253.131 remote-as '65001'
    set protocols bgp 65000 neighbor 172.16.253.132 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.253.132 remote-as '65002'
    set protocols bgp 65000 neighbor 172.16.253.133 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.253.133 remote-as '65003'
    set protocols bgp 65000 parameters log-neighbor-changes
    set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export 'AS65000-PREP'
    set protocols bgp 65000 peer-group DMVPN bfd


    # consecutive pings check from Spoke-1 to the HQ local network during the failure
    --- 192.168.0.10 ping statistics ---
    223 packets transmitted, 219 received, 1.79372% packet loss, time 679ms
    rtt min/avg/max/mdev = 0.918/2.191/2.957/0.364 ms
    vyos@Spoke-1:~$


    # consecutive pings check from Spoke-3 to the Spoke-2 local network during the failure
    --- 192.168.2.1 ping statistics ---
    265 packets transmitted, 265 received, 0% packet loss, time 690ms
    rtt min/avg/max/mdev = 0.663/1.128/2.272/0.285 ms
    vyos@Spoke-3:~$

**Note**: After bringing ISP interface towards Hub-1 back to UP state, 
VRRP state will remain unchanged due to "no-preempt" option enabled 
under the VRRP configuration on the Hub-1 and Hub-2 and will be changed 
only during link/device failure on Hub-2.
    
- Failover during Hub-2 device failure:

.. code-block:: none
   
    # Checking VRRP state and eBGP configuration on Hub-2 before reboot
    vyos@Hub-2:~$ show vrrp
    Name    Interface      VRID  State      Priority  Last Transition
    ------  -----------  ------  -------  ----------  -----------------
    HQ      eth1v1            1  MASTER          100  20m22s

    vyos@Hub-2:~$ show configuration commands | match bgp
    set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
    set protocols bgp 65000 neighbor 172.16.254.131 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.254.131 remote-as '65001'
    set protocols bgp 65000 neighbor 172.16.254.132 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.254.132 remote-as '65002'
    set protocols bgp 65000 neighbor 172.16.254.133 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.254.133 remote-as '65003'
    set protocols bgp 65000 parameters log-neighbor-changes
    set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map
    set protocols bgp 65000 peer-group DMVPN bfd


    # Rebooting Hub-2
    vyos@Hub-2:~$ reboot
    Are you sure you want to reboot this system? [y/N]  y

    
    # Checking VRRP state and eBGP configuration on Hub-1
    vyos@Hub-1:~$ show vrrp
    Name    Interface      VRID  State      Priority  Last Transition
    ------  -----------  ------  -------  ----------  -----------------
    HQ      eth1v1            1  MASTER          200  1m57s
    
    vyos@Hub-1:~$ show configuration commands | match bgp
    set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
    set protocols bgp 65000 neighbor 172.16.253.131 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.253.131 remote-as '65001'
    set protocols bgp 65000 neighbor 172.16.253.132 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.253.132 remote-as '65002'
    set protocols bgp 65000 neighbor 172.16.253.133 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.253.133 remote-as '65003'
    set protocols bgp 65000 parameters log-neighbor-changes
    set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map
    set protocols bgp 65000 peer-group DMVPN bfd
    
    
    # Checking VRRP state and eBGP configuration on Hub-2 after reboot completed
    vyos@Hub-2:~$ show vrrp
    Name    Interface      VRID  State      Priority  Last Transition
    ------  -----------  ------  -------  ----------  -----------------
    HQ      eth1v1            1  BACKUP          100  1m46s
    
    vyos@Hub-2:~$ show configuration commands | match bgp
    set protocols bgp 65000 address-family ipv4-unicast network 192.168.0.0/24
    set protocols bgp 65000 neighbor 172.16.254.131 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.254.131 remote-as '65001'
    set protocols bgp 65000 neighbor 172.16.254.132 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.254.132 remote-as '65002'
    set protocols bgp 65000 neighbor 172.16.254.133 peer-group 'DMVPN'
    set protocols bgp 65000 neighbor 172.16.254.133 remote-as '65003'
    set protocols bgp 65000 parameters log-neighbor-changes
    set protocols bgp 65000 peer-group DMVPN address-family ipv4-unicast route-map export 'AS65000-PREP'
    set protocols bgp 65000 peer-group DMVPN bfd


    # consecutive pings check from Spoke-1 to the HQ local network during the failure
    --- 192.168.0.10 ping statistics ---
    1182 packets transmitted, 1182 received, 0% packet loss, time 1921ms
    rtt min/avg/max/mdev = 0.890/1.692/3.305/0.503 ms
    vyos@Spoke-1:~$


    # consecutive pings check from Spoke-3 to the Spoke-2 local network during the failure
    --- 192.168.2.1 ping statistics ---
    1186 packets transmitted, 1186 received, 0% packet loss, time 2100ms
    rtt min/avg/max/mdev = 0.506/1.236/8.497/0.369 ms
    vyos@Spoke-3:~$

From the results, we can see that the switchover performed as expected with 
0 packets loss both from Spoke-1 to HQ and Spoke-3 to Spoke-2 networks.