Update l3vpn-hub-and-spoke.rst

1 files changed, 655 insertions, 28 deletions

diff --git a/docs/configexamples/l3vpn-hub-and-spoke.rst b/docs/configexamples/l3vpn-hub-and-spoke.rst
index 66ad32fe..3a616873 100644
--- a/docs/configexamples/l3vpn-hub-and-spoke.rst
+++ b/docs/configexamples/l3vpn-hub-and-spoke.rst
@@ -46,6 +46,18 @@ The following software was used in the creation of this document:
 **NOTE:** VyOS Router (tested with VyOS 1.4-rolling-202110310317) 
 –  The configurations below are specifically for VyOS 1.4.x.
 
+General information can be found in the :ref:`l3vpn-vrfs` chapter.
+
+
+
+********
+Topology
+********
+.. image:: /_static/images/L3VPN_hub_spoke.png
+   :width: 80%
+   :align: center
+   :alt: Network Topology Diagram
+
 
 
 *****************
@@ -70,35 +82,650 @@ routes exchange and minimize iBGP peerings between devices.
 
 L3VPN configuration parameters table:
 
-+---------------------------------------+---------------------+
-| WAN Interface                         | eth0                |
-+---------------------------------------+---------------------+
-| On-premises address space             | 10.10.0.0/16        |
-+---------------------------------------+---------------------+
-| Azure address space                   |  10.0.0.0/16        |
-+---------------------------------------+---------------------+
-| Vyos public IP                        | 198.51.100.3        |
-+---------------------------------------+---------------------+
-| Vyos private IP                       | 10.10.0.5           |
-+---------------------------------------+---------------------+
-| Azure VNet Gateway public IP          |  203.0.113.2        |
-+---------------------------------------+---------------------+
-| Azure VNet Gateway BGP IP             |  10.0.0.4           |
-+---------------------------------------+---------------------+
-| Pre-shared key                        | ch00s3-4-s3cur3-psk |
-+---------------------------------------+---------------------+
-| Vyos ASN                              | 64499               |
-+---------------------------------------+---------------------+
-| Azure ASN                             | 65540               |
-+---------------------------------------+---------------------+
++----------+-------+------------+-----------------+-------------+-------------+
+|   Node   |  Role |     VRF    |        RD       |  RT import  |  RT export  |
++----------+-------+------------+-----------------+-------------+-------------+
+| VyOS-PE2 | Hub   | BLUE_HUB   | 10.80.80.1:1011 | 65035:1011  | 65035:1030  |
+|          |       |            |                 | 65035:1030  |             |
++----------+-------+------------+-----------------+-------------+-------------+
+| VyOS-PE1 | Spoke | BLUE_SPOKE | 10.50.50.1:1011 | 65035:1030  | 65035:1011  |
++----------+-------+------------+-----------------+-------------+-------------+
+| VyOS-PE3 | Spoke | BLUE_SPOKE | 10.60.60.1:1011 | 65035:1030  | 65035:1011  |
++----------+-------+------------+-----------------+-------------+-------------+
 
 
-********
-Topology
-********
-.. image:: /_static/images/Wan_load_balancing1.png
-   :width: 80%
-   :align: center
-   :alt: Network Topology Diagram
 
+*************
+Configuration
+*************
+
+
+
+Step-1: Configuring IGP and enabling MPLS LDP
+=====================================
+
+At the first step we need to configure the IP/MPLS backbone network using OSPF as 
+IGP protocol and LDP as label-switching protocol for the base connectivity between 
+**P** (rovider), **P** (rovider) **E** (dge) and **R** (oute) **R** (eflector) nodes:
+
+- VyOS-P1:
+
+.. code-block:: none
+   
+   # interfaces 
+   set interfaces dummy dum10 address '3.3.3.3/32'
+   set interfaces ethernet eth0 address '172.16.30.1/24'
+   set interfaces ethernet eth1 address '172.16.40.1/24'
+   set interfaces ethernet eth2 address '172.16.90.1/24'
+   set interfaces ethernet eth3 address '172.16.10.1/24'
+   set interfaces ethernet eth5 address '172.16.100.1/24'
+   
+   # protocols ospf+ldp
+   set protocols mpls interface 'eth1'
+   set protocols mpls interface 'eth2'
+   set protocols mpls interface 'eth3'
+   set protocols mpls interface 'eth5'
+   set protocols mpls interface 'eth0'
+   set protocols mpls ldp discovery transport-ipv4-address '3.3.3.3'
+   set protocols mpls ldp interface 'eth0'
+   set protocols mpls ldp interface 'eth1'
+   set protocols mpls ldp interface 'eth2'
+   set protocols mpls ldp interface 'eth3'
+   set protocols mpls ldp interface 'eth5'
+   set protocols mpls ldp router-id '3.3.3.3'
+   set protocols ospf area 0 network '0.0.0.0/0'
+   set protocols ospf parameters abr-type 'cisco'
+   set protocols ospf parameters router-id '3.3.3.3
+
+
+- VyOS-P2:
+
+.. code-block:: none
+   
+   # interfaces
+   set interfaces dummy dum10 address '4.4.4.4/32'
+   set interfaces ethernet eth0 address '172.16.30.2/24'
+   set interfaces ethernet eth1 address '172.16.20.1/24'
+   set interfaces ethernet eth2 address '172.16.120.1/24'
+   set interfaces ethernet eth3 address '172.16.60.1/24'
+   
+   # protocols ospf+ldp
+   set protocols mpls interface 'eth1'
+   set protocols mpls interface 'eth2'
+   set protocols mpls interface 'eth3'
+   set protocols mpls interface 'eth0'
+   set protocols mpls ldp discovery transport-ipv4-address '4.4.4.4'
+   set protocols mpls ldp interface 'eth0'
+   set protocols mpls ldp interface 'eth1'
+   set protocols mpls ldp interface 'eth2'
+   set protocols mpls ldp interface 'eth3'
+   set protocols mpls ldp router-id '4.4.4.4'
+   set protocols ospf area 0 network '0.0.0.0/0'
+   set protocols ospf parameters abr-type 'cisco'
+   set protocols ospf parameters router-id '4.4.4.4'
+
+- VyOS-P3:
+
+.. code-block:: none
+   
+   # interfaces
+   set interfaces dummy dum10 address '5.5.5.5/32'
+   set interfaces ethernet eth0 address '172.16.110.1/24'
+   set interfaces ethernet eth1 address '172.16.40.2/24'
+   set interfaces ethernet eth2 address '172.16.50.1/24'
+   set interfaces ethernet eth3 address '172.16.70.1/24'
+   
+   # protocols ospf + ldp
+   set protocols mpls interface 'eth1'
+   set protocols mpls interface 'eth2'
+   set protocols mpls interface 'eth3'
+   set protocols mpls interface 'eth0'
+   set protocols mpls ldp discovery transport-ipv4-address '5.5.5.5'
+   set protocols mpls ldp interface 'eth0'
+   set protocols mpls ldp interface 'eth1'
+   set protocols mpls ldp interface 'eth2'
+   set protocols mpls ldp interface 'eth3'
+   set protocols mpls ldp router-id '5.5.5.5'
+   set protocols ospf area 0 network '0.0.0.0/0'
+   set protocols ospf parameters abr-type 'cisco'
+   set protocols ospf parameters router-id '5.5.5.5'
+
+- VyOS-P4:
+
+.. code-block:: none
+   
+   # interfaces
+   set interfaces dummy dum10 address '6.6.6.6/32'
+   set interfaces ethernet eth0 address '172.16.80.2/24'
+   set interfaces ethernet eth1 address '172.16.130.1/24'
+   set interfaces ethernet eth2 address '172.16.50.2/24'
+   set interfaces ethernet eth3 address '172.16.60.2/24'
+   set interfaces ethernet eth5 address '172.16.140.1/24'
+   
+   
+   # protocols ospf + ldp
+   set protocols mpls interface 'eth1'
+   set protocols mpls interface 'eth2'
+   set protocols mpls interface 'eth3'
+   set protocols mpls interface 'eth0'
+   set protocols mpls interface 'eth5'
+   set protocols mpls ldp discovery transport-ipv4-address '6.6.6.6'
+   set protocols mpls ldp interface 'eth0'
+   set protocols mpls ldp interface 'eth1'
+   set protocols mpls ldp interface 'eth2'
+   set protocols mpls ldp interface 'eth3'
+   set protocols mpls ldp interface 'eth5'
+   set protocols mpls ldp router-id '6.6.6.6'
+   set protocols ospf area 0 network '0.0.0.0/0'
+   set protocols ospf parameters abr-type 'cisco'
+   set protocols ospf parameters router-id '6.6.6.6'
+
+- VyOS-PE1:
+
+.. code-block:: none
+   
+   # interfaces
+   set interfaces dummy dum10 address '7.7.7.7/32'
+   set interfaces ethernet eth0 address '172.16.90.2/24'
+   
+   # protocols  ospf + ldp 
+   set protocols mpls interface 'eth0'
+   set protocols mpls ldp discovery transport-ipv4-address '7.7.7.7'
+   set protocols mpls ldp interface 'eth0'
+   set protocols mpls ldp router-id '7.7.7.7'
+   set protocols ospf area 0 network '0.0.0.0/0'
+   set protocols ospf parameters abr-type 'cisco'
+   set protocols ospf parameters router-id '7.7.7.7'
+
+- VyOS-PE2:
+
+.. code-block:: none
+   
+   # interfaces
+   set interfaces dummy dum10 address '8.8.8.8/32'
+   set interfaces ethernet eth0 address '172.16.110.2/24'
+   set interfaces ethernet eth1 address '172.16.100.2/24'
+   set interfaces ethernet eth2 address '172.16.80.1/24'
+   
+   # protocols  ospf + ldp 
+   set protocols mpls interface 'eth0'
+   set protocols mpls interface 'eth1'
+   set protocols mpls ldp discovery transport-ipv4-address '8.8.8.8'
+   set protocols mpls ldp interface 'eth0'
+   set protocols mpls ldp interface 'eth1'
+   set protocols mpls ldp router-id '8.8.8.8'
+   set protocols ospf area 0 network '0.0.0.0/0'
+   set protocols ospf parameters abr-type 'cisco'
+   set protocols ospf parameters router-id '8.8.8.8'
+
+- VyOS-PE3:
+
+.. code-block:: none
+   
+   # interfaces
+   set interfaces dummy dum10 address '10.10.10.10/32'
+   set interfaces ethernet eth0 address '172.16.140.2/24'
+   
+   # protocols ospf + ldp
+   set protocols mpls interface 'eth0'
+   set protocols mpls ldp discovery transport-ipv4-address '10.10.10.10'
+   set protocols mpls ldp interface 'eth0'
+   set protocols mpls ldp router-id '10.10.10.10'
+   set protocols ospf area 0 network '0.0.0.0/0'
+   set protocols ospf parameters abr-type 'cisco'
+   set protocols ospf parameters router-id '10.10.10.10'
+
+- VyOS-RR1:
+
+.. code-block:: none
+   
+   # interfaces
+   set interfaces ethernet eth1 address '172.16.20.2/24'
+   set interfaces ethernet eth2 address '172.16.10.2/24'
+   set interfaces dummy dum10 address '1.1.1.1/32'
+   
+   # protocols ospf + ldp
+   set protocols mpls interface 'eth1'
+   set protocols mpls interface 'eth2'
+   set protocols mpls ldp discovery transport-ipv4-address '1.1.1.1'
+   set protocols mpls ldp interface 'eth1'
+   set protocols mpls ldp interface 'eth2'
+   set protocols mpls ldp router-id '1.1.1.1'
+   set protocols ospf area 0 network '0.0.0.0/0'
+   set protocols ospf parameters abr-type 'cisco'
+   set protocols ospf parameters router-id '1.1.1.1'
+
+- VyOS-RR2:
+
+.. code-block:: none
+   
+   # interfaces
+   set interfaces ethernet eth0 address '172.16.80.1/24'
+   set interfaces ethernet eth1 address '172.16.70.2/24'
+   set interfaces dummy dum10 address '2.2.2.2/32'
+   
+   # protocols ospf + ldp
+   set protocols mpls interface 'eth0'
+   set protocols mpls interface 'eth1'
+   set protocols mpls ldp discovery transport-ipv4-address '2.2.2.2'
+   set protocols mpls ldp interface 'eth1'
+   set protocols mpls ldp interface 'eth0'
+   set protocols mpls ldp router-id '2.2.2.2'
+   set protocols ospf area 0 network '0.0.0.0/0'
+   set protocols ospf parameters abr-type 'cisco'
+   set protocols ospf parameters router-id '2.2.2.2'
+
+
+
+Step-2: Configuring iBGP for L3VPN control-plane 
+================================================
+
+At this step we are going to enable iBGP protocol on MPLS nodes and 
+Route Reflectors (two routers for redundancy) that will deliver IPv4 
+VPN (L3VPN) routes between them:
+
+- VyOS-RR1:
+
+.. code-block:: none
+   
+   set protocols bgp local-as '65001'
+   set protocols bgp neighbor 7.7.7.7 address-family ipv4-vpn route-reflector-client
+   set protocols bgp neighbor 7.7.7.7 peer-group 'RR_VPNv4'
+   set protocols bgp neighbor 8.8.8.8 address-family ipv4-vpn route-reflector-client
+   set protocols bgp neighbor 8.8.8.8 peer-group 'RR_VPNv4'
+   set protocols bgp neighbor 9.9.9.9 address-family ipv4-vpn route-reflector-client
+   set protocols bgp neighbor 9.9.9.9 peer-group 'RR_VPNv4'
+   set protocols bgp neighbor 10.10.10.10 address-family ipv4-vpn route-reflector-client
+   set protocols bgp neighbor 10.10.10.10 peer-group 'RR_VPNv4'
+   set protocols bgp parameters cluster-id '1.1.1.1'
+   set protocols bgp parameters default no-ipv4-unicast
+   set protocols bgp parameters log-neighbor-changes
+   set protocols bgp parameters router-id '1.1.1.1'
+   set protocols bgp peer-group RR_VPNv4 remote-as '65001'
+   set protocols bgp peer-group RR_VPNv4 update-source 'dum10'
+
+- VyOS-RR2:
+
+.. code-block:: none
+   
+   set protocols bgp local-as '65001'
+   set protocols bgp neighbor 7.7.7.7 address-family ipv4-vpn route-reflector-client
+   set protocols bgp neighbor 7.7.7.7 peer-group 'RR_VPNv4'
+   set protocols bgp neighbor 8.8.8.8 address-family ipv4-vpn route-reflector-client
+   set protocols bgp neighbor 8.8.8.8 peer-group 'RR_VPNv4'
+   set protocols bgp neighbor 9.9.9.9 address-family ipv4-vpn route-reflector-client
+   set protocols bgp neighbor 9.9.9.9 peer-group 'RR_VPNv4'
+   set protocols bgp neighbor 10.10.10.10 address-family ipv4-vpn route-reflector-client
+   set protocols bgp neighbor 10.10.10.10 peer-group 'RR_VPNv4'
+   set protocols bgp parameters cluster-id '1.1.1.1'
+   set protocols bgp parameters default no-ipv4-unicast
+   set protocols bgp parameters log-neighbor-changes
+   set protocols bgp parameters router-id '2.2.2.2'
+   set protocols bgp peer-group RR_VPNv4 remote-as '65001'
+   set protocols bgp peer-group RR_VPNv4 update-source 'dum10'
+
+- VyOS-PE1:
+
+.. code-block:: none
+   
+   set protocols bgp local-as '65001'
+   set protocols bgp neighbor 1.1.1.1 address-family ipv4-vpn nexthop-self
+   set protocols bgp neighbor 1.1.1.1 peer-group 'RR_VPNv4'
+   set protocols bgp neighbor 2.2.2.2 address-family ipv4-vpn nexthop-self
+   set protocols bgp neighbor 2.2.2.2 peer-group 'RR_VPNv4'
+   set protocols bgp parameters default no-ipv4-unicast
+   set protocols bgp parameters log-neighbor-changes
+   set protocols bgp parameters router-id '7.7.7.7'
+   set protocols bgp peer-group RR_VPNv4 remote-as '65001'
+   set protocols bgp peer-group RR_VPNv4 update-source 'dum10'
+
+- VyOS-PE2:
+
+.. code-block:: none
+   
+   set protocols bgp local-as '65001'
+   set protocols bgp neighbor 1.1.1.1 address-family ipv4-vpn nexthop-self
+   set protocols bgp neighbor 1.1.1.1 peer-group 'RR_VPNv4'
+   set protocols bgp neighbor 2.2.2.2 address-family ipv4-vpn nexthop-self
+   set protocols bgp neighbor 2.2.2.2 peer-group 'RR_VPNv4'
+   set protocols bgp parameters default no-ipv4-unicast
+   set protocols bgp parameters log-neighbor-changes
+   set protocols bgp parameters router-id '8.8.8.8'
+   set protocols bgp peer-group RR_VPNv4 remote-as '65001'
+   set protocols bgp peer-group RR_VPNv4 update-source 'dum10'
+
+- VyOS-PE3:
+
+.. code-block:: none
+   
+   set protocols bgp local-as '65001'
+   set protocols bgp neighbor 1.1.1.1 address-family ipv4-vpn nexthop-self
+   set protocols bgp neighbor 1.1.1.1 peer-group 'RR_VPNv4'
+   set protocols bgp neighbor 2.2.2.2 address-family ipv4-vpn nexthop-self
+   set protocols bgp neighbor 2.2.2.2 peer-group 'RR_VPNv4'
+   set protocols bgp parameters default no-ipv4-unicast
+   set protocols bgp parameters log-neighbor-changes
+   set protocols bgp parameters router-id '10.10.10.10'
+   set protocols bgp peer-group RR_VPNv4 remote-as '65001'
+   set protocols bgp peer-group RR_VPNv4 update-source 'dum10'
+
+
+
+Step-3: Configuring L3VPN VRFs on PE nodes
+==========================================
+
+This section provides configuration steps for setting up VRFs on our 
+PE nodes including CE facing interfaces, BGP, rd and route-target 
+import/export based on the pre-defined parameters.
+
+- VyOS-PE1:
+
+.. code-block:: none
+   
+   # VRF settings
+   set vrf name BLUE_SPOKE table '200'
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast export vpn
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast import vpn
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast label vpn export 'auto'
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast network 10.50.50.0/24
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast rd vpn export '10.50.50.1:1011'
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast redistribute connected
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast route-target vpn export '65035:1011'
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast route-target vpn import '65035:1030'
+   set vrf name BLUE_SPOKE protocols bgp local-as '65001'
+   set vrf name BLUE_SPOKE protocols bgp neighbor 10.50.50.2 address-family ipv4-unicast as-override
+   set vrf name BLUE_SPOKE protocols bgp neighbor 10.50.50.2 remote-as '65035'
+   
+   # interfaces
+   set interfaces ethernet eth3 address '10.50.50.1/24'
+   set interfaces ethernet eth3 vrf 'BLUE_SPOKE'
+
+- VyOS-PE2:
+
+.. code-block:: none
+   
+   # VRF settings 
+   set vrf name BLUE_HUB table '400'
+   set vrf name BLUE_HUB protocols bgp address-family ipv4-unicast export vpn
+   set vrf name BLUE_HUB protocols bgp address-family ipv4-unicast import vpn
+   set vrf name BLUE_HUB protocols bgp address-family ipv4-unicast label vpn export 'auto'
+   set vrf name BLUE_HUB protocols bgp address-family ipv4-unicast network 10.80.80.0/24
+   set vrf name BLUE_HUB protocols bgp address-family ipv4-unicast rd vpn export '10.80.80.1:1011'
+   set vrf name BLUE_HUB protocols bgp address-family ipv4-unicast redistribute connected
+   set vrf name BLUE_HUB protocols bgp address-family ipv4-unicast route-target vpn export '65035:1030'
+   set vrf name BLUE_HUB protocols bgp address-family ipv4-unicast route-target vpn import '65035:1011 65050:2011 65035:1030'
+   set vrf name BLUE_HUB protocols bgp local-as '65001'
+   set vrf name BLUE_HUB protocols bgp neighbor 10.80.80.2 address-family ipv4-unicast as-override
+   set vrf name BLUE_HUB protocols bgp neighbor 10.80.80.2 remote-as '65035'
+   
+   # interfaces
+   set interfaces ethernet eth3 address '10.80.80.1/24'
+   set interfaces ethernet eth3 vrf 'BLUE_HUB'
+
+- VyOS-PE3:
+
+.. code-block:: none
+   
+   # VRF settings
+   set vrf name BLUE_SPOKE table '200'
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast export vpn
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast import vpn
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast label vpn export 'auto'
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast network 10.60.60.0/24
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast rd vpn export '10.60.60.1:1011'
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast redistribute connected
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast route-target vpn export '65035:1011'
+   set vrf name BLUE_SPOKE protocols bgp address-family ipv4-unicast route-target vpn import '65035:1030'
+   set vrf name BLUE_SPOKE protocols bgp local-as '65001'
+   set vrf name BLUE_SPOKE protocols bgp neighbor 10.60.60.2 address-family ipv4-unicast as-override
+   set vrf name BLUE_SPOKE protocols bgp neighbor 10.60.60.2 remote-as '65035'
+   
+   # interfaces
+   set interfaces ethernet eth3 address '10.60.60.1/24'
+   set interfaces ethernet eth3 vrf 'BLUE_SPOKE'
+
+
+
+Step-4: Configuring CE nodes
+============================
+
+Dynamic routing used between CE and PE nodes and eBGP peering 
+established for the route exchanging between them. All routes 
+received by PEs are then exported to L3VPN and delivered from 
+Spoke sites to Hub and vise-versa based on previously 
+configured L3VPN parameters.
+
+- VyOS-CE1-SPOKE:
+
+.. code-block:: none
+   
+   # interfaces
+   set interfaces dummy dum20 address '80.80.80.80/32'
+   set interfaces ethernet eth0 address '10.50.50.2/24'
+   
+   # BGP for peering with PE
+   set protocols bgp 65035 address-family ipv4-unicast network 80.80.80.80/32
+   set protocols bgp 65035 neighbor 10.50.50.1 ebgp-multihop '2'
+   set protocols bgp 65035 neighbor 10.50.50.1 remote-as '65001'
+   set protocols bgp 65035 neighbor 10.50.50.1 update-source 'eth0'
+   set protocols bgp 65035 parameters default no-ipv4-unicast
+   set protocols bgp 65035 parameters log-neighbor-changes
+   set protocols bgp 65035 parameters router-id '10.50.50.2'
+
+- VyOS-CE1-HUB:
+
+.. code-block:: none
+   
+   # interfaces
+   set interfaces dummy dum20 address '100.100.100.100/32'
+   set interfaces ethernet eth0 address '10.80.80.2/24'
+   
+   # BGP for peering with PE
+   set protocols bgp 65035 address-family ipv4-unicast network 100.100.100.100/32
+   set protocols bgp 65035 address-family ipv4-unicast redistribute connected
+   set protocols bgp 65035 neighbor 10.80.80.1 ebgp-multihop '2'
+   set protocols bgp 65035 neighbor 10.80.80.1 remote-as '65001'
+   set protocols bgp 65035 neighbor 10.80.80.1 update-source 'eth0'
+   set protocols bgp 65035 parameters default no-ipv4-unicast
+   set protocols bgp 65035 parameters log-neighbor-changes
+   set protocols bgp 65035 parameters router-id '10.80.80.2'
+
+- VyOS-CE2-SPOKE:
+
+.. code-block:: none
+   
+   # interfaces
+   set interfaces dummy dum20 address '90.90.90.90/32'
+   set interfaces ethernet eth0 address '10.60.60.2/24'
+   
+   # BGP for peering with PE 
+   set protocols bgp 65035 address-family ipv4-unicast network 90.90.90.90/32
+   set protocols bgp 65035 neighbor 10.60.60.1 ebgp-multihop '2'
+   set protocols bgp 65035 neighbor 10.60.60.1 remote-as '65001'
+   set protocols bgp 65035 neighbor 10.60.60.1 update-source 'eth0'
+   set protocols bgp 65035 parameters default no-ipv4-unicast
+   set protocols bgp 65035 parameters log-neighbor-changes
+   set protocols bgp 65035 parameters router-id '10.60.60.2'
+
+
+
+Step-5: Verification
+====================
+
+This section describes verification commands for MPLS/BGP/LDP 
+protocols and L3VPN related routes as well as diagnosis and 
+reachability checks between CE nodes.
+
+Let’s check IPv4 routing and MPLS information on provider nodes 
+(same procedure for all P nodes):
+
+- “show ip ospf neighbor” for checking ospf relationship
+
+.. code-block:: none
+   
+   vyos@VyOS-P1:~$  show ip ospf neighbor
+   
+   Neighbor ID 	Pri State       	Dead Time Address     	Interface            RXmtL RqstL DBsmL
+   4.4.4.4       	1 Full/Backup   	34.718s   172.16.30.2 	eth0:172.16.30.1       0 	  0 	  0
+   5.5.5.5       	1 Full/Backup   	35.132s   172.16.40.2 	eth1:172.16.40.1       0 	  0 	  0
+   7.7.7.7       	1 Full/Backup   	34.764s   172.16.90.2 	eth2:172.16.90.1       0 	  0 	  0
+   1.1.1.1       	1 Full/Backup   	35.642s   172.16.10.2 	eth3:172.16.10.1       0 	  0 	  0
+   8.8.8.8       	1 Full/Backup   	35.484s   172.16.100.2	eth5:172.16.100.1      0 	  0     0
+
+- “show mpls ldp neighbor “ for checking ldp neighbors
+
+.. code-block:: none
+   
+   vyos@VyOS-P1:~$ show mpls ldp neighbor
+   AF   ID          	State   	   Remote Address	Uptime
+   ipv4 1.1.1.1     	OPERATIONAL 1.1.1.1     	07w5d06h
+   ipv4 4.4.4.4     	OPERATIONAL 4.4.4.4     	09w3d00h
+   ipv4 5.5.5.5     	OPERATIONAL 5.5.5.5     	09w2d23h
+   ipv4 7.7.7.7     	OPERATIONAL 7.7.7.7     	03w0d01h
+   ipv4 8.8.8.8     	OPERATIONAL 8.8.8.8     	01w3d02h
+
+- “show mpls ldp binding” for checking mpls label assignment
+
+.. code-block:: none
+   
+   vyos@VyOS-P1:~$ show mpls ldp discovery
+   AF   Destination      	Nexthop     	Local    Label Remote Label  In Use
+   ipv4 1.1.1.1/32       	1.1.1.1     	23      	      imp-null     	yes
+   ipv4 1.1.1.1/32       	4.4.4.4     	23      	      20            	no
+   ipv4 1.1.1.1/32       	5.5.5.5     	23      	      17            	no
+   ipv4 1.1.1.1/32       	7.7.7.7     	23      	      16            	no
+   ipv4 1.1.1.1/32       	8.8.8.8     	23      	      16            	no
+   ipv4 2.2.2.2/32       	1.1.1.1     	20      	      16            	no
+   ipv4 2.2.2.2/32       	4.4.4.4     	20      	      22            	no
+   ipv4 2.2.2.2/32       	5.5.5.5     	20      	      24           	yes
+   ipv4 2.2.2.2/32       	7.7.7.7     	20      	      17            	no
+   ipv4 2.2.2.2/32       	8.8.8.8     	20      	      17            	no
+   ipv4 3.3.3.3/32       	1.1.1.1     	imp-null	      17            	no
+   ipv4 3.3.3.3/32       	4.4.4.4     	imp-null	      16            	no
+   ipv4 3.3.3.3/32       	5.5.5.5     	imp-null	      18            	no
+   ipv4 3.3.3.3/32       	7.7.7.7     	imp-null	      18            	no
+   ipv4 3.3.3.3/32       	8.8.8.8     	imp-null	      18            	no
+   ipv4 4.4.4.4/32       	1.1.1.1     	16      	      18            	no
+   ipv4 4.4.4.4/32       	4.4.4.4     	16      	      imp-null     	yes
+   ipv4 4.4.4.4/32       	5.5.5.5     	16      	      19            	no
+   ipv4 4.4.4.4/32       	7.7.7.7     	16      	      19            	no
+   ipv4 4.4.4.4/32       	8.8.8.8     	16      	      19            	no
+   ipv4 5.5.5.5/32       	1.1.1.1     	21      	      19            	no
+   ipv4 5.5.5.5/32       	4.4.4.4     	21      	      17            	no
+   ipv4 5.5.5.5/32       	5.5.5.5     	21      	      imp-null     	yes
+   ipv4 5.5.5.5/32       	7.7.7.7     	21      	      20            	no
+   ipv4 5.5.5.5/32       	8.8.8.8     	21      	      20            	no
+   ipv4 6.6.6.6/32       	1.1.1.1     	17      	      20            	no
+   ipv4 6.6.6.6/32       	4.4.4.4     	17      	      23           	yes
+   ipv4 6.6.6.6/32       	5.5.5.5     	17      	      21           	yes
+   ipv4 6.6.6.6/32       	7.7.7.7     	17      	      21            	no
+   ipv4 6.6.6.6/32       	8.8.8.8     	17      	      21            	no
+   ipv4 7.7.7.7/32       	1.1.1.1     	22      	      21            	no
+   ipv4 7.7.7.7/32       	4.4.4.4     	22      	      18            	no
+   ipv4 7.7.7.7/32       	5.5.5.5     	22      	      20            	no
+   ipv4 7.7.7.7/32       	7.7.7.7     	22      	      imp-null     	yes
+   ipv4 7.7.7.7/32       	8.8.8.8     	22      	      22            	no
+   ipv4 8.8.8.8/32       	1.1.1.1     	24      	      22            	no
+   ipv4 8.8.8.8/32       	4.4.4.4     	24      	      19            	no
+   ipv4 8.8.8.8/32       	5.5.5.5     	24      	      16            	no
+   ipv4 8.8.8.8/32       	7.7.7.7     	24      	      22            	no
+   ipv4 8.8.8.8/32       	8.8.8.8     	24      	      imp-null     	yes
+   ipv4 9.9.9.9/32       	1.1.1.1     	18      	      23            	no
+   ipv4 9.9.9.9/32       	4.4.4.4     	18      	      21           	yes
+   ipv4 9.9.9.9/32       	5.5.5.5     	18      	      22            	no
+   ipv4 9.9.9.9/32       	7.7.7.7     	18      	      23            	no
+   ipv4 9.9.9.9/32       	8.8.8.8     	18      	      23            	no
+   ipv4 10.10.10.10/32   	1.1.1.1     	19      	      24            	no
+   ipv4 10.10.10.10/32   	4.4.4.4     	19      	      24           	yes
+   ipv4 10.10.10.10/32   	5.5.5.5     	19      	      23           	yes
+   ipv4 10.10.10.10/32   	7.7.7.7     	19      	      24            	no
+   ipv4 10.10.10.10/32   	8.8.8.8     	19      	      24            	no
+   
+Now we’re checking iBGP status and routes from route-reflector 
+nodes to other devices:
+
+- “show bgp ipv4 vpn summary” for checking BGP VPNv4 neighbors:
+
+.. code-block:: none
+   
+   vyos@VyOS-RR1:~$ show bgp ipv4 vpn summary
+   BGP router identifier 1.1.1.1, local AS number 65001 vrf-id 0
+   BGP table version 0
+   RIB entries 9, using 1728 bytes of memory
+   Peers 4, using 85 KiB of memory
+   Peer groups 1, using 64 bytes of memory
+   
+   Neighbor    	V     	AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
+   7.7.7.7     	4  	65001  	7719  	7733    	      0	   0	0   5d07h56m        	2   	10
+   8.8.8.8     	4  	65001  	7715  	7724    	      0	   0	0   5d08h28m        	4   	10
+   9.9.9.9     	4  	65001  	7713  	7724    	      0	   0	0   5d08h28m        	2   	10
+   10.10.10.10 	4  	65001  	7713  	7724    	      0	   0	0   5d08h28m        	2   	10
+   
+   Total number of neighbors 4
+
+- “show bgp ipv4 vpn”  for checking all VPNv4 prefixes information: 
+
+.. code-block:: none
+   
+   vyos@VyOS-RR1:~$ show bgp ipv4 vpn
+   BGP table version is 2, local router ID is 1.1.1.1, vrf id 0
+   Default local pref 100, local AS 65001
+   Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
+              	i internal, r RIB-failure, S Stale, R Removed
+   Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
+   Origin codes:  i - IGP, e - EGP, ? - incomplete
+   
+      Network      	Next Hop        	Metric LocPrf Weight Path
+   Route Distinguisher: 10.50.50.1:1011
+   *>i10.50.50.0/24	7.7.7.7              	0	100  	0 i
+   	UN=7.7.7.7 EC{65035:1011} label=80 type=bgp, subtype=0
+   *>i80.80.80.80/32   7.7.7.7              	0	100  	0 65035 i
+   	UN=7.7.7.7 EC{65035:1011} label=80 type=bgp, subtype=0
+   Route Distinguisher: 10.60.60.1:1011
+   *>i10.60.60.0/24	10.10.10.10          	0	100  	0 i
+   	UN=10.10.10.10 EC{65035:1011} label=80 type=bgp, subtype=0
+   *>i90.90.90.90/32   10.10.10.10          	0	100  	0 65035 i
+   	UN=10.10.10.10 EC{65035:1011} label=80 type=bgp, subtype=0
+   Route Distinguisher: 10.80.80.1:1011
+   *>i10.80.80.0/24	8.8.8.8              	0	100  	0 i
+   	UN=8.8.8.8 EC{65035:1030} label=80 type=bgp, subtype=0
+   *>i100.100.100.100/32
+                   	8.8.8.8              	0	100  	0 65035 i
+   	UN=8.8.8.8 EC{65035:1030} label=80 type=bgp, subtype=0
+   Route Distinguisher: 172.16.80.1:2011
+   *>i10.110.110.0/24  8.8.8.8              	0	100  	0 65050 i
+   	UN=8.8.8.8 EC{65050:2011} label=81 type=bgp, subtype=0
+   *>i172.16.80.0/24   8.8.8.8              	0	100  	0 i
+   	UN=8.8.8.8 EC{65050:2011} label=81 type=bgp, subtype=0
+   Route Distinguisher: 172.16.100.1:2011
+   *>i10.210.210.0/24  9.9.9.9              	0	100  	0 65050 i
+   	UN=9.9.9.9 EC{65050:2011} label=80 type=bgp, subtype=0
+   *>i172.16.100.0/24  9.9.9.9              	0	100  	0 i
+   	UN=9.9.9.9 EC{65050:2011} label=80 type=bgp, subtype=0
+
+- “show bgp ipv4 vpn x.x.x.x/x” for checking best path selected 
+  for specific VPNv4 destination
+
+.. code-block:: none
+   
+   vyos@VyOS-RR1:~$ show bgp  ipv4 vpn 100.100.100.100/32
+   BGP routing table entry for 10.80.80.1:1011:100.100.100.100/32
+   not allocated
+   Paths: (1 available, best #1)
+     Advertised to non peer-group peers:
+     7.7.7.7 8.8.8.8 9.9.9.9 10.10.10.10
+     65035, (Received from a RR-client)
+   	8.8.8.8 from 8.8.8.8 (8.8.8.8)
+     	Origin incomplete, metric 0, localpref 100, valid, internal, best (First path received)
+     	Extended Community: RT:65035:1030
+     	Remote label: 80
+     	Last update: Tue Oct 19 13:45:32 202
+   
+Also we can verify how PE devices receives VPNv4 networks from the RRs 
+and installing them to the specific customer VRFs:
+
+- “show bgp ipv4 vpn summary” for checking iBGP neighbors against 
+  route-reflector devices:
 
+.. code-block:: none
+