From b51b7ad1f321cf77dddd06bb597442619ca836ff Mon Sep 17 00:00:00 2001 From: "L. Repetto" Date: Wed, 20 Apr 2022 15:32:03 -0300 Subject: add apendix-b and topology image --- docs/_static/images/inter-vrf-routing-vrf-lite.png | Bin 0 -> 104622 bytes docs/configexamples/index.rst | 1 + docs/configexamples/inter-vrf-routing-vrf-lite.rst | 836 +++++++++++++++++++++ 3 files changed, 837 insertions(+) create mode 100644 docs/_static/images/inter-vrf-routing-vrf-lite.png create mode 100644 docs/configexamples/inter-vrf-routing-vrf-lite.rst (limited to 'docs') diff --git a/docs/_static/images/inter-vrf-routing-vrf-lite.png b/docs/_static/images/inter-vrf-routing-vrf-lite.png new file mode 100644 index 00000000..3acefee6 Binary files /dev/null and b/docs/_static/images/inter-vrf-routing-vrf-lite.png differ diff --git a/docs/configexamples/index.rst b/docs/configexamples/index.rst index aecc3bdd..b90f25ad 100644 --- a/docs/configexamples/index.rst +++ b/docs/configexamples/index.rst @@ -17,6 +17,7 @@ This chapter contains various configuration examples: wan-load-balancing pppoe-ipv6-basic l3vpn-hub-and-spoke + inter-vrf-routing-vrf-lite Configuration Blueprints (autotest) diff --git a/docs/configexamples/inter-vrf-routing-vrf-lite.rst b/docs/configexamples/inter-vrf-routing-vrf-lite.rst new file mode 100644 index 00000000..d5c13e35 --- /dev/null +++ b/docs/configexamples/inter-vrf-routing-vrf-lite.rst @@ -0,0 +1,836 @@ +################################ +Inter-VRF Routing over VRF Lite +################################ + +**Virtual Routing and Forwarding** is a technology that allow multiple instance +of a routing table to exist within a single device. One of the key aspect of +**VRFs** is that do not share the same routes or interfaces, therefore packets +are forwarded between interfaces that belong to the same VRF only. + +Any information related to a VRF is not exchanged between devices -or in the +same device- by default, this is a technique called **VRF-Lite**. + +Keep networks isolated is -in general- a good principle, but there are cases +where you might need that some network can access other in a different VRF. + +The scope of this document is to cover such cases in a dynamic way without the +use of MPLS-LDP. + +******** +Overview +******** + +Let’s say we have a requirement to have multiple networks. + +* LAN 1 +* LAN 2 +* Management +* Internet + +Both LANs have to be able to route between each other, both will have managed +devices through a dedicated management network and both will need Internet +access yet the LAN2 will need access to some set of outside networks, not all. +The management network will need access to both LANs but cannot have access +to/from the outside. + +This scenario could be a nightmare applying regular routing and might need +filtering in multiple interfaces. + +A simple solution could be using different routing tables, or VRFs +for all the networks so we can keep the routing restrictions. +But for us to route between the different VRFs we would need a cable or a +logical connection between each other: + +* One cable/logical connection between LAN1 and LAN2 +* One cable/logical connection between LAN1 and Internet +* One cable/logical connection between LAN2 and Internet +* One cable/logical connection between LAN1 and Management +* One cable/logical connection between LAN2 and Management + +As we can see this is unpractical. + +To address this scenario we will use to our advantage an extension of the BGP +routing protocol that will help us in the “Export” between VRFs without the +need for MPLS. + +MP-BGP or MultiProtocol BGP introduces two main concepts to solve this +limitation: +- Route Distinguisher (RD): Is used to distinguish between different VRFs +–called VPNs- inside the BGP Process. The RD is appended to each IPv4 Network +that is advertised into BGP for that VPN making it a unique VPNv4 route. +- Route Target (RT): This is an extended BGP community append to the VPNv4 route +in the Import/Export process. When a route passes from the VRF routing table +into the BGP process it will add the configured export extended community(ies) +for that VPN. When that route needs to go from BGP into the VRF routing table +will only pass if that given VPN import policy matches any of the appended +community(ies) into that prefix. + +******** +Topology +******** + +.. image:: /_static/images/inter-vrf-routing-vrf-lite.png + :width: 70% + :align: left + :alt: Network Topology Diagram + +IP Schema +========= + ++----------+------------+----------------+------------------+ +| Device-A | Device-B | IPv4 Network | IPv6 Network | ++----------+------------+----------------+------------------+ +| Core | LAN1 | 10.1.1.0/30 | 2001:db8::/127 | ++----------+------------+----------------+------------------+ +| Core | LAN2 | 172.16.2.0/30 | 2001:db8::2/127 | ++----------+------------+----------------+------------------+ +| Core | Management | 192.168.3.0/30 | 2001:db8::4/127 | ++----------+------------+----------------+------------------+ +| Core | ISP | 10.2.2.0/30 | 2001:db8::6/127 | ++----------+------------+----------------+------------------+ + +RD & RT Schema +============== + ++------------+-----------+-----------+ +| VRF | RD | RT | ++------------+-----------+-----------+ +| LAN1 | 64496:1 | 64496:1 | ++------------+-----------+-----------+ +| LAN2 | 64496:2 | 64496:2 | ++------------+-----------+-----------+ +| Management | 64496:50 | 64496:50 | ++------------+-----------+-----------+ +| Internet | 64496:100 | 64496:100 | ++------------+-----------+-----------+ + +************** +Configurations +************** + +.. note:: We use a static route configuration in between the Core and each + LAN and Management router, and BGP between the Core router and the ISP router + but any dynamic routing protocol can be used. + +Remote Networks +=============== + +The following template configuration can be used in each remote router based +in our topology. + +.. code-block:: none + + # Interface Configuration + set interface eth eth address + + # Static default route back to Core + set procotols static route 0.0.0.0/0 next-hop + +Core Router +=========== + +Step 1: VRF and Configurations to remote networks +------------------------------------------------- + +- Configuration +^^^^^^^^^^^^^^^ + +Set the VRF name and Table ID, set interface address and bind it to the VRF. +Last add the static route to the remote network. + +.. code-block:: none + + # VRF name and table ID (MANDATORY) + set vrf name table + + # Interface Configuration + set interface eth eth address + + # Assign interface to VRF + set interface eth eth vrf + + # Static route to remote Network + set vrf name protocols static route next-hop + +- Verification +^^^^^^^^^^^^^^ + +Checking the routing table of the VRF should reveal both static and connected +entries active. A PING test between the Core and remote router is a way to +validate connectivity within the VRF. + +.. code-block:: none + + # show ip route vrf + # show ipv6 route vrf + + vyos@Core:~$ show ip route vrf LAN1 + Codes: K - kernel route, C - connected, S - static, R - RIP, + O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, + T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, + f - OpenFabric, + > - selected route, * - FIB route, q - queued, r - rejected, b - backup + t - trapped, o - offload failure + + VRF LAN1: + S>* 10.0.0.0/24 [1/0] via 10.1.1.2, eth0, weight 1, 00:05:41 + C>* 10.1.1.0/30 is directly connected, eth0, 00:05:44 + + vyos@Core:~$ show ipv6 route vrf LAN1 + Codes: K - kernel route, C - connected, S - static, R - RIPng, + O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table, + v - VNC, V - VNC-Direct, A - Babel, F - PBR, + f - OpenFabric, + > - selected route, * - FIB route, q - queued, r - rejected, b - backup + t - trapped, o - offload failure + + VRF LAN1: + C>* 2001:db8::/127 is directly connected, eth0, 00:18:43 + S>* 2001:db8:0:1::/64 [1/0] via 2001:db8::1, eth0, weight 1, 00:16:03 + C>* fe80::/64 is directly connected, eth0, 00:18:43 + + # ping vrf + + vyos@Core:~$ ping 10.1.1.2 vrf LAN1 + PING 10.1.1.2 (10.1.1.2) 56(84) bytes of data. + 64 bytes from 10.1.1.2: icmp_seq=1 ttl=64 time=1.52 ms + 64 bytes from 10.1.1.2: icmp_seq=2 ttl=64 time=0.830 ms + ^C + --- 10.1.1.2 ping statistics --- + 2 packets transmitted, 2 received, 0% packet loss, time 1002ms + rtt min/avg/max/mdev = 0.830/1.174/1.518/0.344 ms + vyos@Core:~$ ping 10.0.0.1 vrf LAN1 + PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data. + 64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.785 ms + 64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.948 ms + ^C + --- 10.0.0.1 ping statistics --- + 2 packets transmitted, 2 received, 0% packet loss, time 1002ms + rtt min/avg/max/mdev = 0.785/0.866/0.948/0.081 ms + + vyos@Core:~$ ping 2001:db8:0:1::1 vrf LAN1 + PING 2001:db8:0:1::1(2001:db8:0:1::1) 56 data bytes + 64 bytes from 2001:db8:0:1::1: icmp_seq=1 ttl=64 time=3.04 ms + 64 bytes from 2001:db8:0:1::1: icmp_seq=2 ttl=64 time=1.04 ms + 64 bytes from 2001:db8:0:1::1: icmp_seq=3 ttl=64 time=0.925 ms + ^C + --- 2001:db8:0:1::1 ping statistics --- + 3 packets transmitted, 3 received, 0% packet loss, time 2004ms + rtt min/avg/max/mdev = 0.925/1.665/3.035/0.969 ms + +Step 2: BGP Configuration for VRF-Lite +-------------------------------------- + +- Configuration +^^^^^^^^^^^^^^^ + +Setting BGP global local-as as well inside the VRF. Redistribute static routes +to inject configured networks into the BGP process but still inside the VRF. + + +.. code-block:: none + + # set BGP global local-as + set protocols bgp local-as + + # set BGP VRF local-as and redistribution + set vrf name protocols bgp local-as + set vrf name protocols bgp address-family redistribute static + +- Verification +^^^^^^^^^^^^^^ + +Check the BGP VRF table and verify if the static routes are injected showing +the correct next-hop information. + +.. code-block:: none + + # show ip bgp vrf + # show bgp vrf ipv6 + + vyos@Core:~$ show ip bgp vrf LAN1 + BGP table version is 3, local router ID is 10.1.1.1, vrf id 8 + Default local pref 100, local AS 64496 + Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, + i internal, r RIB-failure, S Stale, R Removed + Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self + Origin codes: i - IGP, e - EGP, ? - incomplete + RPKI validation codes: V valid, I invalid, N Not found + + Network Next Hop Metric LocPrf Weight Path + *> 10.0.0.0/24 10.1.1.2 0 32768 ? + + vyos@Core# run show bgp vrf LAN1 ipv6 + BGP table version is 13, local router ID is 10.1.1.1, vrf id 8 + Default local pref 100, local AS 64496 + Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, + i internal, r RIB-failure, S Stale, R Removed + Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self + Origin codes: i - IGP, e - EGP, ? - incomplete + RPKI validation codes: V valid, I invalid, N Not found + + Network Next Hop Metric LocPrf Weight Path + *> 2001:db8:0:1::/64 + 2001:db8::1 0 32768 ? + + +Step 3: VPN Configuration +------------------------- + +- Configuration +^^^^^^^^^^^^^^^ + +Within the VRF we set the Route-Distinguisher (RD) and Route-Targets (RT), then +we enable the export/import VPN. + + +.. code-block:: none + + # set Route-distinguisher + set vrf name protocols bgp address-family rd vpn export '' + + # set route-target for import/export + # Note: RT are a list that can be more than one community between apostrophe + # and separated by blank space. Ex: ' ' + set vrf name protocols bgp address-family route-target vpn export '' + set vrf name protocols bgp address-family route-target vpn import '' + + # Enable VPN export/import under this VRF + set vrf name protocols bgp address-family export vpn + set vrf name protocols bgp address-family import vpn + +A key point to understand is that if we need two VRFs to communicate between +each other EXPORT rt from VRF1 has to be in the IMPORT rt list from VRF2. But +this is only in ONE direction, to complete the communication the EXPORT rt from +VRF2 has to be in the IMPORT rt list from VRF1. + +There are some cases where this is not needed -for example, in some +DDoS appliance- but most inter-vrf routing designs use the above configurations. + +- Verification +^^^^^^^^^^^^^^ + +After configured all the VRFs involved in this topology we take a deeper look +at both BGP and Routing table for the VRF LAN1 + +.. code-block:: none + + # show ip bgp vrf + # show bgp vrf ipv6 + + vyos@Core# run show ip bgp vrf LAN1 + BGP table version is 53, local router ID is 10.1.1.1, vrf id 8 + Default local pref 100, local AS 64496 + Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, + i internal, r RIB-failure, S Stale, R Removed + Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self + Origin codes: i - IGP, e - EGP, ? - incomplete + RPKI validation codes: V valid, I invalid, N Not found + + Network Next Hop Metric LocPrf Weight Path + *> 0.0.0.0/0 10.2.2.2@7< 0 64497 i + *> 10.0.0.0/24 10.1.1.2 0 32768 ? + *> 10.2.2.0/30 10.2.2.2@7< 0 0 64497 ? + *> 192.0.2.0/24 10.2.2.2@7< 0 0 64497 ? + *> 192.168.0.0/24 192.168.3.2@11< 0 32768 ? + *> 198.51.100.0/24 10.2.2.2@7< 0 0 64497 ? + *> 203.0.113.0/24 10.2.2.2@7< 0 0 64497 ? + + vyos@Core# run show bgp vrf LAN1 ipv6 + BGP table version is 13, local router ID is 10.1.1.1, vrf id 8 + Default local pref 100, local AS 64496 + Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, + i internal, r RIB-failure, S Stale, R Removed + Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self + Origin codes: i - IGP, e - EGP, ? - incomplete + RPKI validation codes: V valid, I invalid, N Not found + + Network Next Hop Metric LocPrf Weight Path + *> ::/0 fe80::5200:ff:fe02:3@7< + 0 64497 i + *> 2001:db8::6/127 fe80::5200:ff:fe02:3@7< + 0 0 64497 ? + *> 2001:db8:0:1::/64 + 2001:db8::1 0 32768 ? + *> 2001:db8:0:3::/64 + 2001:db8::5@11< 0 32768 ? + *> 2001:db8:1::/48 fe80::5200:ff:fe02:3@7< + 0 0 64497 ? + *> 2001:db8:2::/48 fe80::5200:ff:fe02:3@7< + 0 0 64497 ? + *> 2001:db8:3::/48 fe80::5200:ff:fe02:3@7< + 0 0 64497 ? + + + # show ip route vrf + # show ipv6 route vrf + + vyos@Core:~$ show ip route vrf LAN1 + Codes: K - kernel route, C - connected, S - static, R - RIP, + O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, + T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, + f - OpenFabric, + > - selected route, * - FIB route, q - queued, r - rejected, b - backup + t - trapped, o - offload failure + + VRF LAN1: + B>* 0.0.0.0/0 [20/0] via 10.2.2.2, eth3 (vrf Internet), weight 1, 00:00:38 + S>* 10.0.0.0/24 [1/0] via 10.1.1.2, eth0, weight 1, 00:29:57 + C>* 10.1.1.0/30 is directly connected, eth0, 00:29:59 + B 10.2.2.0/30 [20/0] via 10.2.2.2 (vrf Internet) inactive, weight 1, 00:00:38 + B>* 172.16.0.0/24 [20/0] via 172.16.2.2, eth1 (vrf LAN2), weight 1, 00:00:38 + B>* 192.0.2.0/24 [20/0] via 10.2.2.2, eth3 (vrf Internet), weight 1, 00:00:38 + B>* 198.51.100.0/24 [20/0] via 10.2.2.2, eth3 (vrf Internet), weight 1, 00:00:38 + B>* 203.0.113.0/24 [20/0] via 10.2.2.2, eth3 (vrf Internet), weight 1, 00:00:38 + + vyos@Core# run show ipv6 route vrf LAN1 + Codes: K - kernel route, C - connected, S - static, R - RIPng, + O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table, + v - VNC, V - VNC-Direct, A - Babel, F - PBR, + f - OpenFabric, + > - selected route, * - FIB route, q - queued, r - rejected, b - backup + t - trapped, o - offload failure + + VRF LAN1: + B>* ::/0 [20/0] via fe80::5200:ff:fe02:3, eth3 (vrf Internet), weight 1, 00:07:50 + C>* 2001:db8::/127 is directly connected, eth0, 05:33:43 + B>* 2001:db8::6/127 [20/0] via fe80::5200:ff:fe02:3, eth3 (vrf Internet), weight 1, 00:07:50 + S>* 2001:db8:0:1::/64 [1/0] via 2001:db8::1, eth0, weight 1, 05:31:03 + B>* 2001:db8:0:3::/64 [20/0] via 2001:db8::5, eth2 (vrf Management), weight 1, 00:07:50 + B>* 2001:db8:1::/48 [20/0] via fe80::5200:ff:fe02:3, eth3 (vrf Internet), weight 1, 00:07:50 + B>* 2001:db8:2::/48 [20/0] via fe80::5200:ff:fe02:3, eth3 (vrf Internet), weight 1, 00:07:50 + B>* 2001:db8:3::/48 [20/0] via fe80::5200:ff:fe02:3, eth3 (vrf Internet), weight 1, 00:07:50 + C>* fe80::/64 is directly connected, eth0, 05:33:43 + + +As we can see in the BGP table any imported route has been injected with a "@" +followed by the VPN id; In the routing table of the VRF, if the route was +installed, we can see -between round brackets- the exported VRF table. + +Step 4: End to End verification +------------------------------- + +Now we perform some end-to-end testing + +- From Management to LAN1/LAN2 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: none + + vyos@Management:~$ ping 10.0.0.1 source-address 192.168.0.1 + PING 10.0.0.1 (10.0.0.1) from 192.168.0.1 : 56(84) bytes of data. + 64 bytes from 10.0.0.1: icmp_seq=1 ttl=63 time=1.93 ms + 64 bytes from 10.0.0.1: icmp_seq=2 ttl=63 time=2.12 ms + 64 bytes from 10.0.0.1: icmp_seq=3 ttl=63 time=2.12 ms + ^C + --- 10.0.0.1 ping statistics --- + 3 packets transmitted, 3 received, 0% packet loss, time 2005ms + rtt min/avg/max/mdev = 1.931/2.056/2.123/0.088 ms + vyos@Management:~$ ping 172.16.0.1 source-address 192.168.0.1 + PING 172.16.0.1 (172.16.0.1) from 192.168.0.1 : 56(84) bytes of data. + 64 bytes from 172.16.0.1: icmp_seq=1 ttl=63 time=1.62 ms + 64 bytes from 172.16.0.1: icmp_seq=2 ttl=63 time=1.75 ms + ^C + --- 172.16.0.1 ping statistics --- + 2 packets transmitted, 2 received, 0% packet loss, time 1001ms + rtt min/avg/max/mdev = 1.621/1.686/1.752/0.065 ms + vyos@Management:~$ ping 2001:db8:0:1::1 source-address 2001:db8:0:3::1 + PING 2001:db8:0:1::1(2001:db8:0:1::1) from 2001:db8:0:3::1 : 56 data bytes + 64 bytes from 2001:db8:0:1::1: icmp_seq=1 ttl=63 time=2.44 ms + 64 bytes from 2001:db8:0:1::1: icmp_seq=2 ttl=63 time=2.40 ms + 64 bytes from 2001:db8:0:1::1: icmp_seq=3 ttl=63 time=2.41 ms + ^C + --- 2001:db8:0:1::1 ping statistics --- + 3 packets transmitted, 3 received, 0% packet loss, time 2003ms + rtt min/avg/max/mdev = 2.399/2.418/2.442/0.017 ms + vyos@Management:~$ ping 2001:db8:0:2::1 source-address 2001:db8:0:3::1 + PING 2001:db8:0:2::1(2001:db8:0:2::1) from 2001:db8:0:3::1 : 56 data bytes + 64 bytes from 2001:db8:0:2::1: icmp_seq=1 ttl=63 time=1.66 ms + 64 bytes from 2001:db8:0:2::1: icmp_seq=2 ttl=63 time=1.99 ms + 64 bytes from 2001:db8:0:2::1: icmp_seq=3 ttl=63 time=1.88 ms + 64 bytes from 2001:db8:0:2::1: icmp_seq=4 ttl=63 time=2.32 ms + ^C + --- 2001:db8:0:2::1 ping statistics --- + 4 packets transmitted, 4 received, 0% packet loss, time 3005ms + rtt min/avg/max/mdev = 1.660/1.960/2.315/0.236 ms + +- From Management to Outside (fails as intended) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: none + + vyos@Management:~$ show ip route + Codes: K - kernel route, C - connected, S - static, R - RIP, + O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, + T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, + f - OpenFabric, + > - selected route, * - FIB route, q - queued, r - rejected, b - backup + t - trapped, o - offload failure + + S>* 0.0.0.0/0 [1/0] via 192.168.3.1, eth2, weight 1, 00:01:58 + C>* 192.168.0.0/24 is directly connected, dum0, 00:02:05 + C>* 192.168.3.0/30 is directly connected, eth2, 00:02:03 + vyos@Management:~$ ping 192.0.2.1 + PING 192.0.2.1 (192.0.2.1) 56(84) bytes of data. + From 192.168.3.1 icmp_seq=1 Destination Net Unreachable + From 192.168.3.1 icmp_seq=2 Destination Net Unreachable + ^C + --- 192.0.2.1 ping statistics --- + 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1002ms + + vyos@Management:~$ ping 195.51.100.1 + PING 195.51.100.1 (195.51.100.1) 56(84) bytes of data. + From 192.168.3.1 icmp_seq=1 Destination Net Unreachable + From 192.168.3.1 icmp_seq=2 Destination Net Unreachable + From 192.168.3.1 icmp_seq=3 Destination Net Unreachable + ^C + --- 195.51.100.1 ping statistics --- + 3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2003ms + + vyos@Management:~$ ping 2001:db8:1::1 + PING 2001:db8:1::1(2001:db8:1::1) 56 data bytes + From 2001:db8::4 icmp_seq=1 Destination unreachable: No route + From 2001:db8::4 icmp_seq=2 Destination unreachable: No route + ^C + --- 2001:db8:1::1 ping statistics --- + 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1002ms + + vyos@Management:~$ ping 2001:db8:2::1 + PING 2001:db8:2::1(2001:db8:2::1) 56 data bytes + From 2001:db8::4 icmp_seq=1 Destination unreachable: No route + From 2001:db8::4 icmp_seq=2 Destination unreachable: No route + ^C + --- 2001:db8:2::1 ping statistics --- + 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1002ms + + +- LAN1 to Outside +^^^^^^^^^^^^^^^^^ + +.. code-block:: none + + vyos@LAN1:~$ ping 192.0.2.1 source-address 10.0.0.1 + PING 192.0.2.1 (192.0.2.1) from 10.0.0.1 : 56(84) bytes of data. + 64 bytes from 192.0.2.1: icmp_seq=1 ttl=63 time=1.47 ms + 64 bytes from 192.0.2.1: icmp_seq=2 ttl=63 time=1.41 ms + 64 bytes from 192.0.2.1: icmp_seq=3 ttl=63 time=1.80 ms + ^C + --- 192.0.2.1 ping statistics --- + 3 packets transmitted, 3 received, 0% packet loss, time 2004ms + rtt min/avg/max/mdev = 1.414/1.563/1.803/0.171 ms + vyos@LAN1:~$ ping 198.51.100.1 source-address 10.0.0.1 + PING 198.51.100.1 (198.51.100.1) from 10.0.0.1 : 56(84) bytes of data. + 64 bytes from 198.51.100.1: icmp_seq=1 ttl=63 time=1.71 ms + 64 bytes from 198.51.100.1: icmp_seq=2 ttl=63 time=1.83 ms + ^C + --- 198.51.100.1 ping statistics --- + 2 packets transmitted, 2 received, 0% packet loss, time 1002ms + rtt min/avg/max/mdev = 1.705/1.766/1.828/0.061 ms + vyos@LAN1:~$ ping 203.0.113.1 source-address 10.0.0.1 + PING 203.0.113.1 (203.0.113.1) from 10.0.0.1 : 56(84) bytes of data. + 64 bytes from 203.0.113.1: icmp_seq=1 ttl=63 time=1.25 ms + 64 bytes from 203.0.113.1: icmp_seq=2 ttl=63 time=1.88 ms + ^C + --- 203.0.113.1 ping statistics --- + 2 packets transmitted, 2 received, 0% packet loss, time 1003ms + rtt min/avg/max/mdev = 1.249/1.566/1.884/0.317 ms + vyos@LAN1:~$ ping 2001:db8:1::1 source-address 2001:db8:0:1::1 + PING 2001:db8:1::1(2001:db8:1::1) from 2001:db8:0:1::1 : 56 data bytes + 64 bytes from 2001:db8:1::1: icmp_seq=1 ttl=63 time=2.35 ms + 64 bytes from 2001:db8:1::1: icmp_seq=2 ttl=63 time=2.29 ms + 64 bytes from 2001:db8:1::1: icmp_seq=3 ttl=63 time=2.22 ms + ^C + --- 2001:db8:1::1 ping statistics --- + 3 packets transmitted, 3 received, 0% packet loss, time 2004ms + rtt min/avg/max/mdev = 2.215/2.285/2.352/0.055 ms + vyos@LAN1:~$ ping 2001:db8:2::1 source-address 2001:db8:0:1::1 + PING 2001:db8:2::1(2001:db8:2::1) from 2001:db8:0:1::1 : 56 data bytes + 64 bytes from 2001:db8:2::1: icmp_seq=1 ttl=63 time=1.37 ms + 64 bytes from 2001:db8:2::1: icmp_seq=2 ttl=63 time=2.68 ms + 64 bytes from 2001:db8:2::1: icmp_seq=3 ttl=63 time=2.00 ms + ^C + --- 2001:db8:2::1 ping statistics --- + 3 packets transmitted, 3 received, 0% packet loss, time 2003ms + rtt min/avg/max/mdev = 1.367/2.015/2.679/0.535 ms + + +.. note:: we are using "source-address" option cause we are not redistributing + connected interfaces into BGP on the Core router hence there is no comeback + route and ping will fail. + +- LAN1 to LAN2 +^^^^^^^^^^^^^^ + +.. code-block:: none + + vyos@LAN1:~$ ping 172.16.0.1 source-address 10.0.0.1 + PING 172.16.0.1 (172.16.0.1) from 10.0.0.1 : 56(84) bytes of data. + 64 bytes from 172.16.0.1: icmp_seq=1 ttl=63 time=3.00 ms + 64 bytes from 172.16.0.1: icmp_seq=2 ttl=63 time=2.20 ms + ^C + --- 172.16.0.1 ping statistics --- + 2 packets transmitted, 2 received, 0% packet loss, time 1002ms + rtt min/avg/max/mdev = 2.199/2.600/3.001/0.401 ms + vyos@LAN1:~$ ping 2001:db8:0:2::1 source 2001:db8:0:1::1 + PING 2001:db8:0:2::1(2001:db8:0:2::1) from 2001:db8:0:1::1 : 56 data bytes + 64 bytes from 2001:db8:0:2::1: icmp_seq=1 ttl=63 time=4.82 ms + 64 bytes from 2001:db8:0:2::1: icmp_seq=2 ttl=63 time=1.95 ms + 64 bytes from 2001:db8:0:2::1: icmp_seq=3 ttl=63 time=1.98 ms + ^C + --- 2001:db8:0:2::1 ping statistics --- + 3 packets transmitted, 3 received, 0% packet loss, time 2003ms + rtt min/avg/max/mdev = 1.949/2.915/4.815/1.343 ms + +*********** +Conclusions +*********** + +Inter-VRF routing is a well-known solution to address complex routing scenarios +that enable -in a dynamic way- to leak routes between VRFs. Is recommended to +take special consideration while designing route-targets and its application as +it can minimize future interventions while creating a new VRF will automatically +take the desired effect in its propagation. + +********** +Appendix-A +********** + +Full configuration from all devices +----------------------------------- + +- Core +^^^^^^ + +.. code-block:: none + + set interfaces ethernet eth0 address '10.1.1.1/30' + set interfaces ethernet eth0 address '2001:db8::/127' + set interfaces ethernet eth0 vrf 'LAN1' + set interfaces ethernet eth1 address '172.16.2.1/30' + set interfaces ethernet eth1 address '2001:db8::2/127' + set interfaces ethernet eth1 vrf 'LAN2' + set interfaces ethernet eth2 address '192.168.3.1/30' + set interfaces ethernet eth2 address '2001:db8::4/127' + set interfaces ethernet eth2 vrf 'Management' + set interfaces ethernet eth3 address '10.2.2.1/30' + set interfaces ethernet eth3 address '2001:db8::6/127' + set interfaces ethernet eth3 vrf 'Internet' + set protocols bgp address-family ipv4-unicast + set protocols bgp local-as '64496' + set vrf name Internet protocols bgp address-family ipv4-unicast export vpn + set vrf name Internet protocols bgp address-family ipv4-unicast import vpn + set vrf name Internet protocols bgp address-family ipv4-unicast rd vpn export '64496:100' + set vrf name Internet protocols bgp address-family ipv4-unicast route-target vpn export '64496:100' + set vrf name Internet protocols bgp address-family ipv4-unicast route-target vpn import '64496:1 64496:2' + set vrf name Internet protocols bgp address-family ipv6-unicast export vpn + set vrf name Internet protocols bgp address-family ipv6-unicast import vpn + set vrf name Internet protocols bgp address-family ipv6-unicast rd vpn export '64496:100' + set vrf name Internet protocols bgp address-family ipv6-unicast route-target vpn export '64496:100' + set vrf name Internet protocols bgp address-family ipv6-unicast route-target vpn import '64496:1 64496:2' + set vrf name Internet protocols bgp local-as '64496' + set vrf name Internet protocols bgp neighbor 10.2.2.2 address-family ipv4-unicast + set vrf name Internet protocols bgp neighbor 10.2.2.2 remote-as '64497' + set vrf name Internet protocols bgp neighbor 2001:db8::7 address-family ipv6-unicast + set vrf name Internet protocols bgp neighbor 2001:db8::7 remote-as '64497' + set vrf name Internet table '104' + set vrf name LAN1 protocols bgp address-family ipv4-unicast export vpn + set vrf name LAN1 protocols bgp address-family ipv4-unicast import vpn + set vrf name LAN1 protocols bgp address-family ipv4-unicast rd vpn export '64496:1' + set vrf name LAN1 protocols bgp address-family ipv4-unicast redistribute static + set vrf name LAN1 protocols bgp address-family ipv4-unicast route-target vpn export '64496:1' + set vrf name LAN1 protocols bgp address-family ipv4-unicast route-target vpn import '64496:100 64996:50 64496:2' + set vrf name LAN1 protocols bgp address-family ipv6-unicast export vpn + set vrf name LAN1 protocols bgp address-family ipv6-unicast import vpn + set vrf name LAN1 protocols bgp address-family ipv6-unicast rd vpn export '64496:1' + set vrf name LAN1 protocols bgp address-family ipv6-unicast redistribute static + set vrf name LAN1 protocols bgp address-family ipv6-unicast route-target vpn export '64496:1' + set vrf name LAN1 protocols bgp address-family ipv6-unicast route-target vpn import '64496:100 64496:50 64496:2' + set vrf name LAN1 protocols bgp local-as '64496' + set vrf name LAN1 protocols static route 10.0.0.0/24 next-hop 10.1.1.2 + set vrf name LAN1 protocols static route6 2001:db8:0:1::/64 next-hop 2001:db8::1 + set vrf name LAN1 table '101' + set vrf name LAN2 protocols bgp address-family ipv4-unicast export vpn + set vrf name LAN2 protocols bgp address-family ipv4-unicast import vpn + set vrf name LAN2 protocols bgp address-family ipv4-unicast rd vpn export '64496:2' + set vrf name LAN2 protocols bgp address-family ipv4-unicast redistribute static + set vrf name LAN2 protocols bgp address-family ipv4-unicast route-target vpn export '64496:2' + set vrf name LAN2 protocols bgp address-family ipv4-unicast route-target vpn import '64496:100 64496:50 64496:1' + set vrf name LAN2 protocols bgp address-family ipv6-unicast export vpn + set vrf name LAN2 protocols bgp address-family ipv6-unicast import vpn + set vrf name LAN2 protocols bgp address-family ipv6-unicast rd vpn export '64496:2' + set vrf name LAN2 protocols bgp address-family ipv6-unicast redistribute static + set vrf name LAN2 protocols bgp address-family ipv6-unicast route-target vpn export '64496:2' + set vrf name LAN2 protocols bgp address-family ipv6-unicast route-target vpn import '64496:100 64496:50 64496:1' + set vrf name LAN2 protocols bgp local-as '64496' + set vrf name LAN2 protocols static route 172.16.0.0/24 next-hop 172.16.2.2 + set vrf name LAN2 protocols static route6 2001:db8:0:2::/64 next-hop 2001:db8::3 + set vrf name LAN2 table '102' + set vrf name Management protocols bgp address-family ipv4-unicast export vpn + set vrf name Management protocols bgp address-family ipv4-unicast import vpn + set vrf name Management protocols bgp address-family ipv4-unicast rd vpn export '64496:50' + set vrf name Management protocols bgp address-family ipv4-unicast redistribute static + set vrf name Management protocols bgp address-family ipv4-unicast route-target vpn export '64496:50' + set vrf name Management protocols bgp address-family ipv4-unicast route-target vpn import '64496:1 64496:2' + set vrf name Management protocols bgp address-family ipv6-unicast export vpn + set vrf name Management protocols bgp address-family ipv6-unicast import vpn + set vrf name Management protocols bgp address-family ipv6-unicast rd vpn export '64496:50' + set vrf name Management protocols bgp address-family ipv6-unicast redistribute static + set vrf name Management protocols bgp address-family ipv6-unicast route-target vpn export '64496:50' + set vrf name Management protocols bgp address-family ipv6-unicast route-target vpn import '64496:1 64496:2' + set vrf name Management protocols bgp local-as '64496' + set vrf name Management protocols static route 192.168.0.0/24 next-hop 192.168.3.2 + set vrf name Management protocols static route6 2001:db8:0:3::/64 next-hop 2001:db8::5 + set vrf name Management table '103' + + +- LAN1 +^^^^^^ + +.. code-block:: none + + set interfaces dummy dum0 address '10.0.0.1/24' + set interfaces dummy dum0 address '2001:db8:0:1::1/64' + set interfaces ethernet eth0 address '10.1.1.2/30' + set interfaces ethernet eth0 address '2001:db8::1/127' + set protocols static route 0.0.0.0/0 next-hop 10.1.1.1 + set protocols static route6 ::/0 next-hop 2001:db8::* + +- LAN2 +^^^^^^ + +.. code-block:: none + set interfaces dummy dum0 address '172.16.0.1/24' + set interfaces dummy dum0 address '2001:db8:0:2::1/64' + set interfaces ethernet eth0 hw-id '50:00:00:03:00:00' + set interfaces ethernet eth1 address '172.16.2.2/30' + set interfaces ethernet eth1 address '2001:db8::3/127' + set protocols static route 0.0.0.0/0 next-hop 172.16.2.1 + set protocols static route6 ::/0 next-hop 2001:db8::2 + +- Management +^^^^^^^^^^^^ + +.. code-block:: none + + set interfaces dummy dum0 address '192.168.0.1/24' + set interfaces dummy dum0 address '2001:db8:0:3::1/64' + set interfaces ethernet eth2 address '192.168.3.2/30' + set interfaces ethernet eth2 address '2001:db8::5/127' + set protocols static route 0.0.0.0/0 next-hop 192.168.3.1 + set protocols static route6 ::/0 next-hop 2001:db8::4 + +- ISP +^^^^^ + +.. code-block:: none + + set interfaces dummy dum0 address '192.0.2.1/24' + set interfaces dummy dum0 address '2001:db8:1::1/48' + set interfaces dummy dum1 address '198.51.100.1/24' + set interfaces dummy dum1 address '2001:db8:2::1/48' + set interfaces dummy dum2 address '203.0.113.1/24' + set interfaces dummy dum2 address '2001:db8:3::1/48' + set interfaces ethernet eth3 address '10.2.2.2/30' + set interfaces ethernet eth3 address '2001:db8::7/127' + set protocols bgp address-family ipv4-unicast redistribute connected + set protocols bgp address-family ipv6-unicast redistribute connected + set protocols bgp local-as '64497' + set protocols bgp neighbor 10.2.2.1 address-family ipv4-unicast default-originate + set protocols bgp neighbor 10.2.2.1 remote-as '64496' + set protocols bgp neighbor 2001:db8::6 address-family ipv6-unicast default-originate + set protocols bgp neighbor 2001:db8::6 remote-as '64496' + set protocols static route 0.0.0.0/0 next-hop 10.2.2.1 + set protocols static route6 ::/0 next-hop 2001:db8::6 + +********** +Appendix-B +********** + +Route-Filtering +--------------- + +When importing routes using MP-BGP it is possible to filter a subset of them +before are injected in the BGP table. One of the most common case is to use a +route-map with an prefix-list. + +- Configuration +^^^^^^^^^^^^^^^ + +We create a prefix-list first and add all the routes we need to. + +.. code-block:: none + + # set both ipv4 and ipv6 policies + + set policy prefix-list LAN2-Internet rule 1 action 'permit' + set policy prefix-list LAN2-Internet rule 1 le '24' + set policy prefix-list LAN2-Internet rule 1 prefix '198.51.0.0/16' + set policy prefix-list LAN2-Internet rule 2 action 'permit' + set policy prefix-list LAN2-Internet rule 2 prefix '192.0.2.0/24' + set policy prefix-list LAN2-Internet rule 3 action 'permit' + set policy prefix-list LAN2-Internet rule 3 prefix '192.168.0.0/24' + set policy prefix-list LAN2-Internet rule 4 action 'permit' + set policy prefix-list LAN2-Internet rule 4 prefix '10.0.0.0/24' + + set policy prefix-list6 LAN2-Internet-v6 rule 1 action 'permit' + set policy prefix-list6 LAN2-Internet-v6 rule 1 prefix '2001:db8:1::/48' + set policy prefix-list6 LAN2-Internet-v6 rule 2 action 'permit' + set policy prefix-list6 LAN2-Internet-v6 rule 2 prefix '2001:db8:2::/48' + set policy prefix-list6 LAN2-Internet-v6 rule 3 action 'permit' + set policy prefix-list6 LAN2-Internet-v6 rule 3 prefix '2001:db8:0:3::/64' + set policy prefix-list6 LAN2-Internet-v6 rule 4 action 'permit' + set policy prefix-list6 LAN2-Internet-v6 rule 4 prefix '2001:db8:0:1::/64' + +Then add a route-map and reference to above prefix. Consider that the actions +taken inside the prefix will MATCH the routes that will be affected by the +actions inside the rules of the route-map. + +.. code-block:: none + + set policy route-map LAN2-Internet rule 1 action 'permit' + set policy route-map LAN2-Internet rule 1 match ip address prefix-list 'LAN2-Internet' + + set policy route-map LAN2-Internet-v6 rule 1 action 'permit' + set policy route-map LAN2-Internet-v6 rule 1 match ipv6 address prefix-list 'LAN2-Internet-v6' + +We are using a "white list" approach by allowing only what is necessary. In case +that need to implement a "black list" approach you will ned to change the action +in the route-map for a deny BUT you need to add a rule that permit the rest due +to the implicit deny in the route-map. + +Then we need to attach the policy into the bgp process. This needs to be under +the import statement in the vrf we need to filter. + +.. code-block:: none + + set vrf name LAN2 protocols bgp address-family ipv4-unicast route-map vpn import 'LAN2-Internet' + set vrf name LAN2 protocols bgp address-family ipv6-unicast route-map vpn import 'LAN2-Internet-v6' + + +- Verification +^^^^^^^^^^^^^^ + +.. code-block:: none + + # show ip route vrf LAN2 + + B>* 10.0.0.0/24 [20/0] via 10.1.1.2, eth0 (vrf LAN1), weight 1, 00:45:28 + S>* 172.16.0.0/24 [1/0] via 172.16.2.2, eth1, weight 1, 00:45:32 + C>* 172.16.2.0/30 is directly connected, eth1, 00:45:39 + B>* 192.0.2.0/24 [20/0] via 10.2.2.2, eth3 (vrf Internet), weight 1, 00:45:24 + B>* 192.168.0.0/24 [20/0] via 192.168.3.2, eth2 (vrf Managment), weight 1, 00:45:27 + B>* 198.51.100.0/24 [20/0] via 10.2.2.2, eth3 (vrf Internet), weight 1, 00:45:24 + + # show ipv6 route vrf LAN2 + + C>* 2001:db8::2/127 is directly connected, eth1, 00:46:26 + B>* 2001:db8:0:1::/64 [20/0] via 2001:db8::1, eth0 (vrf LAN1), weight 1, 00:46:17 + S>* 2001:db8:0:2::/64 [1/0] via 2001:db8::3, eth1, weight 1, 00:46:21 + B>* 2001:db8:0:3::/64 [20/0] via 2001:db8::5, eth2 (vrf Managment), weight 1, 00:46:16 + B>* 2001:db8:1::/48 [20/0] via fe80::5200:ff:fe02:3, eth3 (vrf Internet), weight 1, 00:46:13 + B>* 2001:db8:2::/48 [20/0] via fe80::5200:ff:fe02:3, eth3 (vrf Internet), weight 1, 00:46:13 + C>* fe80::/64 is directly connected, eth1, 00:46:27 + +As we can see even if both VRF LAN1 and LAN2 has the same import RTs we are able +to select which routes are effectively imported and installed. + -- cgit v1.2.3 From 86271c1d8358839c5815cb2fbe90a8495b330be7 Mon Sep 17 00:00:00 2001 From: "L. Repetto" Date: Wed, 20 Apr 2022 16:17:50 -0300 Subject: correct ident, remove titles and image. --- docs/configexamples/inter-vrf-routing-vrf-lite.rst | 69 ++++++++++++++-------- 1 file changed, 44 insertions(+), 25 deletions(-) (limited to 'docs') diff --git a/docs/configexamples/inter-vrf-routing-vrf-lite.rst b/docs/configexamples/inter-vrf-routing-vrf-lite.rst index d5c13e35..e109c12c 100644 --- a/docs/configexamples/inter-vrf-routing-vrf-lite.rst +++ b/docs/configexamples/inter-vrf-routing-vrf-lite.rst @@ -16,6 +16,8 @@ where you might need that some network can access other in a different VRF. The scope of this document is to cover such cases in a dynamic way without the use of MPLS-LDP. +General information about L3VPNs can be found in the :ref:`configuration/vrf/index:L3VPN VRFs` chapter. + ******** Overview ******** @@ -68,12 +70,14 @@ community(ies) into that prefix. ******** Topology ******** - .. image:: /_static/images/inter-vrf-routing-vrf-lite.png :width: 70% - :align: left + :align: center :alt: Network Topology Diagram + + + IP Schema ========= @@ -133,7 +137,8 @@ Step 1: VRF and Configurations to remote networks ------------------------------------------------- - Configuration -^^^^^^^^^^^^^^^ + + Set the VRF name and Table ID, set interface address and bind it to the VRF. Last add the static route to the remote network. @@ -153,7 +158,8 @@ Last add the static route to the remote network. set vrf name protocols static route next-hop - Verification -^^^^^^^^^^^^^^ + + Checking the routing table of the VRF should reveal both static and connected entries active. A PING test between the Core and remote router is a way to @@ -221,8 +227,10 @@ validate connectivity within the VRF. Step 2: BGP Configuration for VRF-Lite -------------------------------------- + - Configuration -^^^^^^^^^^^^^^^ + + Setting BGP global local-as as well inside the VRF. Redistribute static routes to inject configured networks into the BGP process but still inside the VRF. @@ -238,7 +246,8 @@ to inject configured networks into the BGP process but still inside the VRF. set vrf name protocols bgp address-family redistribute static - Verification -^^^^^^^^^^^^^^ + + Check the BGP VRF table and verify if the static routes are injected showing the correct next-hop information. @@ -277,8 +286,9 @@ the correct next-hop information. Step 3: VPN Configuration ------------------------- + - Configuration -^^^^^^^^^^^^^^^ + Within the VRF we set the Route-Distinguisher (RD) and Route-Targets (RT), then we enable the export/import VPN. @@ -308,7 +318,8 @@ There are some cases where this is not needed -for example, in some DDoS appliance- but most inter-vrf routing designs use the above configurations. - Verification -^^^^^^^^^^^^^^ + + After configured all the VRFs involved in this topology we take a deeper look at both BGP and Routing table for the VRF LAN1 @@ -410,10 +421,11 @@ installed, we can see -between round brackets- the exported VRF table. Step 4: End to End verification ------------------------------- + Now we perform some end-to-end testing - From Management to LAN1/LAN2 -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + .. code-block:: none @@ -455,7 +467,8 @@ Now we perform some end-to-end testing rtt min/avg/max/mdev = 1.660/1.960/2.315/0.236 ms - From Management to Outside (fails as intended) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + .. code-block:: none @@ -505,7 +518,8 @@ Now we perform some end-to-end testing - LAN1 to Outside -^^^^^^^^^^^^^^^^^ + + .. code-block:: none @@ -559,7 +573,8 @@ Now we perform some end-to-end testing route and ping will fail. - LAN1 to LAN2 -^^^^^^^^^^^^^^ + + .. code-block:: none @@ -596,10 +611,10 @@ Appendix-A ********** Full configuration from all devices ------------------------------------ +=================================== - Core -^^^^^^ + .. code-block:: none @@ -684,7 +699,7 @@ Full configuration from all devices - LAN1 -^^^^^^ + .. code-block:: none @@ -696,9 +711,11 @@ Full configuration from all devices set protocols static route6 ::/0 next-hop 2001:db8::* - LAN2 -^^^^^^ + + .. code-block:: none + set interfaces dummy dum0 address '172.16.0.1/24' set interfaces dummy dum0 address '2001:db8:0:2::1/64' set interfaces ethernet eth0 hw-id '50:00:00:03:00:00' @@ -708,7 +725,7 @@ Full configuration from all devices set protocols static route6 ::/0 next-hop 2001:db8::2 - Management -^^^^^^^^^^^^ + .. code-block:: none @@ -720,7 +737,7 @@ Full configuration from all devices set protocols static route6 ::/0 next-hop 2001:db8::4 - ISP -^^^^^ + .. code-block:: none @@ -747,14 +764,16 @@ Appendix-B ********** Route-Filtering ---------------- +=============== + When importing routes using MP-BGP it is possible to filter a subset of them before are injected in the BGP table. One of the most common case is to use a route-map with an prefix-list. - Configuration -^^^^^^^^^^^^^^^ + + We create a prefix-list first and add all the routes we need to. @@ -794,11 +813,11 @@ actions inside the rules of the route-map. set policy route-map LAN2-Internet-v6 rule 1 match ipv6 address prefix-list 'LAN2-Internet-v6' We are using a "white list" approach by allowing only what is necessary. In case -that need to implement a "black list" approach you will ned to change the action -in the route-map for a deny BUT you need to add a rule that permit the rest due -to the implicit deny in the route-map. +that need to implement a "black list" approach then you will need to change the +action in the route-map for a deny BUT you need to add a rule that permits the +rest due to the implicit deny in the route-map. -Then we need to attach the policy into the bgp process. This needs to be under +Then we need to attach the policy to the BGP process. This needs to be under the import statement in the vrf we need to filter. .. code-block:: none @@ -808,7 +827,7 @@ the import statement in the vrf we need to filter. - Verification -^^^^^^^^^^^^^^ + .. code-block:: none -- cgit v1.2.3