VyOS v1.4 support for BGP (T6892 & T6888 & T6822) (#367)

* VyOS v1.4 support for BGP (T6892)

* WIP: chnages to scanners and facts for BGP on v1.4

* scanners and facts modification to support as_numberless commands

* remove optional regex groups

* VyOS v1.4 BGP system-as  functionality

* bgp_af linter

* bgp_global mods and testing

* bgp_global v.1.4 support

* linter fixes

* bgp_global and bgp_af fixed 1.3 test cases

* unit tests for bgp modules for both versions

* obsolete stranzas removed from bgp_global for both versions

* some typos removed

* bgp_global rst updates

* t6829 related draft changes

* Draft changes to unit tests fot original set and t6888 updates

* changelog

* D.R.Y for bgp_global and testmodule names

* linter fixes

* fixtures for bgp_af options  testts

* stranzas options processing

* T6888 bgp option test func and  cases

* clean-up draft code and update to rm_templates

* v14 system-as change supported in conf and tests

* T6822 BGP global passive bugfix

* clean-up

* T6829: update integration tests

* T6829: fix integration tests for global

* T6829: fix integration tests for bgp_global

* T6829: fix for 1.4 AF integration tests

* T6829: fix unit tests after removing obsolete items

* T6829: fix sanity test failures

* T6829: fix documentation

* Lint and comments are addressed

* rtt.yaml and fix to integration tests

* updated fragments for bgp_global

* lint fixes

---------

Co-authored-by: Gaige B. Paulsen <gaige@cluetrust.com>

2 files changed, 210 insertions, 164 deletions

diff --git a/plugins/modules/vyos_bgp_address_family.py b/plugins/modules/vyos_bgp_address_family.py
index b3756e2e..14c3605d 100644
--- a/plugins/modules/vyos_bgp_address_family.py
+++ b/plugins/modules/vyos_bgp_address_family.py
@@ -19,6 +19,7 @@ version_added: 1.0.0
 short_description: BGP Address Family resource module
 description:
 - This module manages BGP address family configuration of interfaces on devices running VYOS.
+- Tested against VYOS 1.3, 1.4
 author: Gomathi Selvi Srinivasan (@GomathiselviS)
 options:
   config:
diff --git a/plugins/modules/vyos_bgp_global.py b/plugins/modules/vyos_bgp_global.py
index 7d47e4ad..4d7db472 100644
--- a/plugins/modules/vyos_bgp_global.py
+++ b/plugins/modules/vyos_bgp_global.py
@@ -19,6 +19,7 @@ version_added: 1.0.0
 short_description: BGP global resource module
 description:
 - This module manages BGP global configuration of interfaces on devices running VYOS.
+- Tested against VYOS 1.3, 1.4
 author:
 - Gomathi Selvi Srinivasan (@GomathiselviS)
 options:
@@ -30,35 +31,35 @@ options:
         description:
         - AS number.
         type: int
-      aggregate_address:
-        description:
-        - BGP aggregate network.
-        type: list
-        elements: dict
-        suboptions:
-          prefix:
-            description:
-            - BGP aggregate network.
-            type: str
-          as_set:
-            description:
-            - Generate AS-set path information for this aggregate address.
-            type: bool
-          summary_only:
-            description:
-            - Announce the aggregate summary network only.
-            type: bool
-      maximum_paths:
-        description: BGP multipaths
-        type: list
-        elements: dict
-        suboptions:
-          path:
-            description: BGP multipaths
-            type: str
-          count:
-            description: No. of paths.
-            type: int
+      # aggregate_address:
+      #   description:
+      #   - BGP aggregate network.
+      #   type: list
+      #   elements: dict
+      #   suboptions:
+      #     prefix:
+      #       description:
+      #       - BGP aggregate network.
+      #       type: str
+      #     as_set:
+      #       description:
+      #       - Generate AS-set path information for this aggregate address.
+      #       type: bool
+      #     summary_only:
+      #       description:
+      #       - Announce the aggregate summary network only.
+      #       type: bool
+      #maximum_paths: --> moved to address-family before 1.3
+      #  description: BGP multipaths
+      #  type: list
+      #  elements: dict
+      #  suboptions:
+      #    path:
+      #      description: BGP multipaths
+      #      type: str
+      #    count:
+      #      description: No. of paths.
+      #      type: int
       neighbor:
         description: BGP neighbor
         type: list
@@ -72,28 +73,37 @@ options:
             description:
             - Minimum interval for sending routing updates.
             type: int
-          allowas_in:
-            description:
-            - Number of occurrences of AS number.
-            type: int
-          as_override:
-            description:
-            - AS for routes sent to this neighbor to be the local AS.
-            type: bool
-          attribute_unchanged:
-            description:
-            - BGP attributes are sent unchanged.
-            type: dict
-            suboptions:
-              as_path:
-                description: as_path
-                type: bool
-              med:
-                description: med
-                type: bool
-              next_hop:
-                description: next_hop
-                type: bool
+      #    bfd: # <-- added in 1.3
+      #      description: Enable Bidirectional Forwarding Detection (BFD) support
+      #      type: dict
+      #      suboptions:
+      #        check-control-plane-failure:
+      #          description:
+      #          - Allow to write CBIT independence in BFD outgoing packets
+      #            and read both C-BIT value of BFD and lookup BGP peer status
+      #          type: bool
+      #    allowas_in:   --> Moved to address-family before 1.3
+      #      description:
+      #      - Number of occurrences of AS number.
+      #      type: int
+      #    as_override:  --> Moved to address-family before 1.3
+      #      description:
+      #      - AS for routes sent to this neighbor to be the local AS.
+      #      type: bool
+      #    attribute_unchanged: --> Moved to address-family before 1.3
+      #      description:
+      #      - BGP attributes are sent unchanged.
+      #      type: dict
+      #      suboptions:
+      #        as_path:
+      #          description: as_path
+      #          type: bool
+      #        med:
+      #          description: med
+      #          type: bool
+      #        next_hop:
+      #          description: next_hop
+      #          type: bool
           capability:
             description:
             - Advertise capabilities to this neighbor.
@@ -103,13 +113,17 @@ options:
                 description:
                 - Advertise dynamic capability to this neighbor.
                 type: bool
-              orf:
+              extended_nexthop:
                 description:
-                 - Advertise ORF capability to this neighbor.
-                type: str
-                choices:
-                - send
-                - receive
+                - Advertise extended nexthop capability to this neighbor.
+                type: bool
+      #       orf:   --> Removed before 1.3
+      #         description:
+      #          - Advertise ORF capability to this neighbor.
+      #         type: str
+      #         choices:
+      #         - send
+      #         - receive
           default_originate:
             description:
             - Send default route to this neighbor
@@ -131,45 +145,70 @@ options:
             - Disable sending community attributes to this neighbor.
             type: str
             choices: ['extended', 'standard']
-          distribute_list:
-            description: Access-list to filter route updates to/from this neighbor.
-            type: list
-            elements: dict
-            suboptions:
-              action:
-                description: Access-list to filter outgoing/incoming route updates to this neighbor
-                type: str
-                choices: ['export', 'import']
-              acl:
-                description: Access-list number.
-                type: int
+      #   distribute_list:  --> Moved to address-family before 1.3
+      #     description: Access-list to filter route updates to/from this neighbor.
+      #     type: list
+      #     elements: dict
+      #     suboptions:
+      #       action:
+      #         description: Access-list to filter outgoing/incoming route updates to this neighbor
+      #         type: str
+      #         choices: ['export', 'import']
+      #       acl:
+      #         description: Access-list number.
+      #         type: int
           ebgp_multihop:
             description:
               - Allow this EBGP neighbor to not be on a directly connected network. Specify
                 the number hops.
             type: int
-          filter_list:
-            description: As-path-list to filter route updates to/from this neighbor.
-            type: list
-            elements: dict
-            suboptions:
-              action:
-                description: filter outgoing/incoming route updates
-                type: str
-                choices: ['export', 'import']
-              path_list:
-                description: As-path-list to filter
-                type: str
+      #    interface: # <-- added in 1.3
+      #      description: interface parameters
+      #      type: dict
+      #      suboptions:
+      #        peer_group:
+      #          description: Peer group for this neighbor
+      #          type: str
+      #        remote_as:
+      #          description:
+      #          - Remote AS number
+      #          - Or 'external' for any number except this AS number
+      #          - or 'internal' for this AS number
+      #          type: str
+      #        v6only:
+      #          description: Enable BGP with v6 link-local only
+      #          type: dict
+      #          suboptions:
+      #            peer_group:
+      #              description: Peer group for this neighbor
+      #              type: str
+      #            remote_as:
+      #              description:
+      #              - Remote AS number
+      #              - Or 'external' for any number except this AS number
+      #              - or 'internal' for this AS number
+      #    filter_list:  --> Moved to address-family before 1.3
+      #      description: As-path-list to filter route updates to/from this neighbor.
+      #      type: list
+      #      elements: dict
+      #      suboptions:
+      #        action:
+      #          description: filter outgoing/incoming route updates
+      #          type: str
+      #          choices: ['export', 'import']
+      #        path_list:
+      #          description: As-path-list to filter
+      #          type: str
           local_as:
             description: local as number not to be prepended to updates from EBGP peers
             type: int
-          maximum_prefix:
-            description: Maximum number of prefixes to accept from this neighbor
-               nexthop-self Nexthop for routes sent to this neighbor to be the local router.
-            type: int
-          nexthop_self:
-            description: Nexthop for routes sent to this neighbor to be the local router.
-            type: bool
+      #   maximum_prefix:  --> Moved to address-family before 1.3
+      #     description: Maximum number of prefixes to accept from this neighbor
+      #        nexthop-self Nexthop for routes sent to this neighbor to be the local router.
+      #     type: int
+      #   nexthop_self: --> Moved to address-family before 1.3
+      #     description: Nexthop for routes sent to this neighbor to be the local router.
+      #     type: bool
           override_capability:
             description: Ignore capability negotiation with specified neighbor.
             type: bool
@@ -188,63 +227,61 @@ options:
           port:
             description: Neighbor's BGP port
             type: int
-          prefix_list:
-            description: Prefix-list to filter route updates to/from this neighbor.
-            type: list
-            elements: dict
-            suboptions:
-              action:
-                description: filter outgoing/incoming route updates
-                type: str
-                choices: ['export', 'import']
-              prefix_list:
-                description: Prefix-list to filter
-                type: str
+      #    prefix_list:  --> Moved to address-family before 1.3
+      #      description: Prefix-list to filter route updates to/from this neighbor.
+      #      type: list
+      #      elements: dict
+      #      suboptions:
+      #        action:
+      #          description: filter outgoing/incoming route updates
+      #          type: str
+      #          choices: ['export', 'import']
+      #        prefix_list:
+      #          description: Prefix-list to filter
+      #          type: str
           remote_as:
             description: Neighbor BGP AS number
             type: int
-          remove_private_as:
-            description: Remove private AS numbers from AS path in outbound route updates
-            type: bool
-          route_map:
-            description: Route-map to filter route updates to/from this neighbor.
-            type: list
-            elements: dict
-            suboptions:
-              action:
-                description: filter outgoing/incoming route updates
-                type: str
-                choices: ['export', 'import']
-              route_map:
-                description: route-map to filter
-                type: str
-          route_reflector_client:
-            description: Neighbor as a route reflector client
-            type: bool
-          route_server_client:
-            description: Neighbor is route server client
-            type: bool
+      #    remove_private_as: --> Moved to address-family before 1.3
+      #      description: Remove private AS numbers from AS path in outbound route updates
+      #      type: bool
+      #   route_map:    --> Moved to address-family before 1.3
+      #     description: Route-map to filter route updates to/from this neighbor.
+      #     type: list
+      #     elements: dict
+      #     suboptions:
+      #       action:
+      #         description: filter outgoing/incoming route updates
+      #         type: str
+      #         choices: ['export', 'import']
+      #       route_map:
+      #         description: route-map to filter
+      #         type: str
+      #    route_reflector_client:   --> Moved to address-family before 1.3
+      #      description: Neighbor as a route reflector client
+      #      type: bool
+      #    route_server_client:   --> Removed prior to 1.3
+      #      description: Neighbor is route server client
+      #      type: bool
           shutdown:
             description: Administratively shut down neighbor
             type: bool
-          soft_reconfiguration:
-            description: Soft reconfiguration for neighbor
+      #    soft_reconfiguration:  --> Moved to address-family before 1.3
+      #      description: Soft reconfiguration for neighbor
+      #      type: bool
+          solo: # <-- added in 1.3
+            description: Do not send back prefixes learned from the neighbor
             type: bool
           strict_capability_match:
             description: Enable strict capability negotiation
             type: bool
-          unsuppress_map:
-            description: Route-map to selectively unsuppress suppressed routes
-            type: str
-          update_source:
-            description: Source IP of routing updates
-            type: str
-          weight:
-            description: Default weight for routes from this neighbor
-            type: int
-          ttl_security:
-            description: Ttl security mechanism for this BGP peer
-            type: int
+      #    unsuppress_map:   --> Moved to address-family before 1.3
+      #      description: Route-map to selectively unsuppress suppressed routes
+      #      type: str
+
+      #    weight:    --> Moved to address-family before 1.3
+      #      description: Default weight for routes from this neighbor
+      #      type: int
           timers:
             description: Neighbor timers
             type: dict
@@ -258,35 +295,41 @@ options:
               keepalive:
                 description: BGP keepalive interval for this neighbor
                 type: int
-      network:
-        description: BGP network
-        type: list
-        elements: dict
-        suboptions:
-          address:
-            description: BGP network address
-            type: str
-          backdoor:
-            description: Network as a backdoor route
-            type: bool
-          route_map:
-            description: Route-map to modify route attributes
-            type: str
-      redistribute:
-        description: Redistribute routes from other protocols into BGP
-        type: list
-        elements: dict
-        suboptions:
-          protocol:
-            description: types of routes to be redistributed.
-            type: str
-            choices: ['connected', 'kernel', 'ospf', 'rip', 'static']
-          route_map:
-            description: Route map to filter redistributed routes
-            type: str
-          metric:
-            description: Metric for redistributed routes.
+          ttl_security:
+            description: Number of the maximum number of hops to the BGP peer
             type: int
+          update_source:
+            description: Source IP of routing updates
+            type: str
+      # network:
+      #   description: BGP network
+      #   type: list
+      #   elements: dict
+      #   suboptions:
+      #     address:
+      #       description: BGP network address
+      #       type: str
+      #     backdoor:
+      #       description: Network as a backdoor route
+      #       type: bool
+      #     route_map:
+      #       description: Route-map to modify route attributes
+      #       type: str
+      # redistribute:
+      #   description: Redistribute routes from other protocols into BGP
+      #   type: list
+      #   elements: dict
+      #   suboptions:
+      #     protocol:
+      #       description: types of routes to be redistributed.
+      #       type: str
+      #       choices: ['connected', 'kernel', 'ospf', 'rip', 'static']
+      #     route_map:
+      #       description: Route map to filter redistributed routes
+      #       type: str
+      #     metric:
+      #       description: Metric for redistributed routes.
+      #       type: int
       timers:
         description: BGP protocol timers
         type: dict
@@ -357,7 +400,9 @@ options:
                 description: Default local preference
                 type: int
               no_ipv4_unicast:
-                description: Deactivate IPv4 unicast for a peer by default
+                description: |
+                  Deactivate IPv4 unicast for a peer by default
+                  Deprecated: Unavailable after 1.4
                 type: bool
           deterministic_med:
             description: Compare MEDs between different peers in the same AS