Merge pull request #595 from sever-sever/automate

automate: Add netmiko and napalm examples

5 files changed, 432 insertions, 2 deletions

diff --git a/docs/automation/cloud-init.rst b/docs/automation/cloud-init.rst
new file mode 100644
index 00000000..2be66f7f
--- /dev/null
+++ b/docs/automation/cloud-init.rst
@@ -0,0 +1,118 @@
+:lastproofread: 2021-07-12
+
+.. _cloud-init:
+
+###############
+VyOS cloud-init
+###############
+
+Cloud and virtualized instances of VyOS are initialized using the
+industry-standard cloud-init. Via cloud-init, the system performs tasks such as
+injecting SSH keys and configuring the network. In addition, the user can supply
+a custom configuration at the time of instance launch.
+
+**************
+Config Sources
+**************
+
+VyOS support three types of config sources.
+
+* Metadata - Metadata is sourced by the cloud platform or hypervisor.
+  In some clouds, there is implemented as an HTTP endpoint at
+  http://169.254.169.254.
+* Network configuration - This config source informs the system about the
+  network settings like IP addresses, routes, DNS. Available only in several
+  cloud and virtualization platforms.
+* User-data - User-data is specified by the user. This config source offers the
+  ability to insert any CLI configuration commands into the configuration before
+  the first boot.
+
+*********
+User-data
+*********
+
+Major cloud providers offer a means of providing user-data at the time of
+instance launch. It can be provided as plain text or as base64-encoded text,
+depending on cloud provider. Also, it can be compressed using gzip, which makes
+sense with a long configuration commands list, because of the hard limit to
+~16384 bytes for the whole user-data.
+
+The easiest way to configure the system via user-data is the Cloud-config syntax
+described below.
+
+********************
+Cloud-config modules
+********************
+
+In VyOS, by default, enabled only two modules:
+
+* ``write_files`` - this module allows to insert any files into the filesystem
+  before the first boot, for example, pre-generated encryption keys,
+  certificates, or even a whole ``config.boot`` file.
+* ``vyos_userdata`` - the module accepts a list of CLI configuration commands in
+  a ``vyos_config_commands`` section, which gives an easy way to configure the
+  system during deployment.
+
+************************
+cloud-config file format
+************************
+
+A cloud-config document is written in YAML. The file must begin
+with ``#cloud-config`` line. The key used to designate a VyOS configuration
+is ``vyos_config_commands``. What follows is VyOS configuration using
+the "set-style" syntax. Both "set" and "delete" commands are supported.
+
+Commands requirements:
+
+* one command per line
+* if command ends in a value, it must be inside single quotes
+* a single-quote symbol is not allowed inside command or value
+
+The commands list produced by the ``show configuration commands`` command on a
+VyOS router should comply with all the requirements, so it is easy to get a 
+proper commands list by copying it from another router.
+
+The configuration specified in the cloud-config document overwrites default
+configuration values and values configured via Metadata.
+
+Here is an example cloud-config.
+
+.. code-block:: yaml
+
+   #cloud-config
+   vyos_config_commands:
+     - set system host-name 'vyos-prod-ashburn'
+     - set system ntp server 1.pool.ntp.org
+     - set system ntp server 2.pool.ntp.org
+     - delete interfaces ethernet eth1 address 'dhcp'
+     - set interfaces ethernet eth1 address '192.0.2.247/24'
+     - set protocols static route 198.51.100.0/24 next-hop '192.0.2.1'
+
+*************************
+System Defaults/Fallbacks
+*************************
+
+These are the VyOS defaults and fallbacks.
+
+* SSH is configured on port 22
+* ``vyos``/``vyos`` credentials if no others specified by data source
+* DHCP on first Ethernet interface if no network configuration is provided
+
+All of these can be overridden using the configuration in user-data.
+
+***************
+Troubleshooting
+***************
+
+If you encounter problems, verify that the cloud-config document contains
+valid YAML. Online resources such as https://yamlvalidator.com/ provide
+a simple tool for validating YAML.
+
+cloud-init logs to /var/log/cloud-init.log. This file can be helpful in
+determining why the configuration varies from what you expect. You can fetch the
+most important data filtering output for ``vyos`` keyword:
+
+.. code-block:: none
+
+    sudo grep vyos /var/log/cloud-init.log
+
diff --git a/docs/automation/index.rst b/docs/automation/index.rst
index c19d819b..2bb11a4d 100644
--- a/docs/automation/index.rst
+++ b/docs/automation/index.rst
@@ -3,7 +3,6 @@ VyOS Automation
 ###############
 
 
-  * Ansible
   * Saltstack
   * startup scripts
 
@@ -12,4 +11,8 @@ VyOS Automation
    :maxdepth: 1
    
    vyos-api
-   command-scripting
\ No newline at end of file
+   vyos-ansible
+   vyos-napalm
+   vyos-netmiko
+   command-scripting
+   cloud-init
\ No newline at end of file
diff --git a/docs/automation/vyos-ansible.rst b/docs/automation/vyos-ansible.rst
new file mode 100644
index 00000000..e02d06a9
--- /dev/null
+++ b/docs/automation/vyos-ansible.rst
@@ -0,0 +1,93 @@
+:lastproofread: 2021-06-28
+
+.. _vyos-ansible:
+
+Ansible
+=======
+
+VyOS supports configuration via ansible.
+Need to install ``ansible`` and ``python3-paramiko`` module
+
+Structure of files
+
+.. code-block:: none
+
+ .
+ ├── ansible.cfg
+ ├── files
+ │   └── id_rsa_docker.pub
+ ├── hosts
+ └── main.yml
+
+
+File contents
+-------------
+
+ansible.cfg
+
+.. code-block:: none
+
+  [defaults]
+  host_key_checking = no
+  retry_files_enabled = False
+  ANSIBLE_INVENTORY_UNPARSED_FAILED = true
+
+id_rsa_docker.pub. Needs to declare only public key exactly.
+
+.. code-block:: none
+
+  AAAAB3NzaC1yc2EAAAADAQABAAABAQCoDgfhQJuJRFWJijHn7ZinZ3NWp4hWVrt7HFcvn0kgtP/5PeCtMt
+
+
+hosts
+
+.. code-block:: none
+
+  [vyos_hosts]
+  r11 ansible_ssh_host=192.0.2.11
+
+  [vyos_hosts:vars]
+  ansible_python_interpreter=/usr/bin/python3
+  ansible_user=vyos
+  ansible_ssh_pass=vyos
+  ansible_network_os=vyos
+  ansible_connection=network_cli
+
+main.yml
+
+.. code-block:: none
+
+  ---
+
+  - hosts: r11
+
+    connection: network_cli
+    gather_facts: 'no'
+
+    tasks:
+      - name: Configure remote r11
+        vyos_config:
+          lines:
+            - set system host-name r11
+            - set system name-server 203.0.113.254
+            - set service ssh disable-host-validation
+            - set system login user vyos authentication public-keys docker@work type ssh-rsa
+            - set system login user vyos authentication public-keys docker@work key "{{ lookup('file', 'id_rsa_docker.pub') }}"
+            - set system time-zone America/Los_Angeles
+            - set interfaces ethernet eth0 description WAN
+
+Run ansible
+-----------
+
+.. code-block:: none
+
+  $ ansible-playbook -i hosts main.yml 
+
+  PLAY [r11] ******************************************************************************************************************************************************************************************************
+
+  TASK [Configure remote r11] *************************************************************************************************************************************************************************************
+  changed: [r11]
+
+  PLAY RECAP *****************************************************************************************************************************************************************************************************
+  r11                         : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
+
diff --git a/docs/automation/vyos-napalm.rst b/docs/automation/vyos-napalm.rst
new file mode 100644
index 00000000..5ff7c01f
--- /dev/null
+++ b/docs/automation/vyos-napalm.rst
@@ -0,0 +1,144 @@
+:lastproofread: 2021-06-28
+
+.. _vyos-napalm:
+
+Napalm
+======
+
+VyOS supports some napalm_ functions for configuration and op-mode.
+It requires more tests.
+
+Install ``napalm-vyos`` module
+
+.. code-block:: none
+
+  apt install python3-pip
+  pip3 install napalm
+  pip3 install napalm-vyos
+
+
+Op-mode
+-------
+
+.. code-block:: none
+
+  #!/usr/bin/env python3
+
+  import json
+  from napalm import get_network_driver
+
+  driver = get_network_driver('vyos')
+
+  vyos_router = driver(
+      hostname="192.0.2.1",
+      username="vyos",
+      password="vyospass",
+      optional_args={"port": 22},
+  )
+
+  vyos_router.open()
+  output = vyos_router.get_facts()
+  print(json.dumps(output, indent=4))
+
+  output = vyos_router.get_arp_table()
+  print(json.dumps(output, indent=4))
+
+  vyos_router.close()
+
+Output op-mode
+
+.. code-block:: none
+
+  $ ./vyos-napalm.py
+  {
+      "uptime": 7185,
+      "vendor": "VyOS",
+      "os_version": "1.3.0-rc5",
+      "serial_number": "",
+      "model": "Standard PC (Q35 + ICH9, 2009)",
+      "hostname": "r4-1.3",
+      "fqdn": "vyos.local",
+      "interface_list": [
+          "eth0",
+          "eth1",
+          "eth2",
+          "lo",
+          "vtun10"
+      ]
+  }
+  [
+      {
+          "interface": "eth1",
+          "mac": "52:54:00:b2:38:2c",
+          "ip": "192.0.2.2",
+          "age": 0.0
+      },
+      {
+          "interface": "eth0",
+          "mac": "52:54:00:a2:b9:5b",
+          "ip": "203.0.113.11",
+          "age": 0.0
+      }
+  ]
+
+Configuration
+-------------
+
+We need 2 files, commands.conf and script itself.
+
+Content of commands.conf
+
+.. code-block:: none
+
+  set service ssh disable-host-validation
+  set service ssh port '2222'
+  set system name-server '192.0.2.8'
+  set system name-server '203.0.113.8'
+  set interfaces ethernet eth1 description 'FOO'
+
+Script vyos-napalm.py
+
+.. code-block:: none
+
+  #!/usr/bin/env python3
+
+  from napalm import get_network_driver
+
+  driver = get_network_driver('vyos')
+
+  vyos_router = driver(
+      hostname="192.0.2.1",
+      username="vyos",
+      password="vyospass",
+      optional_args={"port": 22},
+  )
+
+  vyos_router.open()
+  vyos_router.load_merge_candidate(filename='commands.conf')
+  diffs = vyos_router.compare_config()
+
+  if bool(diffs) == True:
+      print(diffs)
+      vyos_router.commit_config()
+  else:
+      print('No configuration changes to commit')
+      vyos_router.discard_config()
+
+  vyos_router.close()
+
+Output
+
+.. code-block:: none
+
+  $./vyos-napalm.py 
+  [edit interfaces ethernet eth1]
+  +description FOO
+  [edit service ssh]
+  +disable-host-validation
+  +port 2222
+  [edit system]
+  +name-server 192.0.2.8
+  +name-server 203.0.113.8
+  [edit]
+
+.. _napalm: https://napalm.readthedocs.io/en/latest/base.html
\ No newline at end of file
diff --git a/docs/automation/vyos-netmiko.rst b/docs/automation/vyos-netmiko.rst
new file mode 100644
index 00000000..3627e5c3
--- /dev/null
+++ b/docs/automation/vyos-netmiko.rst
@@ -0,0 +1,72 @@
+:lastproofread: 2021-06-28
+
+.. _vyos-netmiko:
+
+Netmiko
+=======
+
+VyOS supports configuration via netmiko_.
+It requires to install ``python3-netmiko`` module.
+
+Example
+-------
+
+.. code-block:: none
+
+  #!/usr/bin/env python3
+
+  from netmiko import ConnectHandler
+
+  vyos_router = {
+    "device_type": "vyos",
+    "host": "192.0.2.1",
+    "username": "vyos",
+    "password": "vyospass",
+    "port": 22,
+    }
+
+  net_connect = ConnectHandler(**vyos_router)
+
+  config_commands = [
+                     'set interfaces ethernet eth0 description WAN',
+                     'set interfaces ethernet eth1 description LAN',
+                    ]
+
+  # set congiguration
+  output = net_connect.send_config_set(config_commands, exit_config_mode=False)
+  print(output)
+
+  # commit configuration
+  output = net_connect.commit()
+  print(output)
+
+  # op-mode commands
+  output = net_connect.send_command("run show interfaces")
+  print(output)
+
+Output
+
+.. code-block:: none
+
+  $ ./vyos-netmiko.py
+  configure
+  set interfaces ethernet eth0 description WAN
+  [edit]
+  vyos@r4-1.3# set interfaces ethernet eth1 description LAN
+  [edit]
+  vyos@r4-1.3# 
+  commit
+  [edit]
+  vyos@r4-1.3# 
+  Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
+  Interface        IP Address                        S/L  Description
+  ---------        ----------                        ---  -----------
+  eth0             203.0.113.1/24                    u/u  WAN
+  eth1             192.0.2.1/30                      u/u  LAN
+  eth2             -                                 u/u  
+  lo               127.0.0.1/8                       u/u  
+                   ::1/128                                
+  vtun10           10.10.0.1/24                      u/u  
+  [edit]
+
+.. _netmiko: https://github.com/ktbyers/netmiko
\ No newline at end of file