summaryrefslogtreecommitdiff
path: root/docs/configuration/interfaces/bonding.rst
blob: 8ec8f34dd45bbc8546f56fd21d58e2343a8b7c0e (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
.. _bond-interface:

#######################
Bond / Link Aggregation
#######################

The bonding interface provides a method for aggregating multiple network
interfaces into a single logical "bonded" interface, or LAG, or ether-channel,
or port-channel. The behavior of the bonded interfaces depends upon the mode;
generally speaking, modes provide either hot standby or load balancing services.
Additionally, link integrity monitoring may be performed.

*************
Configuration
*************

Common interface configuration
==============================

.. cmdinclude:: /_include/interface-common-with-dhcp.txt
   :var0: bond
   :var1: bond0

Member Interfaces
=================

.. cfgcmd:: set interfaces bonding <interface> member interface <member>

   Enslave `<member>` interface to bond `<interface>`.

Bond options
============

.. cfgcmd:: set interfaces bonding <interface> mode <802.3ad | active-backup |
  broadcast | round-robin | transmit-load-balance | adaptive-load-balance |
  xor-hash>

   Specifies one of the bonding policies. The default is 802.3ad. Possible
   values are:

   * ``802.3ad`` - IEEE 802.3ad Dynamic link aggregation. Creates aggregation
     groups that share the same speed and duplex settings. Utilizes all slaves
     in the active aggregator according to the 802.3ad specification.

     Slave selection for outgoing traffic is done according to the transmit
     hash policy, which may be changed from the default simple XOR policy via
     the :cfgcmd:`hash-policy` option, documented below.

     .. note:: Not all transmit policies may be 802.3ad compliant, particularly
        in regards to the packet mis-ordering requirements of section 43.2.4
        of the 802.3ad standard.

   * ``active-backup`` - Active-backup policy: Only one slave in the bond is
     active. A different slave becomes active if, and only if, the active slave
     fails. The bond's MAC address is externally visible on only one port
     (network adapter) to avoid confusing the switch.

     When a failover occurs in active-backup mode, bonding will issue one or
     more gratuitous ARPs on the newly active slave. One gratuitous ARP is
     issued for the bonding master interface and each VLAN interfaces
     configured above it, provided that the interface has at least one IP
     address configured. Gratuitous ARPs issued for VLAN interfaces are tagged
     with the appropriate VLAN id.

     This mode provides fault tolerance. The :cfgcmd:`primary` option,
     documented below, affects the behavior of this mode.

   * ``broadcast`` - Broadcast policy: transmits everything on all slave
     interfaces.

     This mode provides fault tolerance.

   * ``round-robin`` - Round-robin policy: Transmit packets in sequential
     order from the first available slave through the last.

     This mode provides load balancing and fault tolerance.

   * ``transmit-load-balance`` - Adaptive transmit load balancing: channel
     bonding that does not require any special switch support.

     Incoming traffic is received by the current slave. If the receiving slave
     fails, another slave takes over the MAC address of the failed receiving
     slave.

   * ``adaptive-load-balance`` - Adaptive load balancing: includes
     transmit-load-balance plus receive load balancing for IPV4 traffic, and
     does not require any special switch support. The receive load balancing
     is achieved by ARP negotiation. The bonding driver intercepts the ARP
     Replies sent by the local system on their way out and overwrites the
     source hardware address with the unique hardware address of one of the
     slaves in the bond such that different peers use different hardware
     addresses for the server.

     Receive traffic from connections created by the server is also balanced.
     When the local system sends an ARP Request the bonding driver copies and
     saves the peer's IP information from the ARP packet. When the ARP Reply
     arrives from the peer, its hardware address is retrieved and the bonding
     driver initiates an ARP reply to this peer assigning it to one of the
     slaves in the bond. A problematic outcome of using ARP negotiation for
     balancing is that each time that an ARP request is broadcast it uses the
     hardware address of the bond. Hence, peers learn the hardware address
     of the bond and the balancing of receive traffic collapses to the current
     slave. This is handled by sending updates (ARP Replies) to all the peers
     with their individually assigned hardware address such that the traffic
     is redistributed. Receive traffic is also redistributed when a new slave
     is added to the bond and when an inactive slave is re-activated. The
     receive load is distributed sequentially (round robin) among the group
     of highest speed slaves in the bond.

     When a link is reconnected or a new slave joins the bond the receive
     traffic is redistributed among all active slaves in the bond by initiating
     ARP Replies with the selected MAC address to each of the clients. The
     updelay parameter (detailed below) must be set to a value equal or greater
     than the switch's forwarding delay so that the ARP Replies sent to the
     peers will not be blocked by the switch.

   * ``xor-hash`` - XOR policy: Transmit based on the selected transmit
     hash policy.  The default policy is a simple [(source MAC address XOR'd
     with destination MAC address XOR packet type ID) modulo slave count].
     Alternate transmit policies may be selected via the :cfgcmd:`hash-policy`
     option, described below.

     This mode provides load balancing and fault tolerance.

.. cfgcmd:: set interfaces bonding <interface> min-links <0-16>

   Specifies the minimum number of links that must be active before asserting
   carrier. It is similar to the Cisco EtherChannel min-links feature. This
   allows setting the minimum number of member ports that must be up (link-up
   state) before marking the bond device as up (carrier on). This is useful for
   situations where higher level services such as clustering want to ensure a
   minimum number of low bandwidth links are active before switchover.

   This option only affects 802.3ad mode.

   The default value is 0. This will cause carrier to be asserted (for 802.3ad
   mode) whenever there is an active aggregator, regardless of the number of
   available links in that aggregator.

   .. note:: Because an aggregator cannot be active without at least one
      available link, setting this option to 0 or to 1 has the exact same
      effect.

.. cfgcmd:: set interfaces bonding <interface> hash-policy <policy>

   * **layer2** - Uses XOR of hardware MAC addresses and packet type ID field
     to generate the hash. The formula is

     .. code-block:: none

       hash = source MAC XOR destination MAC XOR packet type ID
       slave number = hash modulo slave count

     This algorithm will place all traffic to a particular network peer on
     the same slave.

     This algorithm is 802.3ad compliant.

   * **layer2+3** - This policy uses a combination of layer2 and layer3
     protocol information to generate the hash. Uses XOR of hardware MAC
     addresses and IP addresses to generate the hash. The formula is:

     .. code-block:: none

       hash = source MAC XOR destination MAC XOR packet type ID
       hash = hash XOR source IP XOR destination IP
       hash = hash XOR (hash RSHIFT 16)
       hash = hash XOR (hash RSHIFT 8)

     And then hash is reduced modulo slave count.

     If the protocol is IPv6 then the source and destination addresses are
     first hashed using ipv6_addr_hash.

     This algorithm will place all traffic to a particular network peer on the
     same slave. For non-IP traffic, the formula is the same as for the layer2
     transmit hash policy.

     This policy is intended to provide a more balanced distribution of traffic
     than layer2 alone, especially in environments where a layer3 gateway
     device is required to reach most destinations.

     This algorithm is 802.3ad compliant.

   * **layer3+4** - This policy uses upper layer protocol information, when
     available, to generate the hash. This allows for traffic to a particular
     network peer to span multiple slaves, although a single connection will
     not span multiple slaves.

     The formula for unfragmented TCP and UDP packets is

     .. code-block:: none

       hash = source port, destination port (as in the header)
       hash = hash XOR source IP XOR destination IP
       hash = hash XOR (hash RSHIFT 16)
       hash = hash XOR (hash RSHIFT 8)

     And then hash is reduced modulo slave count.

     If the protocol is IPv6 then the source and destination addresses are
     first hashed using ipv6_addr_hash.

     For fragmented TCP or UDP packets and all other IPv4 and IPv6 protocol
     traffic, the source and destination port information is omitted. For
     non-IP traffic, the formula is the same as for the layer2 transmit hash
     policy.

     This algorithm is not fully 802.3ad compliant. A single TCP or UDP
     conversation containing both fragmented and unfragmented packets will see
     packets striped across two interfaces. This may result in out of order
     delivery. Most traffic types will not meet this criteria, as TCP rarely
     fragments traffic, and most UDP traffic is not involved in extended
     conversations. Other implementations of 802.3ad may or may not tolerate
     this noncompliance.

.. cfgcmd:: set interfaces bonding <interface> primary <interface>

    An `<interface>` specifying which slave is the primary device. The specified
    device will always be the active slave while it is available. Only when the
    primary is off-line will alternate devices be used. This is useful when one
    slave is preferred over another, e.g., when one slave has higher throughput
    than another.

    The primary option is only valid for active-backup, transmit-load-balance,
    and adaptive-load-balance mode.

.. cfgcmd:: set interfaces bonding <interface> arp-monitor interval <time>

   Specifies the ARP link monitoring `<time>` in seconds.

   The ARP monitor works by periodically checking the slave devices to determine
   whether they have sent or received traffic recently (the precise criteria
   depends upon the bonding mode, and the state of the slave). Regular traffic
   is generated via ARP probes issued for the addresses specified by the
   :cfgcmd:`arp-monitor target` option.

   If ARP monitoring is used in an etherchannel compatible mode (modes
   round-robin and xor-hash), the switch should be configured in a mode that
   evenly distributes packets across all links. If the switch is configured to
   distribute the packets in an XOR fashion, all replies from the ARP targets
   will be received on the same link which could cause the other team members
   to fail.

   A value of 0 disables ARP monitoring. The default value is 0.

.. cfgcmd:: set interfaces bonding <interface> arp-monitor target <address>

   Specifies the IP addresses to use as ARP monitoring peers when
   :cfgcmd:`arp-monitor interval` option is > 0. These are the targets of the
   ARP request sent to determine the health of the link to the targets.

   Multiple target IP addresses can be specified. At least one IP address must
   be given for ARP monitoring to function.

   The maximum number of targets that can be specified is 16. The default value
   is no IP addresses.

VLAN
====

.. cmdinclude:: /_include/interface-vlan-8021q.txt
   :var0: bond
   :var1: bond0

*******
Example
*******

The following configuration on VyOS applies to all following 3rd party vendors.
It creates a bond with two links and VLAN 10, 100 on the bonded interfaces with
a per VIF IPv4 address.

.. code-block:: none

  # Create bonding interface bond0 with 802.3ad LACP
  set interfaces bonding bond0 hash-policy 'layer2'
  set interfaces bonding bond0 mode '802.3ad'

  # Add the required vlans and IPv4 addresses on them
  set interfaces bonding bond0 vif 10 address 192.168.0.1/24
  set interfaces bonding bond0 vif 100 address 10.10.10.1/24

  # Add the member interfaces to the bonding interface
  set interfaces bonding bond0 member interface eth1
  set interfaces bonding bond0 member interface eth2

Cisco Catalyst
==============

Assign member interfaces to PortChannel

.. code-block:: none

  interface GigabitEthernet1/0/23
   description VyOS eth1
   channel-group 1 mode active
  !
  interface GigabitEthernet1/0/24
   description VyOS eth2
   channel-group 1 mode active
  !

A new interface becomes present ``Port-channel1``, all configuration like
allowed VLAN interfaces, STP will happen here.

.. code-block:: none

  interface Port-channel1
   description LACP Channel for VyOS
   switchport trunk encapsulation dot1q
   switchport trunk allowed vlan 10,100
   switchport mode trunk
   spanning-tree portfast trunk
  !


Juniper EX Switch
=================

For a headstart you can use the below example on how to build a bond with two
interfaces from VyOS to a Juniper EX Switch system.

.. code-block:: none

  # Create aggregated ethernet device with 802.3ad LACP and port speeds of 10gbit/s
  set interfaces ae0 aggregated-ether-options link-speed 10g
  set interfaces ae0 aggregated-ether-options lacp active

  # Create layer 2 on the aggregated ethernet device with trunking for our vlans
  set interfaces ae0 unit 0 family ethernet-switching port-mode trunk

  # Add the required vlans to the device
  set interfaces ae0 unit 0 family ethernet-switching vlan members 10
  set interfaces ae0 unit 0 family ethernet-switching vlan members 100

  # Add the two interfaces to the aggregated ethernet device, in this setup both
  # ports are on the same switch (switch 0, module 1, port 0 and 1)
  set interfaces xe-0/1/0 ether-options 802.3ad ae0
  set interfaces xe-0/1/1 ether-options 802.3ad ae0

  # But this can also be done with multiple switches in a stack, a virtual
  # chassis on Juniper (switch 0 and switch 1, module 1, port 0 on both switches)
  set interfaces xe-0/1/0 ether-options 802.3ad ae0
  set interfaces xe-1/1/0 ether-options 802.3ad ae0

Aruba/HP
========

For a headstart you can use the below example on how to build a bond,port-channel
with two interfaces from VyOS to a Aruba/HP 2510G switch.

.. code-block:: none

  # Create trunk with 2 member interfaces (interface 1 and 2) and LACP
  trunk 1-2 Trk1 LACP

  # Add the required vlans to the trunk
  vlan 10 tagged Trk1
  vlan 100 tagged Trk1

Arista EOS
==========

When utilizing VyOS in an environment with Arista gear you can use this blue
print as an initial setup to get an LACP bond / port-channel operational between
those two devices.

Lets assume the following topology:

.. figure:: /_static/images/vyos_arista_bond_lacp.png
   :alt: VyOS Arista EOS setup

**R1**

  .. code-block:: none

     interfaces {
         bonding bond10 {
             hash-policy layer3+4
             member {
                 interface eth1
                 interface eth2
             }
             mode 802.3ad
             vif 100 {
                 address 192.0.2.1/30
                 address 2001:db8::1/64
             }
         }

**R2**

  .. code-block:: none

     interfaces {
         bonding bond10 {
             hash-policy layer3+4
             member {
                 interface eth1
                 interface eth2
             }
             mode 802.3ad
             vif 100 {
                 address 192.0.2.2/30
                 address 2001:db8::2/64
             }
         }

**SW1**

  .. code-block:: none

     !
     vlan 100
        name FOO
     !
     interface Port-Channel10
        switchport trunk allowed vlan 100
        switchport mode trunk
        spanning-tree portfast
     !
     interface Port-Channel20
        switchport mode trunk
        no spanning-tree portfast auto
        spanning-tree portfast network
     !
     interface Ethernet1
        channel-group 10 mode active
     !
     interface Ethernet2
        channel-group 10 mode active
     !
     interface Ethernet3
        channel-group 20 mode active
     !
     interface Ethernet4
        channel-group 20 mode active
     !

**SW2**

  .. code-block:: none

     !
     vlan 100
        name FOO
     !
     interface Port-Channel10
        switchport trunk allowed vlan 100
        switchport mode trunk
        spanning-tree portfast
     !
     interface Port-Channel20
        switchport mode trunk
        no spanning-tree portfast auto
        spanning-tree portfast network
     !
     interface Ethernet1
        channel-group 10 mode active
     !
     interface Ethernet2
        channel-group 10 mode active
     !
     interface Ethernet3
        channel-group 20 mode active
     !
     interface Ethernet4
        channel-group 20 mode active
     !

.. note:: When using EVE-NG to lab this environment ensure you are using e1000
   as the desired driver for your VyOS network interfaces. When using the regular
   virtio network driver no LACP PDUs will be sent by VyOS thus the port-channel
   will never become active!

*********
Operation
*********

.. opcmd:: show interfaces bonding

   Show brief interface information.

   .. code-block:: none

     vyos@vyos:~$ show interfaces bonding
     Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
     Interface        IP Address                        S/L  Description
     ---------        ----------                        ---  -----------
     bond0            -                                 u/u  my-sw1 int 23 and 24
     bond0.10         192.168.0.1/24                    u/u  office-net
     bond0.100        10.10.10.1/24                     u/u  management-net


.. opcmd:: show interfaces bonding <interface>

   Show detailed information on given `<interface>`

   .. code-block:: none

     vyos@vyos:~$ show interfaces bonding bond5
     bond5: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
         link/ether 00:50:56:bf:ef:aa brd ff:ff:ff:ff:ff:ff
         inet6 fe80::e862:26ff:fe72:2dac/64 scope link tentative
            valid_lft forever preferred_lft forever

         RX:  bytes  packets  errors  dropped  overrun       mcast
                  0        0       0        0        0           0
         TX:  bytes  packets  errors  dropped  carrier  collisions
                  0        0       0        0        0           0

.. opcmd:: show interfaces bonding <interface> detail

   Show detailed information about the underlaying physical links on given
   bond `<interface>`.

   .. code-block:: none

     vyos@vyos:~$ show interfaces bonding bond5 detail
     Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

     Bonding Mode: IEEE 802.3ad Dynamic link aggregation
     Transmit Hash Policy: layer2 (0)
     MII Status: down
     MII Polling Interval (ms): 100
     Up Delay (ms): 0
     Down Delay (ms): 0

     802.3ad info
     LACP rate: slow
     Min links: 0
     Aggregator selection policy (ad_select): stable

     Slave Interface: eth1
     MII Status: down
     Speed: Unknown
     Duplex: Unknown
     Link Failure Count: 0
     Permanent HW addr: 00:50:56:bf:ef:aa
     Slave queue ID: 0
     Aggregator ID: 1
     Actor Churn State: churned
     Partner Churn State: churned
     Actor Churned Count: 1
     Partner Churned Count: 1

     Slave Interface: eth2
     MII Status: down
     Speed: Unknown
     Duplex: Unknown
     Link Failure Count: 0
     Permanent HW addr: 00:50:56:bf:19:26
     Slave queue ID: 0
     Aggregator ID: 2
     Actor Churn State: churned
     Partner Churn State: churned
     Actor Churned Count: 1
     Partner Churned Count: 1