From 1d1fb3f579dfbe83f0e89a051b2d7ddce9289fac Mon Sep 17 00:00:00 2001 From: LiudmylaNad Date: Tue, 4 Aug 2026 19:16:07 +0200 Subject: docs: Update Conntrack sync page to VyOS 1.5 standards (#2147) * docs: Update Conntrack sync page to VyOS 1.5 standards * Minor corrections * Update conntrack-sync.md * Use "active/backup" terminology consistent with official conntrackd docs Co-authored-by: Daniil Baturin --------- Co-authored-by: Daniil Baturin --- docs/configuration/service/conntrack-sync.md | 431 +++++++++++++++++++-------- 1 file changed, 313 insertions(+), 118 deletions(-) (limited to 'docs') diff --git a/docs/configuration/service/conntrack-sync.md b/docs/configuration/service/conntrack-sync.md index 61fa7034..6c471720 100644 --- a/docs/configuration/service/conntrack-sync.md +++ b/docs/configuration/service/conntrack-sync.md @@ -1,200 +1,357 @@ +--- +myst: + html_meta: + description: | + Conntrack sync preserves active sessions during failover between + active and backup routers in a VyOS high-availability pair by + continuously syncing conntrack entries between them. + keywords: conntrack-sync, connection-tracking, ha, high-availability, vrrp +--- + (conntrack-sync)= -# Conntrack Sync - -One of the important features built on top of the Netfilter framework is -connection tracking. Connection tracking allows the kernel to keep track of all -logical network connections or sessions, and thereby relate all of the packets -which may make up that connection. NAT relies on this information to translate -all related packets in the same way, and iptables can use this information to -act as a stateful firewall. - -The connection state however is completely independent of any upper-level -state, such as TCP's or SCTP's state. Part of the reason for this is that when -merely forwarding packets, i.e. no local delivery, the TCP engine may not -necessarily be invoked at all. Even connectionless-mode transmissions such as -UDP, IPsec (AH/ESP), GRE and other tunneling protocols have, at least, a pseudo -connection state. The heuristic for such protocols is often based upon a preset -timeout value for inactivity, after whose expiration a Netfilter connection is -dropped. - -Each Netfilter connection is uniquely identified by a (layer-3 protocol, source -address, destination address, layer-4 protocol, layer-4 key) tuple. The layer-4 -key depends on the transport protocol; for TCP/UDP it is the port numbers, for -tunnels it can be their tunnel ID, but otherwise is just zero, as if it were -not part of the tuple. To be able to inspect the TCP port in all cases, packets -will be mandatorily defragmented. - -It is possible to use either Multicast or Unicast to sync conntrack traffic. -Most examples below show Multicast, but unicast can be specified by using the -"peer" keyword after the specified interface, as in the following example: - -{cfgcmd}`set service conntrack-sync interface eth0 peer 192.168.0.250` +# Conntrack sync + +Conntrack Sync (Connection Tracking Synchronization) is used to +preserve active sessions during a failover between the active and +backup routers in a high-availability +({abbr}`HA (High Availability)`) pair. + +Each active session on the active router is tracked as a local conntrack +entry, and for some protocols, expect entries are also created for +anticipated follow-up connections (for example, the FTP data channel +or SIP media streams). Conntrack sync continuously copies these +entries to backup routers over one or more dedicated interfaces, so it +holds a current picture of every active session in near real time. +This traffic is carried over IPv4, either as multicast (the default) +or unicast to a configured peer address. + +When a failover occurs, a backup router takes over all active +sessions and maintains them in their current state, without dropping +or resetting them. A takeover is triggered by VRRP (Virtual Router +Redundancy Protocol), which detects when the current active becomes +unavailable and transitions one of the backup routers to active. ## Configuration -```{cfgcmd} set service conntrack-sync accept-protocol +### Failover integration -Accept only certain protocols: You may want to replicate the state of flows -depending on their layer 4 protocol. +```{cfgcmd} set service conntrack-sync failover-mechanism vrrp sync-group \ -Protocols are: tcp, sctp, dccp, udp, icmp and ipv6-icmp. +**Bind the conntrack-sync service to the specified VRRP sync-group.** + +This setting is mandatory. The referenced group must already be +configured under `high-availability vrrp sync-group`. ``` +Example: -```{cfgcmd} set service conntrack-sync event-listen-queue-size \ +```none +set service conntrack-sync failover-mechanism vrrp sync-group syncgrp +``` -The daemon doubles the size of the netlink event socket buffer size if it -detects netlink event message dropping. This clause sets the maximum buffer -size growth that can be reached. +### Sync transport -Queue size for listening to local conntrack events in MB. +```{cfgcmd} set service conntrack-sync interface \ + +**Configure the interface used to exchange conntrack state with the +peer.** + +The interface must have an IPv4 address assigned. + +Repeat the command to configure multiple interfaces. In this case, +interfaces must use the same transport mode (either all multicast or +all unicast). ``` +Example: -```{cfgcmd} set service conntrack-sync expect-sync \ +```none +set service conntrack-sync interface eth1 +``` + +```{cfgcmd} set service conntrack-sync interface \ peer \ -Protocol for which expect entries need to be synchronized. +**Configure the peer's IPv4 address for unicast sync on the specified +interface.** + +Setting `peer` switches this interface from multicast (the default) to +unicast mode. + +Repeat the command for all interfaces configured for conntrack state +exchange, since mixing unicast and multicast interfaces causes commit +to fail. ``` +Example: + +```none +set service conntrack-sync interface eth1 peer 192.0.2.2 +``` + +```{cfgcmd} set service conntrack-sync interface \ port \<1-65535\> + +**Configure the UDP port used on the specified sync interface.** -```{cfgcmd} set service conntrack-sync failover-mechanism vrrp sync-group \ +In multicast mode, it is the UDP port used by the multicast group. In +unicast mode, it is the UDP destination port on the peer. -Failover mechanism to use for conntrack-sync. +The default port is 3780. +``` + +Example: -Only VRRP is supported. Required option. +```none +set service conntrack-sync interface eth1 port 3781 ``` +```{cfgcmd} set service conntrack-sync listen-address \ + +**Configure the local IPv4 address the router listens on for unicast +sync traffic.** -```{cfgcmd} set service conntrack-sync ignore-address \ +The command applies only when conntrack sync operates in unicast mode. -IP addresses or networks for which local conntrack entries will not be synced +The listen address must be assigned to the sync interface. Using an +address from another interface may cause conntrack sync to fail +silently. + +Repeat the command to configure multiple listen addresses. ``` +Example: -```{cfgcmd} set service conntrack-sync interface \ +```none +set service conntrack-sync listen-address 192.0.2.1 +``` + +```{cfgcmd} set service conntrack-sync mcast-group \ + +**Configure the IPv4 multicast group both routers in the HA pair join +to exchange sync traffic.** + +The command applies only when conntrack sync operates in multicast +mode. All routers must be configured with the same group. + +The default is 225.0.0.50. +``` + +Example: -Interface to use for syncing conntrack entries. +```none +set service conntrack-sync mcast-group 225.0.0.60 ``` +### Sync scope + +```{cfgcmd} set service conntrack-sync accept-protocol \ -```{cfgcmd} set service conntrack-sync interface \ port \ +**Configure the protocol whose local conntrack entries are +synchronized with the HA peer.** -Port number used by connection. +Repeat the command to configure multiple protocols. When omitted, +conntrack entries for all tracked protocols are synchronized. ``` +Example: + +```none +set service conntrack-sync accept-protocol tcp +set service conntrack-sync accept-protocol udp +set service conntrack-sync accept-protocol icmp +``` + +```{cfgcmd} set service conntrack-sync ignore-address \
+ +**Exclude local conntrack entries involving the specified IP address +or prefix from being synchronized with the HA peer.** -```{cfgcmd} set service conntrack-sync listen-address \ +Accepts IPv4 and IPv6 addresses or prefixes. -Local IPv4 addresses for service to listen on. +Repeat the command to configure multiple values. ``` +Example: -```{cfgcmd} set service conntrack-sync mcast-group \ +```none +set service conntrack-sync ignore-address 192.0.2.0/24 +set service conntrack-sync ignore-address 2001:db8::/32 +``` + +```{cfgcmd} set service conntrack-sync expect-sync \ + +**Configure the protocol whose expect entries are synchronized with +the HA peer.** -Multicast group to use for syncing conntrack entries. +Repeat the command to configure multiple protocols. Use `all` to +synchronize expect entries for all supported protocols. `all` cannot +be combined with any other value. -Defaults to 225.0.0.50. +When omitted, no expect entries are synchronized. ``` +Example: -```{cfgcmd} set service conntrack-sync interface \ peer \ +```none +set service conntrack-sync expect-sync ftp +set service conntrack-sync expect-sync sip +``` + +### Buffers and timers + +```{cfgcmd} set service conntrack-sync event-listen-queue-size \<0-4294967295\> + +**Configure the maximum size of the conntrack event buffer (in megabytes).** + +The buffer starts at 2 MB and grows up to the specified value if +events arrive faster than they can be processed. + +The default is 8. +``` + +Example: -Peer to send unicast UDP conntrack sync entries to, if not using Multicast -configuration from above. +```none +set service conntrack-sync event-listen-queue-size 16 ``` +```{cfgcmd} set service conntrack-sync sync-queue-size \<0-4294967295\> + +**Configure the size of the queue that holds sync messages sent between peers (in megabytes)** -```{cfgcmd} set service conntrack-sync sync-queue-size \ +The same value is applied to both directions, in either multicast or +unicast mode. -Queue size for syncing conntrack entries in MB. +The default is 1. ``` +Example: -```{cfgcmd} set service conntrack-sync disable-external-cache +```none +set service conntrack-sync sync-queue-size 4 +``` + +```{cfgcmd} set service conntrack-sync purge-timeout \<1-2147483647\> -This disables the external cache and directly injects the flow-states into the -in-kernel Connection Tracking System of the backup firewall. +**Configure the delay, in seconds, before synchronized entries are +purged after a handover.** + +The default is 60. ``` +```{note} +If your setup allows a recovered router to reclaim the active role, +set the VRRP `preempt-delay` to at least this value on the VRRP +`sync-group` bound to the conntrack sync service. This gives the +recovered router time to receive the current conntrack entries before +taking over. +``` -```{cfgcmd} set service conntrack-sync purge-timeout \ +Example: -Timeout (in seconds) for purging synchronized entries on handover events. +```none +set service conntrack-sync purge-timeout 60 +``` -On handover, ``conntrackd -t`` is invoked, which schedules a conntrack table -flush after ```` seconds to purge stale (“zombie”) entries and -reduce clashes when multiple handovers occur in a short period. -The default is 60 seconds. +### Behavior + +```{cfgcmd} set service conntrack-sync disable-external-cache + +**Inject conntrack entries directly to backup routers' kernel connection +tracking table as they arrive, rather than holding them in the +external cache used by default.** ``` -:::{note} -In VRRP stateful firewall deployments, align VRRP timing with this -behavior: because synchronized conntrack state is purged after the purge -timeout, set **VRRP preempt-delay** to ≥ **purge-timeout** so mastership -can be restored before conntrack state is purged. -::: +Example: + +```none +set service conntrack-sync disable-external-cache +``` ```{cfgcmd} set service conntrack-sync disable-syslog -Disable connection logging via Syslog. +**Disable syslog logging of conntrack-sync operational events.** ``` +Example: + +```none +set service conntrack-sync disable-syslog +``` ```{cfgcmd} set service conntrack-sync startup-resync -Order conntrackd to request a complete conntrack table resync against -the other node at startup. +**Request a full copy of conntrack entries from the HA peer when the +conntrack sync service starts.** +``` + +Example: + +```none +set service conntrack-sync startup-resync ``` ## Operation -```{opcmd} show conntrack table \ +### Show -Make sure conntrack is enabled by running and show connection tracking table. +```{opcmd} show conntrack table \ -:::{code-block} none -vyos@vyos:~$ show conntrack table ipv4 -Id Original src Original dst Original packets Original bytes Reply src Reply dst Reply packets Reply bytes Protocol State Timeout Mark Zone ----------- -------------- -------------- ------------------ ---------------- ------------ -------------- --------------- ------------- ---------- ----------- --------- ------ ------ -282920088 10.0.2.2:41724 10.0.2.15:22 79 8741 10.0.2.15:22 10.0.2.2:41724 52 8495 tcp TIME_WAIT 3 0 -102466872 192.168.50.1 192.168.50.4 31953 2684052 192.168.50.4 192.168.50.1 31953 2684052 icmp 29 0 110 -1445684978 10.0.2.2:37762 10.0.2.15:22 111 9969 10.0.2.15:22 10.0.2.2:37762 86 13323 tcp ESTABLISHED 431999 0 -3302843234 10.0.2.2:37758 10.0.2.15:22 2612 3845685 10.0.2.15:22 10.0.2.2:37758 254 17447 tcp TIME_WAIT 11 0 -::: -:::{note} -If the table is empty and you have a warning message, it means -conntrack is not enabled. To enable conntrack, just create a NAT or a firewall -rule. -{cfgcmd}`set firewall global-options state-policy established action accept` -::: +**Show conntrack entries for the specified address family (IPv4 or +IPv6).** ``` +Example output: -```{opcmd} show conntrack table \ vrf \ +```none +vyos@vyos:~$ show conntrack table ipv4 +Id Original src Original dst Original packets Original bytes Reply src Reply dst Reply packets Reply bytes Protocol State Timeout Mark Zone +---------- ---------------- ---------------- ------------------ ---------------- ---------------- ---------------- --------------- ------------- ---------- ----------- --------- ------ ------ +282920088 192.0.2.2:41724 192.0.2.15:22 79 8741 192.0.2.15:22 192.0.2.2:41724 52 8495 tcp TIME_WAIT 3 0 +102466872 198.51.100.1 198.51.100.4 31953 2684052 198.51.100.4 198.51.100.1 31953 2684052 icmp 29 0 110 +1445684978 192.0.2.2:37762 192.0.2.15:22 111 9969 192.0.2.15:22 192.0.2.2:37762 86 13323 tcp ESTABLISHED 431999 0 +3302843234 192.0.2.2:37758 192.0.2.15:22 2612 3845685 192.0.2.15:22 192.0.2.2:37758 254 17447 tcp TIME_WAIT 11 0 +``` -Show connection tracking table filtered by VRF name. Only entries originating -from the specified VRF are displayed. +```{note} +If the table is empty and you see a warning message, conntrack is +not enabled. To enable it, create a NAT or a firewall rule, for +example: `set firewall global-options state-policy established +action accept`. ``` +```{opcmd} show conntrack-sync cache external [main] + +**Show the regular conntrack entries in the external cache.** + +The `main` keyword is optional and produces the same output. +``` -```{opcmd} show conntrack-sync cache external +```{opcmd} show conntrack-sync cache external expect -Show connection syncing external cache entries +**Show the expect entries in the external cache.** ``` +```{opcmd} show conntrack-sync cache internal [main] -```{opcmd} show conntrack-sync cache internal +**Show the regular conntrack entries in the internal cache.** -Show connection syncing internal cache entries +The `main` keyword is optional and produces the same output. ``` +```{opcmd} show conntrack-sync cache internal expect + +**Show the expect entries in the internal cache.** +``` ```{opcmd} show conntrack-sync statistics -Retrieve current statistics of connection tracking subsystem. +**Show operational statistics for the conntrack sync service.** +``` + +Example output: -:::{code-block} none +```none vyos@vyos:~$ show conntrack-sync statistics Main Table Statistics: @@ -220,34 +377,71 @@ multicast traffic (active device=eth0.5): message tracking: 0 Malformed msgs 263 Lost msgs -::: ``` + +```{opcmd} show conntrack statistics + +**Show operational statistics for the kernel connection tracking +subsystem.** +``` + ```{opcmd} show conntrack-sync status -Retrieve current status of connection tracking subsystem. +**Show the current operational status of the conntrack sync +service.** +``` + +Example output: -:::{code-block} none +```none vyos@vyos:~$ show conntrack-sync status sync-interface : eth0.5 failover-mechanism : vrrp [sync-group GEFOEKOM] last state transition : no transition yet! ExpectationSync : disabled -::: +``` + +### Restart/reset + +```{opcmd} restart conntrack-sync + +**Restart the conntrack sync service. On restart, the local cache is +cleared.** +``` + +```{opcmd} reset conntrack-sync external-cache + +**Clear the external cache and request a fresh copy of conntrack +entries from the HA peer.** +``` + +```{opcmd} reset conntrack-sync internal-cache + +**Clear the internal cache and request a fresh copy of conntrack +entries from the HA peer.** ``` ## Example -The next example is a simple configuration of conntrack-sync. +The following shows how to configure a two-node HA pair with +conntrack sync. :::{figure} /_static/images/service_conntrack_sync-schema.webp -:alt: Conntrack Sync Example -:scale: 60 % +:alt: Conntrack sync example +:scale: 80 % +Conntrack sync example ::: -Now configure conntrack-sync service on `router1` **and** `router2` +Apply the following configuration on both `router1` and `router2`. The +only difference between the two nodes is the VRRP priority: use `200` +on `router1` (which becomes VRRP master, i.e., active) and `100` on `router2` (which +becomes backup). ```none -set high-availability vrrp group internal virtual-address ... etc ... +set high-availability vrrp group internal interface 'eth1' +set high-availability vrrp group internal vrid '10' +set high-availability vrrp group internal priority '200' +set high-availability vrrp group internal virtual-address '192.0.2.254/24' set high-availability vrrp sync-group syncgrp member 'internal' set service conntrack-sync accept-protocol 'tcp' set service conntrack-sync accept-protocol 'udp' @@ -257,11 +451,12 @@ set service conntrack-sync interface 'eth0' set service conntrack-sync mcast-group '225.0.0.50' ``` -On the active router, you should have information in the internal-cache of -conntrack-sync. The same current active connections number should be shown in -the external-cache of the standby router +After commit, the active router populates its internal cache with +local conntrack entries and sends them to the backup, which stores +them in its external cache. Running `show conntrack-sync statistics` +on each peer reflects this asymmetry. -On active router run: +On the active router: ```none $ show conntrack-sync statistics @@ -292,7 +487,7 @@ message tracking: 0 Malformed msgs 0 Lost msgs ``` -On standby router run: +On the backup router: ```none $ show conntrack-sync statistics -- cgit v1.2.3