diff options
Diffstat (limited to 'docs/vpp/configuration/dataplane')
24 files changed, 985 insertions, 0 deletions
diff --git a/docs/vpp/configuration/dataplane/buffers.md b/docs/vpp/configuration/dataplane/buffers.md new file mode 100644 index 00000000..c9e38a54 --- /dev/null +++ b/docs/vpp/configuration/dataplane/buffers.md @@ -0,0 +1,102 @@ +--- +lastproofread: '2026-02-23' +--- + +(vpp-config-dataplane-buffers)= + +```{include} /_include/need_improvement.txt +``` + + +# VPP Dataplane Buffers Configuration + +Buffers are essential for handling network packets efficiently. Proper +configuration enhances performance and reliability, and is mandatory for +VPP to work. Buffers temporarily store packets during processing. Therefore, +their configuration must be in sync with NIC configuration, CPU threads, and +overall system resources. + +:::{important} +VPP buffers are allocated from the physical memory pool (`physmem`). The +total amount of memory available for buffer allocation is controlled by the +`physmem-max-size` setting, while the buffer configuration parameters +below control how that memory is used for buffer allocation. + +See {ref}`VPP Physical Memory Configuration <vpp-config-dataplane-physmem>` +for details on configuring `physmem`. +::: + +## Buffer Configuration Parameters + +The following parameters can be configured for VPP buffers: + +### buffers-per-numa + +Number of buffers allocated per NUMA node. This setting optimizes +memory access patterns for multi-CPU systems. + +Typically, you need to tune this value if: +- The system has many interfaces +- NICs have many queues +- NICs have large descriptor sizes + +Set this value carefully to balance memory usage and performance. + +```{cfgcmd} set vpp settings resource-allocation buffers buffers-per-numa \<value\> +``` + +The common approach for the calculation is to use the formula: + +```none +buffers-per-numa = (num-rx-queues * num-rx-desc) + (num-tx-queues * num-tx-desc) +``` + +Calculate this formula for each NIC and sum the results. Multiply the +total by 2.5 to get the minimum recommended value for +`buffers-per-numa`. + +Avoid setting this value too low to prevent packet drops. + +### data-size + +This value sets how much payload data can be stored in a single buffer +allocated by VPP. Larger values reduce buffer chains for large packets, +while smaller values conserve memory for environments handling mostly +small packets. + +```{cfgcmd} set vpp settings resource-allocation buffers data-size \<value\> +``` + +Optimal size depends on the typical packet size in your network. If +unsure, use the largest MTU in your network plus overhead (for example, +128 bytes). + +### page-size + +A memory pages type used for buffer allocation. Common values are 4K, 2M, or 1G. + +Use page sizes configured in your system settings. + +```{cfgcmd} set vpp settings resource-allocation buffers page-size \<value\> +``` + + +## Potential Issues and Troubleshooting + +Improper buffer configuration can lead to issues such as: + +- Increased latency and packet loss +- Inefficient CPU utilization +- Interface initialization failures + +Indicators of such issues are: + +- Errors during interfaces initialization in VPP logs +- Packet drops observed in VPP statistics + +To troubleshoot buffer-related issues, consider the following steps: + +- Review VPP logs for errors related to buffer allocation. Look for + error `-5` messages. +- Tune available buffers by adjusting the `buffers-per-numa` and + `data-size` parameters. diff --git a/docs/vpp/configuration/dataplane/cpu.md b/docs/vpp/configuration/dataplane/cpu.md new file mode 100644 index 00000000..d92f6587 --- /dev/null +++ b/docs/vpp/configuration/dataplane/cpu.md @@ -0,0 +1,71 @@ +--- +lastproofread: '2026-02-23' +--- + +(vpp-config-dataplane-cpu)= + +```{include} /_include/need_improvement.txt +``` + + +# VPP Dataplane CPU Configuration + +VPP can utilize multiple CPU cores for better packet processing +performance. Proper CPU configuration is essential for optimal +throughput and low latency. + +VPP CPU assignment is handled automatically. You specify how many CPU +cores VPP may use, and the system distributes them between the main +thread and worker threads. + +:::{important} +Review the system configuration settings page before changing CPU +settings: {doc}`system`. +::: + +If you don't configure CPU settings, VPP uses a single core for the +main thread and doesn't create worker threads. + +## CPU Configuration Parameters + +### `cpu-cores` + +This parameter defines the total number of CPU cores allocated to VPP. + +```{cfgcmd} set vpp settings resource-allocation cpu-cores \<core-number\> +``` + +The system automatically assigns cores using the following rules: + +> - The first two CPU cores are always reserved for the operating system and +> other services. +> - The main VPP thread is assigned to the first available core after the +> reserved ones. +> - The remaining allocated cores are used for worker threads. + +For example: + +> - If cpu-cores is set to 1, VPP runs only a main thread. +> +> - If cpu-cores is set to 4, VPP uses: +> +> > - 1 core for the main thread +> > - 3 cores for worker threads + +Choose a value based on available hardware resources and expected +traffic load. Too few cores may limit performance, while too many can +negatively impact other system services. + +## Potential Issues and Troubleshooting + +Improper CPU configuration can lead to issues such as: + +- VPP underperformance when not enough cores are assigned, or kernel + underperformance when too many cores are assigned to VPP. +- Resource conflicts with other processes and services. + +Indicators of such issues are: + +- VPP or kernel forwarding performance is lower than expected +- Degraded performance of system components or services, such as DNS, + DHCP, and dynamic routing diff --git a/docs/vpp/configuration/dataplane/index.md b/docs/vpp/configuration/dataplane/index.md new file mode 100644 index 00000000..c9ad7746 --- /dev/null +++ b/docs/vpp/configuration/dataplane/index.md @@ -0,0 +1,36 @@ +--- +lastproofread: '2026-02-23' +--- + +(vpp-config-dataplane-index)= + +```{include} /_include/need_improvement.txt +``` + +# VPP Dataplane Core Configuration + +This section covers the core configuration options for the VPP dataplane in +VyOS. It includes settings for memory management, CPU allocation, hugepages, +and other essential parameters that influence the performance and behavior +of the VPP dataplane. + +Please review the general system configuration, before starting to configure +VPP. Without proper VyOS preconditions, VPP will not start or its efficiency +will be significantly degraded. + +```{toctree} +:includehidden: true +:maxdepth: 1 + +system +buffers +cpu +interface +ipsec +ipv6 +l2learn +lcp +logging +memory +unix +``` diff --git a/docs/vpp/configuration/dataplane/interface.md b/docs/vpp/configuration/dataplane/interface.md new file mode 100644 index 00000000..02a65ff8 --- /dev/null +++ b/docs/vpp/configuration/dataplane/interface.md @@ -0,0 +1,104 @@ +--- +lastproofread: '2026-02-23' +--- + +(vpp-config-dataplane-interface)= + +```{include} /_include/need_improvement.txt +``` + + +# VPP Dataplane Interfaces Configuration + +Only Ethernet interfaces (physical or virtual) can be connected to the +VPP dataplane. Interfaces configured here act as a bridge between VPP +and the outside world, allowing VPP to send and receive network +packets. + +## Interface Configuration Parameters + +Interfaces connected to the VPP dataplane use the DPDK driver by default, +providing high performance and low latency. + +```{cfgcmd} set vpp settings interface \<interface-name\> +``` + +Some network interface cards (NICs) may not be compatible with the DPDK driver. + +### DPDK interface options + +This section shows how to configure DPDK-specific settings for an interface. + +```{cfgcmd} set vpp settings interface \<interface-name\> num-rx-queues \<value\> +``` + +Specifies the number of receive queues for the interface. More queues +improve performance on multi-core systems by allowing parallel +processing of incoming packets. Each queue is assigned to a separate +CPU core. + +```{cfgcmd} set vpp settings interface \<interface-name\> num-tx-queues \<value\> +``` + +Specifies the number of transmit queues for the interface. Similar to +receive queues, more transmit queues improve performance by enabling +parallel processing of outgoing packets. By default, the VPP Dataplane +has one TX queue per enabled CPU worker, or a single queue if no +workers are configured. + +:::{seealso} +{doc}`cpu` +::: +```{cfgcmd} set vpp settings interface \<interface-name\> num-rx-desc \<value\> +``` + +Defines the size of each receive queue. Larger queue sizes accommodate +bursts of incoming traffic and reduce the likelihood of packet drops +during high traffic periods. + +```{cfgcmd} set vpp settings interface \<interface-name\> num-tx-desc \<value\> +``` + +Defines the size of each transmit queue. Larger sizes help manage +bursts of outgoing traffic more effectively. + +## Global Interface Parameters + +(vpp-config-dataplane-interface-rx-mode)= + +### interface-rx-mode + +The `interface-rx-mode` parameter defines how VPP handles incoming +packets on interfaces. There are several modes available, each with its +own advantages and use cases: +- `interrupt`: In this mode, VPP relies on hardware interrupts to + notify it of incoming packets. This mode suits low to moderate + traffic loads and reduces CPU usage during idle periods. It is not + recommended for low-latency processing. Some NICs may not support + this mode. +- `polling`: In polling mode, VPP continuously checks the interface + for incoming packets. This mode is ideal for high-throughput + scenarios where low latency is critical, as it minimizes packet + waiting time. However, it can increase CPU usage, especially during + low traffic periods, as the polling process is always active. +- `adaptive`: Adaptive mode combines the benefits of interrupt and + polling modes. VPP starts in interrupt mode and switches to polling + mode when traffic load increases. + +```{cfgcmd} set vpp settings interface-rx-mode \<mode\> +``` + +Choose an rx-mode based on expected traffic patterns and performance +requirements of your network. + +## Potential Issues and Troubleshooting + +Improper interface configuration can lead to issues such as: + +- Failure to initialize the interface +- Poor performance due to suboptimal driver selection or settings + +Indicators of such issues are: + +- Failed commits after adding or modifying an interface settings +- Low throughput or high latency on the interface diff --git a/docs/vpp/configuration/dataplane/ipsec.md b/docs/vpp/configuration/dataplane/ipsec.md new file mode 100644 index 00000000..0a66221f --- /dev/null +++ b/docs/vpp/configuration/dataplane/ipsec.md @@ -0,0 +1,74 @@ +--- +lastproofread: '2026-02-23' +--- + +(vpp-config-dataplane-ipsec)= + +```{include} /_include/need_improvement.txt +``` + + +# VPP IPsec Configuration + +VPP supports IPsec (Internet Protocol Security) offloading from the +kernel, which speeds up cryptographic operations by leveraging VPP's +high-performance packet processing capabilities. + +IPsec does not require any specific configuration on VPP side. If both +sources and destinations of the IPsec traffic are reachable via VPP +interfaces, VPP will automatically offload the IPsec processing from +the kernel. IPsec tunnels are configured in the VPN configuration +section, see {ref}`ipsec_general`. + +## IPsec Configuration Parameters + +### enable IPsec acceleration + +When VPP is used for offloading IPsec, it creates a virtual interface to +connect to peers. The interface type is always 'ipsec', which is used for +IPsec tunnels. + +```{cfgcmd} set vpp settings ipsec-acceleration +``` + +Enabling this option allows VPP to handle IPsec traffic more efficiently by +offloading processing from the kernel. + +### netlink + +VPP uses netlink to receive IPsec event messages from the kernel. Proper +settings of the following parameters are crucial for ensuring that VPP can +process all such messages: + +```{cfgcmd} set vpp settings lcp netlink batch-delay-ms \<milliseconds\> +``` + +This parameter specifies the delay in milliseconds between processing +batch netlink messages. + +```{cfgcmd} set vpp settings lcp netlink batch-size \<number\> +``` + +This parameter specifies the maximum number of netlink messages to +process in a single batch. + +```{cfgcmd} set vpp settings lcp netlink rx-buffer-size \<number\> +``` + +This parameter specifies the size of the receive buffer for netlink +socket. If you expect to offload many IPsec tunnels or get frequent and +intensive rekeying, you may need to increase this value. + +:::{note} +IPsec uses the same netlink parameters as LCP, so tuning them +affects both LCP and IPsec processing. +::: + +## Potential Issues and Troubleshooting + +Improper IPsec configuration can lead to various issues, including: + +- Failure to offload IPsec tunnels to VPP +- Lost IPsec event messages due to insufficient netlink buffer size or + batch settings +- IPsec states or SAs are not synchronized between kernel and VPP diff --git a/docs/vpp/configuration/dataplane/ipv6.md b/docs/vpp/configuration/dataplane/ipv6.md new file mode 100644 index 00000000..5f2ba3c5 --- /dev/null +++ b/docs/vpp/configuration/dataplane/ipv6.md @@ -0,0 +1,46 @@ +--- +lastproofread: '2026-02-26' +--- + +(vpp-config-dataplane-ipv6)= + +```{include} /_include/need_improvement.txt +``` + + +# VPP IPv6 Configuration + +VPP lets you configure resources allocated for IPv6 traffic processing +independently from IPv4. This helps ensure that in networks without IPv6 +traffic, resources are not wasted. If IPv6 traffic is present, especially +with large routing tables, you must allocate additional resources for IPv6 +processing to keep the dataplane stable. + +You can configure two main resources for IPv6 traffic processing: + +```{cfgcmd} set vpp settings resource-allocation ipv6 hash-buckets \<value\> +``` + +This parameter configures the number of hash buckets used for IPv6 +routing. If you have a large IPv6 routing table, you may need to increase +this value to ensure efficient routing table performance and fast lookups. + +```{cfgcmd} set vpp settings resource-allocation ipv6 heap-size \<value\> +``` + +This parameter configures the heap size used for IPv6 forwarding. If you +have a large IPv6 routing table, you may need to increase this value to +ensure the routing table can accommodate all routes. + +## Potential Issues and Troubleshooting + +Improper IPv6 configuration can lead to various issues, including: + +- Inefficient, slow routing table lookups and traffic processing due to + insufficient hash buckets +- Dataplane crashes or instability due to insufficient heap size when + handling a large number of IPv6 routes +- Overall dataplane instability when handling IPv6 traffic + +Consider increasing configuration values if you experience issues with +IPv6 traffic processing or if you have a large IPv6 routing table. diff --git a/docs/vpp/configuration/dataplane/l2learn.md b/docs/vpp/configuration/dataplane/l2learn.md new file mode 100644 index 00000000..2ce572a1 --- /dev/null +++ b/docs/vpp/configuration/dataplane/l2learn.md @@ -0,0 +1,35 @@ +--- +lastproofread: '2026-02-26' +--- + +(vpp-config-dataplane-l2learn)= + +```{include} /_include/need_improvement.txt +``` + +# VPP L2LEARN Configuration + +When VPP dataplane connects to an L2 domain, it learns MAC addresses of +devices in the domain. By default, the number of MAC addresses it can +learn is limited. + +You can configure the limit using the following command: + +```{cfgcmd} set vpp settings resource-allocation mac-limit \<value\> +``` + +This parameter sets the maximum number of MAC addresses that can be +learned in the L2 domain. If you have many devices, you may need to +increase this limit to ensure VPP learns all MAC addresses. + +## Potential Issues and Troubleshooting + +Improper L2LEARN configuration can lead to various issues, including: + +- MAC address learning failure in the L2 domain if the limit is set too + low +- Increased packet loss or latency for devices that aren't learned +- Overall dataplane instability when handling L2 traffic + +Consider increasing the L2LEARN limit if you experience issues with MAC +address learning or if you have many devices in the L2 domain. diff --git a/docs/vpp/configuration/dataplane/lcp.md b/docs/vpp/configuration/dataplane/lcp.md new file mode 100644 index 00000000..a68247e1 --- /dev/null +++ b/docs/vpp/configuration/dataplane/lcp.md @@ -0,0 +1,46 @@ +--- +lastproofread: '2026-02-26' +--- + +```{include} /_include/need_improvement.txt +``` + +# VPP LCP Configuration + +Linux Control Plane (LCP) is a core component of VPP that lets you +offload various control plane functions to the Linux kernel. LCP provides +seamless integration with other VyOS components, letting you use system +components like DHCP clients and routing daemons together with the VPP +dataplane. + +VPP integration in VyOS relies heavily on LCP. Almost all control plane +functions are handled by other daemons and services, while VPP handles +high-performance packet forwarding exclusively. This approach also reduces +VPP management processing load, improving overall dataplane performance and +stability. + +VyOS integrates the kernel and VPP routing tables uniquely. By default, +all routes, even those not directly connected to VPP interfaces, are +imported from the kernel routing table to the VPP routing table, pointing +to the kernel. This lets you forward traffic to any destination known to +the kernel, even if VPP doesn't have a route to that destination. + +However, in some scenarios this behavior may not be desired. For example, +if you have many routes in the kernel routing table not directly connected +to VPP interfaces, and you don't need forwarding between those +destinations and destinations reachable via VPP, you can disable this +behavior using the following command: +(vpp_config_dataplane_lcp_ignore_kernel_routes)= + +```{cfgcmd} set vpp settings ignore-kernel-routes +``` + +Pay attention that disabling this option leads to loss of connectivity to +destinations if there are no direct routes in VPP routing table. + +## Potential Issues and Troubleshooting + +Disabling kernel route import can result in: + +- Loss of connectivity to certain destinations if kernel routes are ignored +- Incomplete route synchronization between the kernel and VPP diff --git a/docs/vpp/configuration/dataplane/logging.md b/docs/vpp/configuration/dataplane/logging.md new file mode 100644 index 00000000..50e6277b --- /dev/null +++ b/docs/vpp/configuration/dataplane/logging.md @@ -0,0 +1,59 @@ +--- +lastproofread: '2026-02-27' +--- + +(vpp-config-dataplane-logging)= + +```{include} /_include/need_improvement.txt +``` + + +# VPP Logging Configuration + +VPP logging is an important part of monitoring and troubleshooting +the performance and behavior of the VPP dataplane. + +VPP stores logs in two places: +- `/var/log/vpp.log` — This file contains logs related to daemon + startup and logs of commands executed directly via VPP CLI. Pay + attention: VyOS does not use VPP CLI for configuration, so this log + will not contain any configuration changes made via VyOS CLI and will + not be informative in most cases. +- System journal — contains logs related to the VPP daemon work, + including errors, warnings, and informational messages. It is the + main destination of logs generated by VPP. + +Logging detail level can be configured via the next command: + +```{cfgcmd} set vpp settings logging default-level \<level\> +``` + +Where `<level>` can be one of the following: + +- `emerg` (Emergency) - System is unusable. +- `alert` (Alert) - Immediate action required. +- `crit` (Critical) - Critical conditions. +- `error` (Error) - Error conditions. +- `warn` (Warning) - Warning conditions. +- `notice` (Notice) - Normal but significant. +- `info` (Informational) - Routine informational messages. +- `debug` (Debug) - Detailed debugging messages. +- `disabled` (Disabled) - Logging disabled. + +It is recommended to set logging level to `debug` only for +troubleshooting purposes, as it can generate a large volume of log +data. For regular operation, a level of `info` or `warn` is usually +sufficient. + +## Troubleshooting + +Improper logging configuration can lead to various issues, including: + +- Excessive log file sizes if the logging level is set too high + (for example, `debug`). +- Missing critical information if the logging level is set too low + (for example, `alert`). +- Performance degradation due to excessive logging overhead + +Consider adjusting the logging level if you experience issues mentioned +above. diff --git a/docs/vpp/configuration/dataplane/memory.md b/docs/vpp/configuration/dataplane/memory.md new file mode 100644 index 00000000..2465e3b3 --- /dev/null +++ b/docs/vpp/configuration/dataplane/memory.md @@ -0,0 +1,142 @@ +--- +lastproofread: '2026-02-27' +--- + +(vpp_config_dataplane_memory)= + +```{include} /_include/need_improvement.txt +``` + +# VPP Memory Configuration + +VPP heavily relies on hugepages for its memory management. Hugepages +are larger memory pages that reduce the overhead of page management and +improve performance for applications that require large amounts of +memory, such as VPP. + +VPP supports both 2MB and 1GB hugepages, but the default and most +commonly used size is 2MB. The choice of hugepage size can impact +performance, with larger pages generally providing better performance +for memory-intensive applications. + +Before configuring memory in VPP dataplane settings, you need to +ensure that hugepages are enabled and properly configured on your +system. + +:::{seealso} +{ref}`Hugepages in VyOS Configuration for VPP <vpp-config-hugepages>` +::: + +To configure memory settings for VPP, you can use the following +commands in the VPP CLI: + +VPP uses a main heap as a central memory pool for FIB data structures +entry allocations. + +Efficient memory management is crucial for VPP's performance, and the +main heap plays a significant role in this. + +It can be configured using the following command: + +```{cfgcmd} set vpp settings resource-allocation memory main-heap-page-size \<size\> +``` + +Sets the main heap page size for VPP. + +```{cfgcmd} set vpp settings resource-allocation memory main-heap-size \<size\> +``` + +Sets the main heap size for VPP. +(vpp-config-dataplane-physmem)= + +## Physical Memory Configuration + +VPP uses physical memory for packet buffers and interface operations. +The `physmem` setting controls how much memory VPP can allocate for +these operations. + +```{cfgcmd} set vpp settings resource-allocation memory physmem-max-size \<size\> +``` + +Sets the maximum amount of physical memory VPP can use for packet +processing and interface buffers. + +**Default**: 16GB (usually sufficient for most deployments) + +You may need to modify the value for high-throughput environments with +many interfaces, large packet buffers, very high packet rates, or +memory-constrained systems where you need to limit VPP's memory usage. + +**Physmem independent of main heap size** — physmem is for packet +buffers, main heap is for routing tables. + +:::{seealso} +- {ref}`Hugepages in VyOS Configuration for VPP <vpp-config-hugepages>` +- {ref}`VPP Buffer Configuration <vpp-config-dataplane-buffers>` - for + controlling buffer allocation within physmem +::: + +### Common configurations + +```none +# Reduce for memory-constrained systems +set vpp settings physmem max-size 4G + +# Increase for high-throughput environments +set vpp settings physmem max-size 32G +``` + +## Stats Memory Configuration + +VPP uses a dedicated statistics memory segment to store runtime +counters and telemetry data. This segment is used by the VPP CLI and +monitoring tools to access performance and status information. + +The statistics segment is allocated from hugepage memory and can be +configured independently from the main heap and physmem settings. + +You can configure statistics memory using the following commands: + +```{cfgcmd} set vpp settings resource-allocation memory stats page-size \<size\> +``` + +Sets the hugepage page size used for the statistics memory segment. + +```{cfgcmd} set vpp settings resource-allocation memory stats size \<size\> +``` + +Sets the total size of the statistics memory segment. + +Increasing this value may be required in large deployments with many +interfaces or enabled features that generate a high number of counters. + +Statistics memory is used only for telemetry and monitoring. It does +not affect packet buffer allocation or routing table memory. + +## Troubleshooting + +Improper configuration of main heap size can lead to performance +degradation or even system instability. If VPP runs out of memory in the +main heap, it may crash or exhibit erratic behavior. Symptoms you may +observe include: + +- Increased latency or packet loss +- Crashes or restarts of VPP processes, especially during routing table + population (for example, BGP session establishment) +- Error messages related to memory allocation failures + +You need to tune the main heap size based on expected FIB entries. Pay +attention: the same amount of routes with a single next-hop and with +multiple next-hops will consume different amounts of memory. + +For physmem, insufficient allocation can lead to packet drops, interface +initialization failures, and overall degraded performance. Symptoms +include: + +- Packet drops or failures to allocate buffers +- Increased latency or jitter in packet processing +- Crashes or restarts of VPP processes under heavy load + +You need to tune the physmem settings based on expected traffic patterns +and interface usage. Monitor memory usage closely and adjust the +configuration as needed to ensure optimal performance. diff --git a/docs/vpp/configuration/dataplane/buffers.rst b/docs/vpp/configuration/dataplane/rst-buffers.rst index b08eb691..b08eb691 100644 --- a/docs/vpp/configuration/dataplane/buffers.rst +++ b/docs/vpp/configuration/dataplane/rst-buffers.rst diff --git a/docs/vpp/configuration/dataplane/cpu.rst b/docs/vpp/configuration/dataplane/rst-cpu.rst index 22158ce3..22158ce3 100644 --- a/docs/vpp/configuration/dataplane/cpu.rst +++ b/docs/vpp/configuration/dataplane/rst-cpu.rst diff --git a/docs/vpp/configuration/dataplane/index.rst b/docs/vpp/configuration/dataplane/rst-index.rst index 323d7588..323d7588 100644 --- a/docs/vpp/configuration/dataplane/index.rst +++ b/docs/vpp/configuration/dataplane/rst-index.rst diff --git a/docs/vpp/configuration/dataplane/interface.rst b/docs/vpp/configuration/dataplane/rst-interface.rst index e4556021..e4556021 100644 --- a/docs/vpp/configuration/dataplane/interface.rst +++ b/docs/vpp/configuration/dataplane/rst-interface.rst diff --git a/docs/vpp/configuration/dataplane/ipsec.rst b/docs/vpp/configuration/dataplane/rst-ipsec.rst index 31734c86..31734c86 100644 --- a/docs/vpp/configuration/dataplane/ipsec.rst +++ b/docs/vpp/configuration/dataplane/rst-ipsec.rst diff --git a/docs/vpp/configuration/dataplane/ipv6.rst b/docs/vpp/configuration/dataplane/rst-ipv6.rst index 94f38476..94f38476 100644 --- a/docs/vpp/configuration/dataplane/ipv6.rst +++ b/docs/vpp/configuration/dataplane/rst-ipv6.rst diff --git a/docs/vpp/configuration/dataplane/l2learn.rst b/docs/vpp/configuration/dataplane/rst-l2learn.rst index b204c41f..b204c41f 100644 --- a/docs/vpp/configuration/dataplane/l2learn.rst +++ b/docs/vpp/configuration/dataplane/rst-l2learn.rst diff --git a/docs/vpp/configuration/dataplane/lcp.rst b/docs/vpp/configuration/dataplane/rst-lcp.rst index cb39439a..cb39439a 100644 --- a/docs/vpp/configuration/dataplane/lcp.rst +++ b/docs/vpp/configuration/dataplane/rst-lcp.rst diff --git a/docs/vpp/configuration/dataplane/logging.rst b/docs/vpp/configuration/dataplane/rst-logging.rst index 70d57028..70d57028 100644 --- a/docs/vpp/configuration/dataplane/logging.rst +++ b/docs/vpp/configuration/dataplane/rst-logging.rst diff --git a/docs/vpp/configuration/dataplane/memory.rst b/docs/vpp/configuration/dataplane/rst-memory.rst index c2d74991..c2d74991 100644 --- a/docs/vpp/configuration/dataplane/memory.rst +++ b/docs/vpp/configuration/dataplane/rst-memory.rst diff --git a/docs/vpp/configuration/dataplane/system.rst b/docs/vpp/configuration/dataplane/rst-system.rst index 608035a1..6a2756fd 100644 --- a/docs/vpp/configuration/dataplane/system.rst +++ b/docs/vpp/configuration/dataplane/rst-system.rst @@ -9,6 +9,7 @@ VyOS Configuration for VPP ########################## .. _vpp_config_hugepages: +.. _vpp-config-hugepages: Hugepages ========= diff --git a/docs/vpp/configuration/dataplane/unix.rst b/docs/vpp/configuration/dataplane/rst-unix.rst index f31d6381..f31d6381 100644 --- a/docs/vpp/configuration/dataplane/unix.rst +++ b/docs/vpp/configuration/dataplane/rst-unix.rst diff --git a/docs/vpp/configuration/dataplane/system.md b/docs/vpp/configuration/dataplane/system.md new file mode 100644 index 00000000..51ee8f54 --- /dev/null +++ b/docs/vpp/configuration/dataplane/system.md @@ -0,0 +1,212 @@ +--- +lastproofread: '2026-02-27' +--- + +(vpp_config_system)= + +```{include} /_include/need_improvement.txt +``` + +# VyOS Configuration for VPP + +(vpp-config-hugepages)= + +## Hugepages + +VPP uses hugepages for efficient memory management. Hugepages are larger +memory pages that reduce the overhead of page management and improve +performance for applications that require large amounts of memory. + +Hugepages can be configured in VyOS using the following commands: + +:::{warning} +Changes to hugepage settings require a system reboot to take effect. + +Hugepages must be enabled before VPP configuration is applied. +::: + +To enable hugepages: + +```{cfgcmd} set system option kernel memory hugepage-size \<size\> hugepage-count '\<count\>' +``` + +Enables hugepages with the specified size and count. The size can be either +2MB or 1GB, and the count specifies the number of hugepages to allocate. + +If your system has multiple NUMA nodes, the total amount of hugepages will be +divided equally among them. + +## Resources Limits + +:::{note} +By default, system will calculate and set the recommended values for +resource limits. Avoid tuning these values if you are not sure what you +are doing. +::: + +During operations VPP utilizes a significant amount of system resources, +especially memory. There are two main settings that may need to be +adjusted to ensure VPP runs smoothly: + +Maximum number of memory map areas a process may have: + +```{cfgcmd} set system option resource-limits max-map-count \<value\> +``` + +Maximum shared memory segment size: + +```{cfgcmd} set system option resource-limits shmmax \<value\> +``` + +Both settings are automatically calculated based on configured hugepages. + +## Kernel Tuning + +VPP performance greatly benefits from proper kernel tuning, especially +CPU isolation and disabling unnecessary kernel features. These +optimizations ensure dedicated CPU cores are available exclusively for +VPP dataplane processing without interference from the kernel scheduler +or other system processes. + +:::{warning} +Kernel tuning changes require a system reboot to take effect. + +Improper CPU isolation can lead to system instability if essential +system processes are starved of CPU resources. +::: + +### CPU Isolation and Optimization + +CPU isolation is crucial for VPP performance as it dedicates specific +CPU cores exclusively to VPP dataplane processing. The isolated cores are +removed from the kernel scheduler and will not run regular system +processes. + +**Disable NMI Watchdog** + +The NMI (Non-Maskable Interrupt) watchdog can interfere with VPP +performance by generating interrupts on isolated cores and is not +compatible with nohz-full mode: + +```{cfgcmd} set system option kernel cpu disable-nmi-watchdog + +Disables the NMI watchdog for detecting hard CPU lockups. This +prevents unnecessary interrupts on VPP worker cores. +``` + +**CPU Core Isolation** + +```{cfgcmd} set system option kernel cpu isolate-cpus \<cpu-range\> + +Isolates specified CPUs from the kernel scheduler. Isolated cores will +not run regular system processes and are dedicated to applications like +VPP. + +The ``<cpu-range>`` can be: +* Single core: ``2`` +* Range: ``2-5`` +* Mixed: ``1,3-5,7`` + +:::{important} +Always reserve at least 2 cores for the operating system to ensure +system stability. For example, on a 4-core system, isolate cores +2-3 for VPP and leave cores 0-1 for the OS. + +Assign the first isolated core as the VPP main core and the +remaining isolated cores as VPP worker cores. Ensure that VPP CPU +assignments match the isolated CPU range. +::: +``` + +**Adaptive-Tick Mode** + +```{cfgcmd} set system option kernel cpu nohz-full \<cpu-range\> + +Enables adaptive-tick mode (NO_HZ_FULL) for specified CPUs. This +causes the kernel to avoid sending scheduling-clock interrupts to CPUs +that have only one runnable task, significantly reducing interrupt +overhead for dedicated workloads like VPP. + +Use the same CPU range as configured for ``isolate-cpus``. +``` + +**RCU Callback Offloading** + +```{cfgcmd} set system option kernel cpu rcu-no-cbs \<cpu-range\> + +Offloads Read-Copy-Update (RCU) callback processing from specified +CPUs. This ensures that RCU callbacks do not prevent the specified CPUs +from entering dyntick-idle or adaptive-tick mode, which is essential +for nohz-full functionality. + +Use the same CPU range as configured for ``isolate-cpus``. +``` + +### System Optimization + +Additional kernel optimizations can further improve VPP performance by +disabling unnecessary features and reducing system overhead. + +**Disable High Precision Event Timer** + +```{cfgcmd} set system option kernel disable-hpet + +Disables the High Precision Event Timer (HPET). HPET can cause +additional interrupts and overhead that may impact VPP performance. +``` + +**Disable Machine Check Exceptions** + +```{cfgcmd} set system option kernel disable-mce + +Disables Machine Check Exception (MCE) reporting and handling. While +MCE provides hardware error detection, it can introduce latency in +high-performance scenarios. +``` + +**Disable CPU Power Saving** + +```{cfgcmd} set system option kernel disable-power-saving + +Disables CPU power saving mechanisms (C-states). This keeps CPU cores +at maximum performance levels, eliminating latency from power state +transitions. +``` + +**Disable Soft Lockup Detection** + +```{cfgcmd} set system option kernel disable-softlockup + +Disables the soft lockup detector for kernel threads. This prevents +false positives when VPP worker threads are busy processing packets. +``` + +**Disable CPU Mitigations** + +```{cfgcmd} set system option kernel disable-mitigations + +Disables all optional CPU mitigations for security vulnerabilities +(for example, Spectre, Meltdown). This may improve performance on some +platforms. +``` + +### Optimal Configuration Example + +For a system with 4 CPU cores (0-3) where cores 2-3 are dedicated to VPP: + +```none +# Kernel CPU optimizations +set system option kernel cpu disable-nmi-watchdog +set system option kernel cpu isolate-cpus '2-3' +set system option kernel cpu nohz-full '2-3' +set system option kernel cpu rcu-no-cbs '2-3' + +# System optimizations +set system option kernel disable-hpet +set system option kernel disable-mce +set system option kernel disable-power-saving +set system option kernel disable-softlockup + +# VPP CPU assignment +set vpp settings resource-allocation cpu-cores '2' +``` diff --git a/docs/vpp/configuration/dataplane/unix.md b/docs/vpp/configuration/dataplane/unix.md new file mode 100644 index 00000000..a1f6a1fd --- /dev/null +++ b/docs/vpp/configuration/dataplane/unix.md @@ -0,0 +1,57 @@ +--- +lastproofread: '2026-02-27' +--- + +(vpp-config-dataplane-unix)= + +```{include} /_include/need_improvement.txt +``` + + +# VPP Unix Dataplane Configuration + +The UNIX configuration section is used to control VPP's interaction +with the underlying operating system, including operations scheduling. + +VPP relies on the polling mechanism to efficiently manage I/O operations +and system events. By default VPP continuously polls for events, which +leads to permanent 100% CPU usage by all cores assigned to VPP dataplane. +This is optimal for performance, but may not be desirable in all +environments, especially where power consumption is a concern or where VPP +is running inside a hypervisor, especially if the VM has burstable +thresholds and CPU usage limits. + +To mitigate this, VPP provides a configurable polling delay that allows +reducing CPU usage by introducing a delay between polling cycles. This +introduces a trade-off between CPU usage and latency, as longer delays +can lead to increased latency in processing events. + +You can configure the polling delay using the following command in the +VyOS CLI: + +```{cfgcmd} set vpp settings poll-sleep-usec \<delay\> +``` + +Sets the polling delay in microseconds. A value of 0 means no delay +(default), while higher values introduce a delay between polling cycles. + +## Troubleshooting + +Setting the polling delay too high can lead to increased latency and +reduced performance, as VPP may not respond to events as quickly. +Conversely, setting it too low may result in high CPU usage, which can be +problematic in resource-constrained environments. + +Symptoms of improper configuration may include: + +- Increased latency in packet processing +- Higher CPU usage than expected +- Packets lost due to buffer overruns + +If you do not need to reduce CPU usage, it is recommended to leave the +polling delay at its default value of 0 for optimal performance. + +If you need to reduce CPU usage, you may also consider using `interrupt` or +`adaptive` {ref}`DPDK driver modes <vpp-config-dataplane-interface-rx-mode>`, +which can provide a balance between performance and resource utilization +without affecting polling behavior. |