openstack-helm/doc/source/install/developer/deploy-ovs-dpdk.rst

Deploy OVS-DPDK

Requirements

A correct DPDK configuration depends heavily on the specific hardware resources and its configuration. Before deploying Openvswitch with DPDK, check the amount and type of available hugepages on the host OS.

cat /proc/meminfo | grep Huge
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:       8
HugePages_Free:        6
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

In this example, 8 hugepages of 1G size have been allocated. 2 of those are being used and 6 are still available.

More information on how to allocate and configure hugepages on the host OS can be found in the Openvswitch documentation.

In order to allow OVS inside a pod to make use of hugepages, the corresponding type and amount of hugepages must be specified in the resource section of the OVS chart's values.yaml:

resources:
  enabled: true
  ovs:
    db:
      requests:
        memory: "128Mi"
        cpu: "100m"
      limits:
        memory: "1024Mi"
        cpu: "2000m"
    vswitchd:
      requests:
        memory: "128Mi"
        cpu: "100m"
      limits:
        memory: "1024Mi"
        cpu: "2000m"
        # set resources to enabled and specify one of the following when using dpdk
        hugepages-1Gi: "1Gi"
        # hugepages-2Mi: "512Mi"

Additionally, the default configuration of the neutron chart must be adapted according to the underlying hardware. The corresponding configuration parameter is labeled with "CHANGE-ME" in the script "values_overrides/dpdk.yaml". Specifically, the "ovs_dpdk" configuration section should list all NICs which should be bound to DPDK with their corresponding PCI-IDs. Moreover, the name of each NIC needs to be unique, e.g., dpdk0, dpdk1, etc.

network:
  interface:
    tunnel: br-phy
conf:
  ovs_dpdk:
    enabled: true
    driver: uio_pci_generic
    nics:
      - name: dpdk0
        # CHANGE-ME: modify pci_id according to hardware
        pci_id: '0000:05:00.0'
        bridge: br-phy
        migrate_ip: true
    bridges:
      - name: br-phy
    bonds: []

In the example above, bonding isn't used and hence an empty list is passed in the "bonds" section.

Deployment

Once the above requirements are met, start deploying Openstack Helm using the deployment scripts under the dpdk directory in an increasing order

./tools/deployment/developer/dpdk/<script-name>

One can also specify the name of Openstack release and container OS distribution as overrides before running the deployment scripts, for instance,

export OPENSTACK_RELEASE=rocky
export CONTAINER_DISTRO_NAME=ubuntu
export CONTAINER_DISTRO_VERSION=bionic

Note that OVS-DPDK deployment has been tested with Openstack Rocky release and Ubuntu Bionic container distributions. If the above variables aren't set, the defaults (currently Openstack Ocata and Ubuntu Xenial) will be used.

Troubleshooting

OVS startup failure

If OVS fails to start up because of no hugepages are available, check the configuration of the OVS daemonset. Older versions of helm-toolkit were not able to render hugepage configuration into the Kubernetes manifest and just removed the hugepage attributes. If no hugepage configuration is defined for the OVS daemonset, consider using a newer version of helm-toolkit.

kubectl get daemonset openvswitch-vswitchd -n openstack -o yaml
[...]
resources:
  limits:
    cpu: "2"
    hugepages-1Gi: 1Gi
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 128Mi
[...]

Adding a DPDK port to Openvswitch fails

When adding a DPDK port (a NIC bound to DPDK) to OVS fails, one source of error is related to an incorrect configuration with regards to the NUMA topology of the underlying hardware. Every NIC is connected to one specific NUMA socket. In order to use a NIC as DPDK port in OVS, the OVS configurations regarding hugepage(s) and PMD thread(s) need to match the NUMA topology.

The NUMA socket a given NIC is connected to can be found in the ovs-vswitchd log:

kubectl logs -n openstack openvswitch-vswitchd-6h928
[...]
2019-07-02T13:42:06Z|00016|dpdk|INFO|EAL: PCI device 0000:00:04.0 on NUMA socket 1
2019-07-02T13:42:06Z|00018|dpdk|INFO|EAL:   probe driver: 1af4:1000 net_virtio
[...]

In this example, the NIC with PCI-ID 0000:00:04.0 is connected to NUMA socket 1. As a result, this NIC can only be used by OVS if

1. hugepages have been allocated on NUMA socket 1 by OVS, and
2. PMD threads have been assigned to NUMA socket 1.

To allocate hugepages to NUMA sockets in OVS, ensure that the socket_memory attribute in values.yaml specifies a value for the corresponding NUMA socket. In the following example, OVS will use one 1G hugepage for NUMA socket 0 and socket 1.

socket_memory: 1024,1024

To allocate PMD threads to NUMA sockets in OVS, ensure that the pmd_cpu_mask attribute in values.yaml includes CPU sockets on the corresponding NUMA socket. In the example below, the mask of 0xf covers the first 4 CPU cores which are distributed across NUMA sockets 0 and 1.

pmd_cpu_mask: 0xf

The mapping of CPU cores to NUMA sockets can be determined by means of lspci, for instance:

lspci | grep NUMA
NUMA node(s):          2
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15

More information can be found in the Openvswitch documentation.