From 766f01c6041938ce0f6fbd20babf686853fdce24 Mon Sep 17 00:00:00 2001 From: Heitor Matsui Date: Wed, 3 Nov 2021 17:55:23 -0300 Subject: [PATCH] Document PCI IRQ Affinity Agent operation This commit includes a documentation about how the agent operates and how to configure OpenStack to operate properly with it. Story: 2009299 Task: 43806 Signed-off-by: Heitor Matsui Change-Id: I5969b1a415daf97dbad02b09d8429f2d54efd533 --- README.rst | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 README.rst diff --git a/README.rst b/README.rst new file mode 100644 index 00000000..473ef1f1 --- /dev/null +++ b/README.rst @@ -0,0 +1,94 @@ + +========= +utilities +========= + +This file serves as documentation for the components and features included +on the utilities repository. + +PCI IRQ Affinity Agent +---------------------- + +While in OpenStack it is possible to enable instances to use PCI devices, +the interrupts generated by these devices may be handled by host CPUs that +are unrelated to the instance, and this can lead to a performance that is +lower than it could be if the device interrupts were handled by the instance +CPUs. + +The agent only acts over instances with dedicated vCPUs. For instances using +shared vCPUs no action will be taken by the agent. + +The expected outcome from the agent operation is achieving a higher +performance by assigning the instances core to handle the interrupts from +PCI devices used by these instances and avoid interrupts consuming excessive +cycles from the platform cores. + +Agent operation +~~~~~~~~~~~~~~~ + +The agent operates by listening to RabbitMQ notifications from Nova. When +an instance is created or moved to the host, the agent checks for an specific +flavor spec (detailed below) and if it does then it queries libvirt to map +the instance vCPUs into pCPUs from the host. + +Once the agent has the CPU mapping, it determines the IRQ for each PCI +device used by the instance, and then it loops over all PCI devices and +determines which host NUMA node is associated with the device, the pCPUs +that are associated with the NUMA node and finally set the CPU affinity +for the IRQs of the PCI device based on the pCPU list. + +There is also a periodic audit that runs every minute and loops over the +existing IRQs, so that if there are new IRQs that weren't mapped before +the agent maps them, and if there are PCI devices that aren't associated +to an instance that they were before, their IRQ affinity is reset to the +default value. + +Flavor spec +~~~~~~~~~~~ + +The PCI IRQ Affinity Agent uses a specific flavor spec for PCI interrupt +affining, that is used to determine which vCPUs assigned to the instance +must handle the interrupts from the PCI devices: + +- ``hw:pci_irq_affinity_mask=`` + +Where ``vcpus_cpulist`` can assume a comma-separated list of values that +can be expressed as: + +- ``int``: the vCPU expressed by ``int`` will be assigned to handle the + interruptions from the PCI devices +- ``int1-int2``: the vCPUs between ``int1`` and ``int2`` (inclusive) will + be used to handle the interruptions from the PCI devices +- ``^int``: the vCPU expressed by ``int`` will not be assigned to handle the + interruptions from the PCI devices and shall be used to exclude a vCPU + that was included in a previous range + +**NOTE**: ``int`` must be a value between ``0`` and ``flavor.vcpus - 1`` + +Example: ``hw_pci_irq_affinity_mask=1-4,^3,6`` means that vCPUs with indexes +``1,2,4 and 6`` from the vCPU list that Nova allocates to the instance will +be assigned to handle interruptions from the PCI devices. + +Limitations +~~~~~~~~~~~ + +- No CPU affining is performed for instances using shared CPUs (i.e., + when using flavor spec ``hw:cpu_policy=shared``) +- No CPU affining will be performed when invalid ranges are specified on + the flavor spec, the agent instead will log error messages indicating + the problem + +Agent packaging +~~~~~~~~~~~~~~~ + +The agent code resides on the ``starlingx/utilities`` repo, along with the +spec and docker_image files that are used to build a CentOS image with the +agent wheel installed on it. + +The agent is deployed by Armada along with the other OpenStack helm charts; +refer to `PCI IRQ Affinity Agent helm chart`_ on +``starlingx/openstack-armada-app`` repository. + +.. _`PCI IRQ Affinity Agent helm chart`: https://opendev.org/starlingx + /openstack-armada-app/src/branch/master/stx-openstack-helm + /stx-openstack-helm/helm-charts/pci-irq-affinity-agent