This commit modifies a number of out-of-tree kernel modules to ensure
that the irqaffinity= kernel command line option is honored by the
interrupts set up and serviced by the device drivers. For further
information about the rationale for the changes, please see the the
following change: Ibf47fd301a460638f3bb4c49865adc3b2429e06d
Here is a summary of the changes made by this commit:
- i40e: Replicate mainline commit d34c54d1739c ("i40e: Use
irq_update_affinity_hint()").
- iavf: Replicate mainline commit 0f9744f4ed53 ("iavf: Use
irq_update_affinity_hint()").
- ice: The device driver is made to use the irq_update_affinity_hint
function instead of the irq_set_affinity_hint function. Please note
that this driver was not modified in the mainline kernel, and hence
this modification is a StarlingX-specific change.
- mlx5: Diverge from mainline commit 7451e9ea8e20 ("net/mlx5: Use
irq_set_affinity_and_hint()") by using irq_update_affinity_hint
instead of irq_set_affinity_and_hint, so that StarlingX users can rely
on the irqaffinity= kernel command line argument to set the affinities
of the interrupts serviced by mlx5.
Please note that, due to the way the Mellanox module build works,
there is a need to have a patch that adds a patch for the build system
to apply; otherwise, there are patch application failures at build
time.
The reasons for not modifying the remaining modules are as follows:
- igb_uio: This driver does not use the deprecated API function.
- intel-opae-fpga: This driver does not use the deprecated API function.
- qat17: A patch to make use of the irq_update_affinity_hint function
was prepared, but after some consideration, it was decided to not
publish the patch due to concerns about unintended side-effects.
Testing:
- An ISO image was built successfully with this patch, via a monolithic
build.
- The built ISO image was successfully installed onto and bootstrapped
on an All-in-One simplex (physical) server with network interfaces
handled by the i40e and ice (as well as ixgbe) device drivers. The
low-latency profile was used during the tests.
The following test steps were carried out with and without this patch
on the aforementioned All-in-One simplex server:
- After bootstrap, /etc/rc.d/init.d/affine-platform.sh and
/usr/bin/affine-interrupts.sh were modified to not manipulate IRQ
affinities to be able to clearly observe the effect of the changes
in this patch.
- The system configuration was changed so that all CPUs other than
platform CPUs and two additional "application" CPUs were made
"application-isolated" CPUs.
- The system was unlocked, and, after the reboot, virtual function
interfaces were set up.
- The CPU affinities of all IRQs and IRQ threads were collected with a
script.
- iperf3 was used as a sanity test between a virtual function and
physical function pair.
Without the patch, the IRQ affinities of the network interfaces were
spread across all CPUs, regardless of the isolated CPUs and the value
of the irqaffinity= kernel command line argument.
With the patch, it was observed that the IRQ affinities of the network
interfaces aligned with the irqaffinity= command line argument's
value, which coincides with the aforementioned "application" CPUs.
iperf3 ran without issues for all tests, but we need to note that with
this patch iperf3's throughput values were more varied depending on
which CPUs the scheduler opted to place iperf3 on and the IRQ threads
that were used by the device driver at the time.
- The device driver changes were also verified using an All-in-One
simplex server that has Mellanox network interfaces managed by the
mlx5 driver, but without changes to init scripts. (Tests with mlx5
were carried out earlier, and we had not thought of modifying the init
scripts at the time.) The tests were carried out in low-latency
profile.
The mlx5 driver appears to initialize all interrupts of an interface
before it is brought up, which allows the init scripts to set the CPU
affinities of all IRQs correctly, with and without this patch,
assuming that the virtual functions are not set up after the
completion of the init scripts.
With this patch, we observed that some IRQ threads' CPU affinities
would initially be set to the platform CPUs instead of aligning with
the irqaffinity= argument's value, until each IRQ thread serviced at
least one interrupt. Running iperf3, or bringing the interface down
and up resulted in correct IRQ thread CPU affinities.
Change-Id: I9574de83738eaed1f03d79cbeb62e9e949cb85ac
Closes-Bug: 1958417
Link: https://lore.kernel.org/netdev/20210903152430.244937-1-nitesh@redhat.com/t/#u
Depends-On: Ibf47fd301a460638f3bb4c49865adc3b2429e06d
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
7ded004316