starlingx/kernel
Code Issues Proposed changes
kernel/kernel-modules/intel-iavf/files/iavf_main-Use-irq_update_affinity_hint.patch
M. Vefa Bicakci 7ded004316 kernel-modules: IRQ affinity hint fix-ups 
This commit modifies a number of out-of-tree kernel modules to ensure
that the irqaffinity= kernel command line option is honored by the
interrupts set up and serviced by the device drivers. For further
information about the rationale for the changes, please see the the
following change: Ibf47fd301a460638f3bb4c49865adc3b2429e06d

Here is a summary of the changes made by this commit:

- i40e: Replicate mainline commit d34c54d1739c ("i40e: Use
  irq_update_affinity_hint()").

- iavf: Replicate mainline commit 0f9744f4ed53 ("iavf: Use
  irq_update_affinity_hint()").

- ice: The device driver is made to use the irq_update_affinity_hint
  function instead of the irq_set_affinity_hint function. Please note
  that this driver was not modified in the mainline kernel, and hence
  this modification is a StarlingX-specific change.

- mlx5: Diverge from mainline commit 7451e9ea8e20 ("net/mlx5: Use
  irq_set_affinity_and_hint()") by using irq_update_affinity_hint
  instead of irq_set_affinity_and_hint, so that StarlingX users can rely
  on the irqaffinity= kernel command line argument to set the affinities
  of the interrupts serviced by mlx5.

  Please note that, due to the way the Mellanox module build works,
  there is a need to have a patch that adds a patch for the build system
  to apply; otherwise, there are patch application failures at build
  time.

The reasons for not modifying the remaining modules are as follows:

- igb_uio: This driver does not use the deprecated API function.

- intel-opae-fpga: This driver does not use the deprecated API function.

- qat17: A patch to make use of the irq_update_affinity_hint function
  was prepared, but after some consideration, it was decided to not
  publish the patch due to concerns about unintended side-effects.

Testing:
- An ISO image was built successfully with this patch, via a monolithic
  build.

- The built ISO image was successfully installed onto and bootstrapped
  on an All-in-One simplex (physical) server with network interfaces
  handled by the i40e and ice (as well as ixgbe) device drivers. The
  low-latency profile was used during the tests.

  The following test steps were carried out with and without this patch
  on the aforementioned All-in-One simplex server:
  - After bootstrap, /etc/rc.d/init.d/affine-platform.sh and
    /usr/bin/affine-interrupts.sh were modified to not manipulate IRQ
    affinities to be able to clearly observe the effect of the changes
    in this patch.
  - The system configuration was changed so that all CPUs other than
    platform CPUs and two additional "application" CPUs were made
    "application-isolated" CPUs.
  - The system was unlocked, and, after the reboot, virtual function
    interfaces were set up.
  - The CPU affinities of all IRQs and IRQ threads were collected with a
    script.
  - iperf3 was used as a sanity test between a virtual function and
    physical function pair.

  Without the patch, the IRQ affinities of the network interfaces were
  spread across all CPUs, regardless of the isolated CPUs and the value
  of the irqaffinity= kernel command line argument.

  With the patch, it was observed that the IRQ affinities of the network
  interfaces aligned with the irqaffinity= command line argument's
  value, which coincides with the aforementioned "application" CPUs.

  iperf3 ran without issues for all tests, but we need to note that with
  this patch iperf3's throughput values were more varied depending on
  which CPUs the scheduler opted to place iperf3 on and the IRQ threads
  that were used by the device driver at the time.

- The device driver changes were also verified using an All-in-One
  simplex server that has Mellanox network interfaces managed by the
  mlx5 driver, but without changes to init scripts. (Tests with mlx5
  were carried out earlier, and we had not thought of modifying the init
  scripts at the time.) The tests were carried out in low-latency
  profile.

  The mlx5 driver appears to initialize all interrupts of an interface
  before it is brought up, which allows the init scripts to set the CPU
  affinities of all IRQs correctly, with and without this patch,
  assuming that the virtual functions are not set up after the
  completion of the init scripts.

  With this patch, we observed that some IRQ threads' CPU affinities
  would initially be set to the platform CPUs instead of aligning with
  the irqaffinity= argument's value, until each IRQ thread serviced at
  least one interrupt. Running iperf3, or bringing the interface down
  and up resulted in correct IRQ thread CPU affinities.

Change-Id: I9574de83738eaed1f03d79cbeb62e9e949cb85ac
Closes-Bug: 1958417
Link: https://lore.kernel.org/netdev/20210903152430.244937-1-nitesh@redhat.com/t/#u
Depends-On: Ibf47fd301a460638f3bb4c49865adc3b2429e06d
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
2022-02-09 10:35:17 -05:00

62 lines
2.3 KiB
Diff
Raw Permalink Blame History

From 1b24525e2971c01eafe7ac0f950dfb3a012035cf Mon Sep 17 00:00:00 2001
From: "M. Vefa Bicakci" <vefa.bicakci@windriver.com>
Date: Fri, 14 Jan 2022 17:39:52 -0500
Subject: [PATCH] iavf_main: Use irq_update_affinity_hint
This commit makes iavf_main use irq_update_affinity_hint instead of
irq_set_affinity_hint to set the CPU affinity hints. This is done
because the latter function sets the IRQ CPU affinities, whereas the
former does not, and this allows the use of the default IRQ affinity CPU
mask provided via the irqaffinity= kernel command line option.
This commit essentially replicates the iavf patch in the following
patch series:
https://lore.kernel.org/netdev/20210903152430.244937-1-nitesh@redhat.com/t/#u
The iavf patch has been mainlined as of this writing:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0f9744f4ed539f2e847d7ed41993b243e3ba5cff
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
---
src/iavf_main.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/iavf_main.c b/src/iavf_main.c
index 8af856576e34..65bf4e939ea3 100644
--- a/src/iavf_main.c
+++ b/src/iavf_main.c
@@ -435,10 +435,10 @@ iavf_request_traffic_irqs(struct iavf_adapter *adapter, char *basename)
#ifdef HAVE_IRQ_AFFINITY_HINT
/* Spread the IRQ affinity hints across online CPUs. Note that
* get_cpu_mask returns a mask with a permanent lifetime so
- * it's safe to use as a hint for irq_set_affinity_hint.
+ * it's safe to use as a hint for irq_update_affinity_hint.
*/
cpu = cpumask_local_spread(q_vector->v_idx, -1);
- irq_set_affinity_hint(irq_num, get_cpu_mask(cpu));
+ irq_update_affinity_hint(irq_num, get_cpu_mask(cpu));
#endif /* HAVE_IRQ_AFFINITY_HINT */
}
@@ -452,7 +452,7 @@ free_queue_irqs:
irq_set_affinity_notifier(irq_num, NULL);
#endif
#ifdef HAVE_IRQ_AFFINITY_HINT
- irq_set_affinity_hint(irq_num, NULL);
+ irq_update_affinity_hint(irq_num, NULL);
#endif
free_irq(irq_num, &adapter->q_vectors[vector]);
}
@@ -508,7 +508,7 @@ static void iavf_free_traffic_irqs(struct iavf_adapter *adapter)
irq_set_affinity_notifier(irq_num, NULL);
#endif
#ifdef HAVE_IRQ_AFFINITY_HINT
- irq_set_affinity_hint(irq_num, NULL);
+ irq_update_affinity_hint(irq_num, NULL);
#endif
free_irq(irq_num, &adapter->q_vectors[vector]);
}
--
2.29.2
View Git Blame Copy Permalink