Reviewed:  https://review.opendev.org/c/starlingx/kernel/+/825344
Committed: https://opendev.org/starlingx/kernel/commit/7ded00431675bbab05fe254b90efd08eb335f101
Submitter: "Zuul (22348)"
Branch:    master

commit 7ded00431675bbab05fe254b90efd08eb335f101
Author: M. Vefa Bicakci <email address hidden>
Date:   Fri Jan 14 17:10:31 2022 -0500

    kernel-modules: IRQ affinity hint fix-ups
    
    This commit modifies a number of out-of-tree kernel modules to ensure
    that the irqaffinity= kernel command line option is honored by the
    interrupts set up and serviced by the device drivers. For further
    information about the rationale for the changes, please see the the
    following change: Ibf47fd301a460638f3bb4c49865adc3b2429e06d
    
    Here is a summary of the changes made by this commit:
    
    - i40e: Replicate mainline commit d34c54d1739c ("i40e: Use
      irq_update_affinity_hint()").
    
    - iavf: Replicate mainline commit 0f9744f4ed53 ("iavf: Use
      irq_update_affinity_hint()").
    
    - ice: The device driver is made to use the irq_update_affinity_hint
      function instead of the irq_set_affinity_hint function. Please note
      that this driver was not modified in the mainline kernel, and hence
      this modification is a StarlingX-specific change.
    
    - mlx5: Diverge from mainline commit 7451e9ea8e20 ("net/mlx5: Use
      irq_set_affinity_and_hint()") by using irq_update_affinity_hint
      instead of irq_set_affinity_and_hint, so that StarlingX users can rely
      on the irqaffinity= kernel command line argument to set the affinities
      of the interrupts serviced by mlx5.
    
      Please note that, due to the way the Mellanox module build works,
      there is a need to have a patch that adds a patch for the build system
      to apply; otherwise, there are patch application failures at build
      time.
    
    The reasons for not modifying the remaining modules are as follows:
    
    - igb_uio: This driver does not use the deprecated API function.
    
    - intel-opae-fpga: This driver does not use the deprecated API function.
    
    - qat17: A patch to make use of the irq_update_affinity_hint function
      was prepared, but after some consideration, it was decided to not
      publish the patch due to concerns about unintended side-effects.
    
    Testing:
    - An ISO image was built successfully with this patch, via a monolithic
      build.
    
    - The built ISO image was successfully installed onto and bootstrapped
      on an All-in-One simplex (physical) server with network interfaces
      handled by the i40e and ice (as well as ixgbe) device drivers. The
      low-latency profile was used during the tests.
    
      The following test steps were carried out with and without this patch
      on the aforementioned All-in-One simplex server:
      - After bootstrap, /etc/rc.d/init.d/affine-platform.sh and
        /usr/bin/affine-interrupts.sh were modified to not manipulate IRQ
        affinities to be able to clearly observe the effect of the changes
        in this patch.
      - The system configuration was changed so that all CPUs other than
        platform CPUs and two additional "application" CPUs were made
        "application-isolated" CPUs.
      - The system was unlocked, and, after the reboot, virtual function
        interfaces were set up.
      - The CPU affinities of all IRQs and IRQ threads were collected with a
        script.
      - iperf3 was used as a sanity test between a virtual function and
        physical function pair.
    
      Without the patch, the IRQ affinities of the network interfaces were
      spread across all CPUs, regardless of the isolated CPUs and the value
      of the irqaffinity= kernel command line argument.
    
      With the patch, it was observed that the IRQ affinities of the network
      interfaces aligned with the irqaffinity= command line argument's
      value, which coincides with the aforementioned "application" CPUs.
    
      iperf3 ran without issues for all tests, but we need to note that with
      this patch iperf3's throughput values were more varied depending on
      which CPUs the scheduler opted to place iperf3 on and the IRQ threads
      that were used by the device driver at the time.
    
    - The device driver changes were also verified using an All-in-One
      simplex server that has Mellanox network interfaces managed by the
      mlx5 driver, but without changes to init scripts. (Tests with mlx5
      were carried out earlier, and we had not thought of modifying the init
      scripts at the time.) The tests were carried out in low-latency
      profile.
    
      The mlx5 driver appears to initialize all interrupts of an interface
      before it is brought up, which allows the init scripts to set the CPU
      affinities of all IRQs correctly, with and without this patch,
      assuming that the virtual functions are not set up after the
      completion of the init scripts.
    
      With this patch, we observed that some IRQ threads' CPU affinities
      would initially be set to the platform CPUs instead of aligning with
      the irqaffinity= argument's value, until each IRQ thread serviced at
      least one interrupt. Running iperf3, or bringing the interface down
      and up resulted in correct IRQ thread CPU affinities.
    
    Change-Id: I9574de83738eaed1f03d79cbeb62e9e949cb85ac
    Closes-Bug: 1958417
    Link: https://<email address hidden>/t/#u
    Depends-On: Ibf47fd301a460638f3bb4c49865adc3b2429e06d
    Signed-off-by: M. Vefa Bicakci <email address hidden>