Cannot set max-tx-rate parameter on N3000 FPGA SR-IOV interface

[Steven Webster]

Brief Description
-----------------
After setting the max-tx-rate on an SR-IOV interface belonging to an N3000 FPGA device, the system may fail to go enabled after unlock.

Severity
--------
Major: System/Feature is usable but degraded

Steps to Reproduce
------------------

Must be performed on an N3000 FPGA network interface (vendor 8086 device 0d58)

- system host-lock
- system host-if-modify <host> <interface_name> -c pci-sriov -N <number of vfs> --max-tx-rate <max_tx_rate>
- system host-unlock

After unlock, the system may fail to go enabled if puppet configures the max-tx-rate on the VFs after the device is reset but before it is brought down/up.

Expected Behavior
------------------
After unlock, the system should be enabled and the SR-IOV VFs should have the appropriate max-tx-rate.

Actual Behavior
----------------
In some cases, the system fails to come up after unlock.

Reproducibility
---------------
intermittent

System Configuration
--------------------
N3000 FPGA required on AIO/Standard

Branch/Pull Time/Commit
-----------------------
master

Test Activity
-------------
Testing

Workaround
----------
A reboot/lock/unlock may resolve the problem.

[Ghada Khalil (gkhalil)]

screening: stx.7.0 / medium - specific failure w/ a particular config; workaround exists. Sufficient to fix in the active branch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/823824

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/823824
Committed: https://opendev.org/starlingx/stx-puppet/commit/9744f4ba6a01d12e95a8554cc223777b5104f935
Submitter: "Zuul (22348)"
Branch: master

commit 9744f4ba6a01d12e95a8554cc223777b5104f935
Author: Steven Webster <email address hidden>
Date: Wed Jan 5 13:40:38 2022 -0500

    Fix for SR-IOV VF rate limiting on N3000 FPGA NICs

    It is possible that a user may encounter puppet failures on system
    lock/unlock after configuring tx max rate-limiting on an SR-IOV
    interface belonging to an N3000 FPGA.

    The problem may occur if puppet configures the max tx rate on the
    VF(s) after the device has been reset, but before it has been
    brought down/up (which causes the VFs to be created).

    This commit fixes the issue by adding an explicit dependency
    that ensures the rate limit setting for each VF occurs after
    it's driver binding (this is the more appropriate thing to
    do even outside of the context of the N3000 issue).

    Since there is an existing dependency on the N3000 for the
    driver bindings to occur after the device reset/ifdown/ifup,
    the particular issue noted above is resolved.

    Testing:

    PASS: examine the puppet logs and ensure the relationship
          between the driver binding and rate-limit setting for each
          configured VF has been noted.
    PASS: ensure the system unlocks successfully and that the
          VFs have the appropriate tx max rate setting and have
          been associated with the appropriate driver.
    PASS: tested the rate-limit setting with VF drivers vfio and
          netdevice (kernel driver)

    Regression:

    PASS: ensure that if a user has not configured rate-limiting
          that the dependency is noted, but the rate-limiting
          action is not performed.
    PASS: ensure that the system behaves as before for SR-IOV
          interfaces that are not attached to an N3000 FPGA.

    Closes-Bug: 1956604
    Signed-off-by: Steven Webster <email address hidden>
    Change-Id: I5321aa94c8bc0e93fcc107429d44f294da1d17ff