SRIOV CNI: Cannot assign requested MAC address

[Ilya Maximets]

SR-IOV binding driver uses pyroute2 library to set MAC addresses to VFs.
This is internally implemented via ioctl(SIOCSIFHWADDR) giving it the
name of that device. This is equal to calling
'ip link set dev $VFDEV address $MAC'.
However, there is another way to set MAC address for VF. It works via
netlink RTM_SETLINK message to the PF. This is equal to calling
'ip link set dev $PFDEV vf $VFID mac $MAC'.

How it works:
* ioctl(SIOCSIFHWADDR) asks the VF driver to set the MAC
  --> VF driver asks PF to set MAC for it
  --> PF sets the MAC for VF.
* RTM_SETLINK message asks the PF to set MAC for VF
  --> PF sets the MAC for VF.

In case of setting directly via PF, PF additionally sets an
"administratively changed MAC" flag for that VF in the PF's driver, and
from that point on (until the PF's driver is reloaded) that VF's MAC
address can't be changed using the method #1.

It's a security feature designed to forbid MAC changing by the guest OS.

Above leads to the issue where SR-IOV CNI is not able to set MAC address
for VF if its MAC was previously administratively set at least once
(by hands or other software):

  ioctl SIOCSIFHWADDR: Cannot assign requested address

  kernel: igb 0000:05:00.0:
    VF 0 attempted to override administratively set MAC address
    Reload the VF driver to resume operations

After that CNI fails the whole transaction, i.e. fails to change the
interface name as well and subsequently fails the binding.

Netlink PF method to change MAC addresses should be used always. This will
additionally forbid the MAC changing from the inside of container.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/653792

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653792
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=d2b223ffca37ce62b799cdec25e427eb97df1ef6
Submitter: Zuul
Branch: master

commit d2b223ffca37ce62b799cdec25e427eb97df1ef6
Author: Ilya Maximets <email address hidden>
Date: Thu Apr 18 16:27:49 2019 +0300

    Set MAC address for VF via netlink message to PF

    SR-IOV binding driver uses pyroute2 library to set MAC addresses
    to VFs. This is internally implemented via ioctl(SIOCSIFHWADDR)
    giving it the name of that device. This is equal to calling
    'ip link set dev $VFDEV address $MAC'. However, there is another
    way to set MAC address for VF. It works via netlink RTM_SETLINK
    message to the PF. This is equal to calling
    'ip link set dev $PFDEV vf $VFID mac $MAC'.

    How it works:
    * ioctl(SIOCSIFHWADDR) asks the VF driver to set the MAC
      --> VF driver asks PF to set MAC for it
      --> PF sets the MAC for VF.
    * RTM_SETLINK message asks the PF to set MAC for VF
      --> PF sets the MAC for VF.

    In case of setting directly via PF, PF additionally sets an
    "administratively changed MAC" flag for that VF in the PF's
    driver, and from that point on (until the PF's driver is
    reloaded) that VF's MAC address can't be changed using the
    method #1.

    It's a security feature designed to forbid MAC changing by the
    guest OS/app inside the container.

    Above leads to the issue where SR-IOV CNI is not able to set MAC
    address for VF if its MAC was previously administratively set at
    least once (by hands or other software):

      ioctl SIOCSIFHWADDR: Cannot assign requested address

      kernel: igb 0000:05:00.0:
        VF 0 attempted to override administratively set MAC address
        Reload the VF driver to resume operations

    After that CNI fails the whole transaction, i.e. fails to change
    the interface name as well and subsequently fails the binding.

    Netlink PF method to change MAC addresses should be used always.
    This will additionally forbid the MAC changing from the inside
    of container.

    Change-Id: Ic47672e4ce645d9d37b520b6a412a44ae61036e1
    Closes-Bug: 1825383
    Co-authored-by: Danil Golov <email address hidden>
    Signed-off-by: Ilya Maximets <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kuryr-kubernetes 1.1.0 release.