Introducing ConnectX-4 Ethernet SRIOV

[Talat Batheesh]

Hi,

This patchset introduces the support of Ethernet SRIOV in ConnectX-4
family of 100G Ethernet NICs.

Basic Introduction:
ConnectX-4 HW architecture provides two kinds of underlying HW switches.

MPFS (Multi Physical Function Switch) or L2 Table in Software terms:

The HCA has one MPFS switch per physical port, this switch is responsible
of forwarding Unicast traffic to the various overlying Physical Functions (PFs).
Multicast traffic is flooded amongst all the PFs, Each PF can request to
forward a unicast MAC to its E-Switch Uplink vport (which we will cover later)
through SET_L2_TABLE_ENTRY HW command.

MPFS has five ports, four are connected to PFs (one for each) and one is connected
directly to the Physical Port (Physical Link).

E-Switch (Ethernet Switch):

The HCA has one per physical function. The main responsibility of this component is
to forward Unicast/Multicast and vlan tagged/untagged traffic to the various
Virtual Functions (VFs) allocated by the PF. Unlike MPFS, the PF needs to explicitly
create the E-Switch FDB table, Which is a HW flow table managed by the PF driver
whenever vport_group_manager capability bit is set for this PF.

E-Switch has Virtual Ports (vports) entities as its ports, vport0 and uplink vport
are special kind of vports that represents PF vport (vport0) and uplink vport which
is connected to the MPFS switch (if exists) as the PF external link.
vport1..vportN represent VF0..VF(N-1) egress/ingress ports.

E-Switch FDB contains forwarding rules such as:
        UC MAC0 -> vport0(PF).
        UC MAC1 -> vport1.
        UC MAC2 -> vport2.
        MC MACX -> vport0, vport2, Uplink.
        MC MACY -> vport1, Uplink.

    For unmatched traffic FDB has the following default rules:
        Unmatched Traffic (src vport != Uplink) -> Uplink.
        Unmatched Traffic (src vport == Uplink) -> vport0(PF).

NIC VPort context:
Each NIC (VF/PF) has its own vport context which will be used to store the current
NIC vport context (UC/MC and vlan lists) and other NIC properties such as MTU, promisc
mode, etc.. NIC (VF/PF) driver is responsible of constantly updating this context.

FDB rules population:
Each NIC vport (VF/PF) will notify E-Switch manager of its UC/MC vport
context changes via modify vport context command, which will be
translated to an event that will be handled by E-Switch manager (PF)
which will update FDB table accordingly.

Both PF and VF use the same driver and submit commands directly to the firmware.
The PF sees the vport_group_manager capability bit and as such runs the code
to populate the embedded switches as explained above.

The patch goes as follows:

Patches 1-2 introduces the basic PCI SRIOV functionalities and the support of
Connectx4 to enable specific VFs via enable/disable HCA commands. These two
patches will be also in use later for the IB SRIOV flow.

Patches 3-8 Introduces the basic E-Switch capabilities and commands to be used later by
VF to modify and update its NIC vport context, and by PF (E-Switch Manager) driver to
Query the VF NIC context and acts accordingly.

Patches 9-10 Provide the needed functionality of a NIC driver VF/PF to support SRIOV,
mainly vport context update support.

Patch 11 ("net/mlx5: Introducing E-Switch and l2 table"), Introduces the basic
E-Switch support and infrastructure to read vport context events and to update
MPFS L2 Table of the UC mac addresses request by the PF.

Patches 12-18 Introduces SRIOV enablemenet and E-Switch FDB table management
It adds the Basic E-Swtich public API to set and get sriov properties to be used
in PF netdev sriov ndos.

Patchset was applied ontop of commit 3f8c0f7 "gianfar: use of_property_read_bool()"

Saeed, Eli and Or.

Eli Cohen (2):
  net/mlx5_core: Modify enable/disable hca functions
  net/mlx5_core: Add base sriov support

Saeed Mahameed (18):
  net/mlx5: Add HW capabilities and structs for SR-IOV E-Switch.
  net/mlx5: Update access functions to Query/Modify vport MAC address
  net/mlx5: Introduce access functions to modify/query vport mac lists
  net/mlx5: Introduce access functions to modify/query vport state
  net/mlx5: Introduce access functions to modify/query vport promisc mode
  net/mlx5: Introduce access functions to modify/query vport vlans
  net/mlx5e: Write UC/MC list and promisc mode into vport context
  net/mlx5e: Write vlan list into vport context
  net/mlx5: Introducing E-Switch and l2 table
  net/mlx5: E-Switch, Introduce FDB hardware capabilities
  net/mlx5: E-Switch, Add SR-IOV (FDB) support
  net/mlx5: E-Switch, Introduce Vport administration functions
  net/mlx5: E-Switch, Introduce HCA cap and E-Switch vport context
  net/mlx5: E-Switch, Introduce set vport vlan (VST mode)
  net/mlx5: E-Switch, Introduce get vf statistics
  net/mlx5e: Add support for SR-IOV ndos
 net/mlx5e: Assign random MAC address if needed
 net/mlx5: Fix query E-Switch capabilities

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1540435

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Tim Gardner (timg-tpi)]

How about a pull request ? Be sure to use 'git cherry-pick -x'.

[Talat Batheesh (talat-b87)]

Thank you Tim,

All this patches is accepted to upstream.

could you please tell me how can i acsess the Launchpad github ?
can i backport them on my setup and attach the patches ?

thanks,
Talat

[Tim Gardner (timg-tpi)]

Use 'git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial master'. Cherry-pick the appropriate patches along with their prerequisites using the '-x' option so that I know from which upstream commit the patch originates. You can create a git repository on your Launchpad account if you have no other public means, then send a pull request (which is more efficient for me then attaching the patches).

[Talat Batheesh (talat-b87)]

Thank you,
you can take the patches from my brunch.
i've cherry-piced them and test them on machine with xenial release.

lp:~talat-b87/+junk/mlx5-sriov

Thanks,
Talat

[Tim Gardner (timg-tpi)]

Talat - I can't use a BZR branch. Plus, it looks like you'e squashed all of the patches together. What I really want is a pull request from a git repository that shows each upstream commit that you've used to enable SR-IOV. Be sure to use the '-x' option to 'git cherry-pick' when constructing your git branch.

[Talat Batheesh (talat-b87)]

I am sorry about that, could you please tell me how can i do this pull request.
also attached a tar file that contain all of the commits.
i do cherry-pick them on machine with Xenial and kernel 16.04, it's work properly.

Thanks,
Talat

[Tim Gardner (timg-tpi)]

The individual commits in the tar file were enough for me to figure it out.

[Tim Gardner (timg-tpi)]

net/mlx5_core: Modify enable/disable hca functions
net/mlx5_core: Add base sriov support
net/mlx5: Add HW capabilities and structs for SR-IOV E-Switch
net/mlx5: Update access functions to Query/Modify vport MAC address
net/mlx5: Introduce access functions to modify/query vport mac lists
net/mlx5: Introduce access functions to modify/query vport state
net/mlx5: Introduce access functions to modify/query vport promisc mode
net/mlx5: Introduce access functions to modify/query vport vlans
net/mlx5e: Write UC/MC list and promisc mode into vport context
net/mlx5e: Write vlan list into vport context
net/mlx5: Introducing E-Switch and l2 table
net/mlx5: E-Switch, Introduce FDB hardware capabilities
net/mlx5: E-Switch, Add SR-IOV (FDB) support
net/mlx5: E-Switch, Introduce Vport administration functions
net/mlx5: E-Switch, Introduce HCA cap and E-Switch vport context
net/mlx5: E-Switch, Introduce set vport vlan (VST mode)
net/mlx5: E-Switch, Introduce get vf statistics
net/mlx5e: Add support for SR-IOV ndos
net/mlx5: Fix query E-Switch capabilities
net/mlx5e: Assign random MAC address if needed

[Talat Batheesh (talat-b87)]

Hi,

Could you please tell me when we have this feature in the Xenial kernel ?
I didn't see them in the master branch yet.

thanks,
Talat

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-6.21

---------------
linux (4.4.0-6.21) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1546283

  * Naples/Zen, NTB Driver (LP: #1542071)
    - [Config] CONFIG_NTB_AMD=m
    - NTB: Add support for AMD PCI-Express Non-Transparent Bridge

  * [Hyper-V] kernel panic occurs when installing Ubuntu Server x32 (LP: #1495983)
    - SAUCE: storvsc: use small sg_tablesize on x86

  * Enable arm64 emulation of removed ARMv7 instructions (LP: #1545542)
    - [Config] CONFIG_ARMV8_DEPRECATED=y

  * Surelock-GA2:kernel panic/ exception @ pcibios_set_pcie_reset_state+0x118/0x280 + cxl_reset+0x5c/0xc0 (LP: #1545037)
    - powerpc/eeh: Fix stale cached primary bus

  * Miscellaneous Ubuntu changes
    - SAUCE: fs: Add user namesapace member to struct super_block
    - SAUCE: fs: Limit file caps to the user namespace of the super block
    - SAUCE: Smack: Add support for unprivileged mounts from user namespaces
    - SAUCE: block_dev: Support checking inode permissions in lookup_bdev()
    - SAUCE: block_dev: Check permissions towards block device inode when mounting
    - SAUCE: fs: Treat foreign mounts as nosuid
    - SAUCE: selinux: Add support for unprivileged mounts from user namespaces
    - SAUCE: userns: Replace in_userns with current_in_userns
    - SAUCE: Smack: Handle labels consistently in untrusted mounts
    - SAUCE: fs: Check for invalid i_uid in may_follow_link()
    - SAUCE: cred: Reject inodes with invalid ids in set_create_file_as()
    - SAUCE: fs: Refuse uid/gid changes which don't map into s_user_ns
    - SAUCE: fs: Update posix_acl support to handle user namespace mounts
    - SAUCE: fs: Ensure the mounter of a filesystem is privileged towards its inodes
    - SAUCE: fs: Don't remove suid for CAP_FSETID in s_user_ns
    - SAUCE: fs: Allow superblock owner to access do_remount_sb()
    - SAUCE: capabilities: Allow privileged user in s_user_ns to set security.* xattrs
    - SAUCE: fuse: Add support for pid namespaces
    - SAUCE: fuse: Support fuse filesystems outside of init_user_ns
    - SAUCE: fuse: Restrict allow_other to the superblock's namespace or a descendant
    - SAUCE: fuse: Allow user namespace mounts
    - SAUCE: mtd: Check permissions towards mtd block device inode when mounting
    - SAUCE: fs: Update i_[ug]id_(read|write) to translate relative to s_user_ns
    - SAUCE: quota: Convert ids relative to s_user_ns
    - SAUCE: evm: Translate user/group ids relative to s_user_ns when computing HMAC
    - SAUCE: fs: Allow CAP_SYS_ADMIN in s_user_ns to freeze and thaw filesystems
    - SAUCE: quota: Treat superblock owner as privilged
    - SAUCE: ima/evm: Allow root in s_user_ns to set xattrs
    - SAUCE: block_dev: Forbid unprivileged mounting when device is opened for writing
    - SAUCE: ext4: Add support for unprivileged mounts from user namespaces
    - SAUCE: ext4: Add module parameter to enable user namespace mounts
    - SAUCE: fuse: Add module parameter to enable user namespace mounts

  * Miscellaneous upstream changes
    - megaraid: Fix possible NULL pointer deference in mraid_mm_ioctl
    - libahci: Implement the capability to override th...