Support IPSEC full offload implementation

[Tony Duan]

Summary:
Align Kernel IPsec Full offload implementation in the DPU to the upstream Full
offload in all components: OFED, Strongswan, etc.
This is in order for DPU Kernel IPsec to include policy offload and be fully
aligned to what CX Kernel customers will use.

How to test:
Host 1
Enable sriov and set namespace.

ip link set eth2 up
echo '1' > /sys/class/net/eth2/device/sriov_numvfs
ip netns add nt1
ip link set eth4 netns nt1
ip netns exec nt1 ifconfig eth4 11.11.11.1/24 up

BF on host 1:
Set steering mode to "dmfs". By default, it is "smfs" and not supported for now.

/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.0/net/p0/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p0/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode switchdev

/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.1/net/p1/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p1/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode switchdev

IPSec configure
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.2 dst 2.2.2.3 offload packet dev p0 dir out tmpl src 2.2.2.2/16 dst 2.2.2.3/16 proto esp reqid 0xb29ed314 mode transport priority 12
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.3 dst 2.2.2.2 offload packet dev p0 dir in tmpl src 2.2.2.3/16 dst 2.2.2.2/16 proto esp reqid 0xc35aa26e mode transport priority 12
/opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.2/16 dst 2.2.2.3/16 proto esp spi 0xb29ed314 reqid 0xb29ed314 mode transport aead 'rfc4106(gcm(aes))' 0x20f01f80a26f633d85617465686c32552c92c42f 128 offload packet dev p0 dir out sel src 2.2.2.2/16 dst 2.2.2.3/16 flag esn replay-window 64
/opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.3/16 dst 2.2.2.2/16 proto esp spi 0xc35aa26e reqid 0xc35aa26e mode transport aead 'rfc4106(gcm(aes))' 0x6cb228189b4c6e82e66e46920a2cde39187de4ba 128 offload packet dev p0 dir in sel src 2.2.2.3/16 dst 2.2.2.2/16 flag esn replay-window 64
OVS configure. Clear all bridges before configure if there's already default bridges in BF.

ovs-vsctl set Open_vSwitch . other_config:hw-offload=false # need to restart ovs after setting this command

ovs-vsctl add-br br-int
ovs-vsctl add-port br-int pf0vf0 -- set interface pf0vf0 options:representor=[0]
ovs-vsctl add-port br-int vxlan0 -- set interface vxlan0 type=vxlan options:key=100 options:local_ip=2.2.2.2 options:remote_ip=2.2.2.3 options:dst_port=4789
Configure IP

ifconfig p0 2.2.2.2/16 up

Host2:
Enable sriov and set namespace.

ip link set eth2 up
echo '1' > /sys/class/net/eth2/device/sriov_numvfs
ip netns add nt1
ip link set eth4 netns nt1
ip netns exec nt1 ifconfig eth4 11.11.11.2/24 up

BF on host 2
Set steering mode

/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.0/net/p0/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p0/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode switchdev

/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.1/net/p1/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p1/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode switchdev
IPSec configure

/opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.3 dst 2.2.2.2 offload packet dev p0 dir out tmpl src 2.2.2.3/16 dst 2.2.2.2/16 proto esp reqid 0xc35aa26e mode transport priority 12
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.2 dst 2.2.2.3 offload packet dev p0 dir in tmpl src 2.2.2.2/16 dst 2.2.2.3/16 proto esp reqid 0xb29ed314 mode transport priority 12
/opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.3/16 dst 2.2.2.2/16 proto esp spi 0xc35aa26e reqid 0xc35aa26e mode transport aead 'rfc4106(gcm(aes))' 0x6cb228189b4c6e82e66e46920a2cde39187de4ba 128 offload packet dev p0 dir out sel src 2.2.2.3/16 dst 2.2.2.2/16 flag esn replay-window 64
/opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.2/16 dst 2.2.2.3/16 proto esp spi 0xb29ed314 reqid 0xb29ed314 mode transport aead 'rfc4106(gcm(aes))' 0x20f01f80a26f633d85617465686c32552c92c42f 128 offload packet dev p0 dir in sel src 2.2.2.2/16 dst 2.2.2.3/16 flag esn replay-window 64
OVS configure

ovs-vsctl set Open_vSwitch . other_config:hw-offload=false # need to restart ovs after setting this command

ovs-vsctl add-br br-int
ovs-vsctl add-port br-int pf0vf0 -- set interface pf0vf0 options:representor=[0]
ovs-vsctl add-port br-int vxlan0 -- set interface vxlan0 type=vxlan options:key=100 options:local_ip=2.2.2.3 options:remote_ip=2.2.2.2 options:dst_port=4789
Configure IP

ifconfig p0 2.2.2.3/16 up

On host 2, check ping 11.11.11.1 is OK and check IPsec counters in "ethtool -S p0" statistics on both BF.

root@c-237-153-60-065:~# ip netns exec nt1 ping 11.11.11.1

How to fix:
Need to backport a series of xfrm patches into BlueField 5.15 kernel, from 6.0 upstream kernel.
Patches needed for 5.15 kernel:
afe9e47 xfrm: fix conflict for netdev and tx stats
6aff54d xfrm: don't skip free of empty state in acquire policy
692fecb xfrm: delete offloaded policy
91b6276 xfrm: Support UDP encapsulation in packet offload mode
69e168a xfrm: add missed call to delete offloaded policies
9724724 xfrm: release all offloaded policy memory
e57b7ec xfrm: don't require advance ESN callback for packet offload
9e98488 xfrm: copy_to_user_state fetch offloaded SA packets/bytes statistics
4778c10 xfrm: add new device offload acquire flag
2601c94 netlink: provide an ability to set default extack message
b4951d5 netlink: add support for formatted extack messages
b5dd0fa xfrm: extend add state callback to set failure reason
326a004 xfrm: extend add policy callback to set failure reason
40b173d1 xfrm: document IPsec packet offload mode
b1737ae xfrm: add support to HW update soft and hard limits
cad4cd7 xfrm: speed-up lookup of HW policies
b347fe7 xfrm: add TX datapath support for IPsec packet offload mode
cfcc50f xfrm: add an interface to offload policy
2f7e5f7 xfrm: propagate extack to all netlink doit handlers
8d459bb xfrm: add extack to verify_policy_type
3563725 xfrm: allow state packet offload mode
207abea xfrm: add extack support to xfrm_dev_state_add
facf282 xfrm: add new packet offload flag
6f12533 xfrm: Remove not-used total variable
46bd9eb xfrm: drop not needed flags variable in XFRM offload struct
bbadbe7 xfrm: store and rely on direction to construct offload flags
c01b278 xfrm: rename xfrm_state_offload struct to allow reuse
f337706 xfrm: delete not used number of external headers
db0cee8 Revert "UBUNTU: SAUCE: net/xfrm: Fix XFRM flags validity check"
57995bb Revert "UBUNTU: SAUCE: net/xfrm: IPsec full offload support for lifetime limit"
244050a Revert "UBUNTU: SAUCE: net/xfrm: Add support for xfrm full offload"

[Bartlomiej Zolnierkiewicz (bzolnier)]

This bug is awaiting verification that the linux-bluefield/5.15.0-1025.27 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-bluefield/5.15.0-1027.29 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-bluefield' to 'verification-done-jammy-linux-bluefield'. If the problem still exists, change the tag 'verification-needed-jammy-linux-bluefield' to 'verification-failed-jammy-linux-bluefield'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!