| 2023-09-06 13:15:34 |
Tony Duan |
bug |
|
|
added bug |
| 2023-09-06 13:16:20 |
Tony Duan |
description |
Summary:
Align Kernel IPsec Full offload implementation in the DPU to the upstream Full
offload in all components: OFED, Strongswan, etc.
This is in order for DPU Kernel IPsec to include policy offload and be fully
aligned to what CX Kernel customers will use.
How to test:
Host 1:
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.0/net/p0/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p0/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode switchdev
BF on host 1:
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.181.165 dst 196.234.182.166 dir out tmpl src 196.234.181.165/16 dst 196.234.182.166/16 proto esp reqid 0xefa83812 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.182.166 dst 196.234.181.165 dir in tmpl src 196.234.182.166/16 dst 196.234.181.165/16 proto esp reqid 0x63a7db74 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.182.166 dst 196.234.181.165 dir fwd tmpl src 196.234.182.166/16 dst 196.234.181.165/16 proto esp reqid 0x63a7db74 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.181.165/16 dst 196.234.182.166/16 proto esp spi 0xefa83812 reqid 0xefa83812 mode transport aead 'rfc4106(gcm(aes))' 0xe2fe3857301d8f72b5d71d295a462ef21868e407 128 offload packet dev p0 dir out sel src 196.234.181.165/16 dst 196.234.182.166/16 flag esn replay-window 32
/opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.182.166/16 dst 196.234.181.165/16 proto esp spi 0x63a7db74 reqid 0x63a7db74 mode transport aead 'rfc4106(gcm(aes))' 0xe916c4d0db1886e8c877b023e8cebef53b4d2d0f 128 offload packet dev p0 dir in sel src 196.234.182.166/16 dst 196.234.181.165/16 flag esn replay-window 32
Start OVS and set following configure on BF:
/usr/bin/ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
/usr/bin/ovs-vsctl set Open_vSwitch . other_config:max-idle=300000
Host2:
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.1/net/p1/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p1/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode switchdev
BF on host 2:
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.182.166 dst 196.234.181.165 dir out tmpl src 196.234.182.166/16 dst 196.234.181.165/16 proto esp reqid 0xefa83812 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.181.165 dst 196.234.182.166 dir in tmpl src 196.234.181.165/16 dst 196.234.182.166/16 proto esp reqid 0x63a7db74 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.181.165 dst 196.234.182.166 dir fwd tmpl src 196.234.181.165/16 dst 196.234.182.166/16 proto esp reqid 0x63a7db74 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.181.165 dst 196.234.182.166 proto esp spi 0xefa83812 reqid 0xefa83812 mode transport aead 'rfc4106(gcm(aes))' 0xe2fe3857301d8f72b5d71d295a462ef21868e407 128 offload packet dev p0 dir out sel src 196.234.181.165/16 dst 196.234.182.166/16 flag esn replay-window 32
/opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.181.165 dst 196.234.182.166 proto esp spi 0x63a7db74 reqid 0x63a7db74 mode transport aead 'rfc4106(gcm(aes))' 0xe916c4d0db1886e8c877b023e8cebef53b4d2d0f 128 offload packet dev p0 dir in sel src 196.234.181.165/16 dst 196.234.182.166/16 flag esn replay-window 32
Start OVS and set following configure on BF:
/usr/bin/ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
/usr/bin/ovs-vsctl set Open_vSwitch . other_config:max-idle=300000
Send the traffic between host 1 and host 2 and check IPsec counters in "ethtool -S" statistics on both BF.
How to fix:
Need to backport a series of xfrm patches into BlueField 5.15 kernel, from 6.0 upstream kernel.
Patches needed for 5.15 kernel: |
Summary:
Align Kernel IPsec Full offload implementation in the DPU to the upstream Full
offload in all components: OFED, Strongswan, etc.
This is in order for DPU Kernel IPsec to include policy offload and be fully
aligned to what CX Kernel customers will use.
How to test:
Host 1:
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.0/net/p0/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p0/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode switchdev
BF on host 1:
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.181.165 dst 196.234.182.166 dir out tmpl src 196.234.181.165/16 dst 196.234.182.166/16 proto esp reqid 0xefa83812 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.182.166 dst 196.234.181.165 dir in tmpl src 196.234.182.166/16 dst 196.234.181.165/16 proto esp reqid 0x63a7db74 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.182.166 dst 196.234.181.165 dir fwd tmpl src 196.234.182.166/16 dst 196.234.181.165/16 proto esp reqid 0x63a7db74 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.181.165/16 dst 196.234.182.166/16 proto esp spi 0xefa83812 reqid 0xefa83812 mode transport aead 'rfc4106(gcm(aes))' 0xe2fe3857301d8f72b5d71d295a462ef21868e407 128 offload packet dev p0 dir out sel src 196.234.181.165/16 dst 196.234.182.166/16 flag esn replay-window 32
/opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.182.166/16 dst 196.234.181.165/16 proto esp spi 0x63a7db74 reqid 0x63a7db74 mode transport aead 'rfc4106(gcm(aes))' 0xe916c4d0db1886e8c877b023e8cebef53b4d2d0f 128 offload packet dev p0 dir in sel src 196.234.182.166/16 dst 196.234.181.165/16 flag esn replay-window 32
Start OVS and set following configure on BF:
/usr/bin/ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
/usr/bin/ovs-vsctl set Open_vSwitch . other_config:max-idle=300000
Host2:
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.1/net/p1/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p1/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode switchdev
BF on host 2:
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.182.166 dst 196.234.181.165 dir out tmpl src 196.234.182.166/16 dst 196.234.181.165/16 proto esp reqid 0xefa83812 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.181.165 dst 196.234.182.166 dir in tmpl src 196.234.181.165/16 dst 196.234.182.166/16 proto esp reqid 0x63a7db74 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.181.165 dst 196.234.182.166 dir fwd tmpl src 196.234.181.165/16 dst 196.234.182.166/16 proto esp reqid 0x63a7db74 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.181.165 dst 196.234.182.166 proto esp spi 0xefa83812 reqid 0xefa83812 mode transport aead 'rfc4106(gcm(aes))' 0xe2fe3857301d8f72b5d71d295a462ef21868e407 128 offload packet dev p0 dir out sel src 196.234.181.165/16 dst 196.234.182.166/16 flag esn replay-window 32
/opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.181.165 dst 196.234.182.166 proto esp spi 0x63a7db74 reqid 0x63a7db74 mode transport aead 'rfc4106(gcm(aes))' 0xe916c4d0db1886e8c877b023e8cebef53b4d2d0f 128 offload packet dev p0 dir in sel src 196.234.181.165/16 dst 196.234.182.166/16 flag esn replay-window 32
Start OVS and set following configure on BF:
/usr/bin/ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
/usr/bin/ovs-vsctl set Open_vSwitch . other_config:max-idle=300000
Send the traffic between host 1 and host 2 and check IPsec counters in "ethtool -S" statistics on both BF.
How to fix:
Need to backport a series of xfrm patches into BlueField 5.15 kernel, from 6.0 upstream kernel.
Patches needed for 5.15 kernel:
afe9e47 xfrm: fix conflict for netdev and tx stats
6aff54d xfrm: don't skip free of empty state in acquire policy
692fecb xfrm: delete offloaded policy
91b6276 xfrm: Support UDP encapsulation in packet offload mode
69e168a xfrm: add missed call to delete offloaded policies
9724724 xfrm: release all offloaded policy memory
e57b7ec xfrm: don't require advance ESN callback for packet offload
9e98488 xfrm: copy_to_user_state fetch offloaded SA packets/bytes statistics
4778c10 xfrm: add new device offload acquire flag
2601c94 netlink: provide an ability to set default extack message
b4951d5 netlink: add support for formatted extack messages
b5dd0fa xfrm: extend add state callback to set failure reason
326a004 xfrm: extend add policy callback to set failure reason
40b173d1 xfrm: document IPsec packet offload mode
b1737ae xfrm: add support to HW update soft and hard limits
cad4cd7 xfrm: speed-up lookup of HW policies
b347fe7 xfrm: add TX datapath support for IPsec packet offload mode
cfcc50f xfrm: add an interface to offload policy
2f7e5f7 xfrm: propagate extack to all netlink doit handlers
8d459bb xfrm: add extack to verify_policy_type
3563725 xfrm: allow state packet offload mode
207abea xfrm: add extack support to xfrm_dev_state_add
facf282 xfrm: add new packet offload flag
6f12533 xfrm: Remove not-used total variable
46bd9eb xfrm: drop not needed flags variable in XFRM offload struct
bbadbe7 xfrm: store and rely on direction to construct offload flags
c01b278 xfrm: rename xfrm_state_offload struct to allow reuse
f337706 xfrm: delete not used number of external headers
db0cee8 Revert "UBUNTU: SAUCE: net/xfrm: Fix XFRM flags validity check"
57995bb Revert "UBUNTU: SAUCE: net/xfrm: IPsec full offload support for lifetime limit"
244050a Revert "UBUNTU: SAUCE: net/xfrm: Add support for xfrm full offload" |
|
| 2023-09-06 16:15:29 |
Tony Duan |
merge proposal linked |
|
https://code.launchpad.net/~yifeid/ubuntu/+source/linux-bluefield/+git/linux-bluefield/+merge/450800 |
|
| 2023-09-07 19:23:01 |
Bodong Wang |
merge proposal linked |
|
https://code.launchpad.net/~bodong-wang/ubuntu/+source/linux-bluefield/+git/jammy/+merge/450970 |
|
| 2023-09-14 15:19:42 |
Bartlomiej Zolnierkiewicz |
nominated for series |
|
Ubuntu Jammy |
|
| 2023-09-14 15:19:42 |
Bartlomiej Zolnierkiewicz |
bug task added |
|
linux-bluefield (Ubuntu Jammy) |
|
| 2023-09-14 15:19:46 |
Bartlomiej Zolnierkiewicz |
linux-bluefield (Ubuntu): status |
New |
Invalid |
|
| 2023-09-20 10:51:40 |
Bartlomiej Zolnierkiewicz |
linux-bluefield (Ubuntu Jammy): status |
New |
Fix Committed |
|
| 2023-09-25 10:51:01 |
Bartlomiej Zolnierkiewicz |
tags |
|
verification-needed-jammy |
|
| 2023-09-25 11:12:21 |
Tony Duan |
description |
Summary:
Align Kernel IPsec Full offload implementation in the DPU to the upstream Full
offload in all components: OFED, Strongswan, etc.
This is in order for DPU Kernel IPsec to include policy offload and be fully
aligned to what CX Kernel customers will use.
How to test:
Host 1:
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.0/net/p0/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p0/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode switchdev
BF on host 1:
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.181.165 dst 196.234.182.166 dir out tmpl src 196.234.181.165/16 dst 196.234.182.166/16 proto esp reqid 0xefa83812 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.182.166 dst 196.234.181.165 dir in tmpl src 196.234.182.166/16 dst 196.234.181.165/16 proto esp reqid 0x63a7db74 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.182.166 dst 196.234.181.165 dir fwd tmpl src 196.234.182.166/16 dst 196.234.181.165/16 proto esp reqid 0x63a7db74 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.181.165/16 dst 196.234.182.166/16 proto esp spi 0xefa83812 reqid 0xefa83812 mode transport aead 'rfc4106(gcm(aes))' 0xe2fe3857301d8f72b5d71d295a462ef21868e407 128 offload packet dev p0 dir out sel src 196.234.181.165/16 dst 196.234.182.166/16 flag esn replay-window 32
/opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.182.166/16 dst 196.234.181.165/16 proto esp spi 0x63a7db74 reqid 0x63a7db74 mode transport aead 'rfc4106(gcm(aes))' 0xe916c4d0db1886e8c877b023e8cebef53b4d2d0f 128 offload packet dev p0 dir in sel src 196.234.182.166/16 dst 196.234.181.165/16 flag esn replay-window 32
Start OVS and set following configure on BF:
/usr/bin/ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
/usr/bin/ovs-vsctl set Open_vSwitch . other_config:max-idle=300000
Host2:
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.1/net/p1/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p1/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode switchdev
BF on host 2:
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.182.166 dst 196.234.181.165 dir out tmpl src 196.234.182.166/16 dst 196.234.181.165/16 proto esp reqid 0xefa83812 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.181.165 dst 196.234.182.166 dir in tmpl src 196.234.181.165/16 dst 196.234.182.166/16 proto esp reqid 0x63a7db74 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.181.165 dst 196.234.182.166 dir fwd tmpl src 196.234.181.165/16 dst 196.234.182.166/16 proto esp reqid 0x63a7db74 mode transport priority 10
/opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.181.165 dst 196.234.182.166 proto esp spi 0xefa83812 reqid 0xefa83812 mode transport aead 'rfc4106(gcm(aes))' 0xe2fe3857301d8f72b5d71d295a462ef21868e407 128 offload packet dev p0 dir out sel src 196.234.181.165/16 dst 196.234.182.166/16 flag esn replay-window 32
/opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.181.165 dst 196.234.182.166 proto esp spi 0x63a7db74 reqid 0x63a7db74 mode transport aead 'rfc4106(gcm(aes))' 0xe916c4d0db1886e8c877b023e8cebef53b4d2d0f 128 offload packet dev p0 dir in sel src 196.234.181.165/16 dst 196.234.182.166/16 flag esn replay-window 32
Start OVS and set following configure on BF:
/usr/bin/ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
/usr/bin/ovs-vsctl set Open_vSwitch . other_config:max-idle=300000
Send the traffic between host 1 and host 2 and check IPsec counters in "ethtool -S" statistics on both BF.
How to fix:
Need to backport a series of xfrm patches into BlueField 5.15 kernel, from 6.0 upstream kernel.
Patches needed for 5.15 kernel:
afe9e47 xfrm: fix conflict for netdev and tx stats
6aff54d xfrm: don't skip free of empty state in acquire policy
692fecb xfrm: delete offloaded policy
91b6276 xfrm: Support UDP encapsulation in packet offload mode
69e168a xfrm: add missed call to delete offloaded policies
9724724 xfrm: release all offloaded policy memory
e57b7ec xfrm: don't require advance ESN callback for packet offload
9e98488 xfrm: copy_to_user_state fetch offloaded SA packets/bytes statistics
4778c10 xfrm: add new device offload acquire flag
2601c94 netlink: provide an ability to set default extack message
b4951d5 netlink: add support for formatted extack messages
b5dd0fa xfrm: extend add state callback to set failure reason
326a004 xfrm: extend add policy callback to set failure reason
40b173d1 xfrm: document IPsec packet offload mode
b1737ae xfrm: add support to HW update soft and hard limits
cad4cd7 xfrm: speed-up lookup of HW policies
b347fe7 xfrm: add TX datapath support for IPsec packet offload mode
cfcc50f xfrm: add an interface to offload policy
2f7e5f7 xfrm: propagate extack to all netlink doit handlers
8d459bb xfrm: add extack to verify_policy_type
3563725 xfrm: allow state packet offload mode
207abea xfrm: add extack support to xfrm_dev_state_add
facf282 xfrm: add new packet offload flag
6f12533 xfrm: Remove not-used total variable
46bd9eb xfrm: drop not needed flags variable in XFRM offload struct
bbadbe7 xfrm: store and rely on direction to construct offload flags
c01b278 xfrm: rename xfrm_state_offload struct to allow reuse
f337706 xfrm: delete not used number of external headers
db0cee8 Revert "UBUNTU: SAUCE: net/xfrm: Fix XFRM flags validity check"
57995bb Revert "UBUNTU: SAUCE: net/xfrm: IPsec full offload support for lifetime limit"
244050a Revert "UBUNTU: SAUCE: net/xfrm: Add support for xfrm full offload" |
Summary:
Align Kernel IPsec Full offload implementation in the DPU to the upstream Full
offload in all components: OFED, Strongswan, etc.
This is in order for DPU Kernel IPsec to include policy offload and be fully
aligned to what CX Kernel customers will use.
How to test:
Host 1
Enable sriov and set namespace.
ip link set eth2 up
echo '1' > /sys/class/net/eth2/device/sriov_numvfs
ip netns add nt1
ip link set eth4 netns nt1
ip netns exec nt1 ifconfig eth4 11.11.11.1/24 up
BF on host 1:
Set steering mode to "dmfs". By default, it is "smfs" and not supported for now.
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.0/net/p0/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p0/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode switchdev
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.1/net/p1/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p1/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode switchdev
IPSec configure
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.2 dst 2.2.2.3 offload packet dev p0 dir out tmpl src 2.2.2.2/16 dst 2.2.2.3/16 proto esp reqid 0xb29ed314 mode transport priority 12
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.3 dst 2.2.2.2 offload packet dev p0 dir in tmpl src 2.2.2.3/16 dst 2.2.2.2/16 proto esp reqid 0xc35aa26e mode transport priority 12
/opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.2/16 dst 2.2.2.3/16 proto esp spi 0xb29ed314 reqid 0xb29ed314 mode transport aead 'rfc4106(gcm(aes))' 0x20f01f80a26f633d85617465686c32552c92c42f 128 offload packet dev p0 dir out sel src 2.2.2.2/16 dst 2.2.2.3/16 flag esn replay-window 64
/opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.3/16 dst 2.2.2.2/16 proto esp spi 0xc35aa26e reqid 0xc35aa26e mode transport aead 'rfc4106(gcm(aes))' 0x6cb228189b4c6e82e66e46920a2cde39187de4ba 128 offload packet dev p0 dir in sel src 2.2.2.3/16 dst 2.2.2.2/16 flag esn replay-window 64
OVS configure. Clear all bridges before configure if there's already default bridges in BF.
ovs-vsctl set Open_vSwitch . other_config:hw-offload=false # need to restart ovs after setting this command
ovs-vsctl add-br br-int
ovs-vsctl add-port br-int pf0vf0 -- set interface pf0vf0 options:representor=[0]
ovs-vsctl add-port br-int vxlan0 -- set interface vxlan0 type=vxlan options:key=100 options:local_ip=2.2.2.2 options:remote_ip=2.2.2.3 options:dst_port=4789
Configure IP
ifconfig p0 2.2.2.2/16 up
Host2:
Enable sriov and set namespace.
ip link set eth2 up
echo '1' > /sys/class/net/eth2/device/sriov_numvfs
ip netns add nt1
ip link set eth4 netns nt1
ip netns exec nt1 ifconfig eth4 11.11.11.2/24 up
BF on host 2
Set steering mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.0/net/p0/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p0/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode switchdev
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode legacy
echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.1/net/p1/compat/devlink/steering_mode
echo 'full' > /sys/class/net/p1/compat/devlink/ipsec_mode
/opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode switchdev
IPSec configure
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.3 dst 2.2.2.2 offload packet dev p0 dir out tmpl src 2.2.2.3/16 dst 2.2.2.2/16 proto esp reqid 0xc35aa26e mode transport priority 12
/opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.2 dst 2.2.2.3 offload packet dev p0 dir in tmpl src 2.2.2.2/16 dst 2.2.2.3/16 proto esp reqid 0xb29ed314 mode transport priority 12
/opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.3/16 dst 2.2.2.2/16 proto esp spi 0xc35aa26e reqid 0xc35aa26e mode transport aead 'rfc4106(gcm(aes))' 0x6cb228189b4c6e82e66e46920a2cde39187de4ba 128 offload packet dev p0 dir out sel src 2.2.2.3/16 dst 2.2.2.2/16 flag esn replay-window 64
/opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.2/16 dst 2.2.2.3/16 proto esp spi 0xb29ed314 reqid 0xb29ed314 mode transport aead 'rfc4106(gcm(aes))' 0x20f01f80a26f633d85617465686c32552c92c42f 128 offload packet dev p0 dir in sel src 2.2.2.2/16 dst 2.2.2.3/16 flag esn replay-window 64
OVS configure
ovs-vsctl set Open_vSwitch . other_config:hw-offload=false # need to restart ovs after setting this command
ovs-vsctl add-br br-int
ovs-vsctl add-port br-int pf0vf0 -- set interface pf0vf0 options:representor=[0]
ovs-vsctl add-port br-int vxlan0 -- set interface vxlan0 type=vxlan options:key=100 options:local_ip=2.2.2.3 options:remote_ip=2.2.2.2 options:dst_port=4789
Configure IP
ifconfig p0 2.2.2.3/16 up
On host 2, check ping 11.11.11.1 is OK and check IPsec counters in "ethtool -S p0" statistics on both BF.
root@c-237-153-60-065:~# ip netns exec nt1 ping 11.11.11.1
How to fix:
Need to backport a series of xfrm patches into BlueField 5.15 kernel, from 6.0 upstream kernel.
Patches needed for 5.15 kernel:
afe9e47 xfrm: fix conflict for netdev and tx stats
6aff54d xfrm: don't skip free of empty state in acquire policy
692fecb xfrm: delete offloaded policy
91b6276 xfrm: Support UDP encapsulation in packet offload mode
69e168a xfrm: add missed call to delete offloaded policies
9724724 xfrm: release all offloaded policy memory
e57b7ec xfrm: don't require advance ESN callback for packet offload
9e98488 xfrm: copy_to_user_state fetch offloaded SA packets/bytes statistics
4778c10 xfrm: add new device offload acquire flag
2601c94 netlink: provide an ability to set default extack message
b4951d5 netlink: add support for formatted extack messages
b5dd0fa xfrm: extend add state callback to set failure reason
326a004 xfrm: extend add policy callback to set failure reason
40b173d1 xfrm: document IPsec packet offload mode
b1737ae xfrm: add support to HW update soft and hard limits
cad4cd7 xfrm: speed-up lookup of HW policies
b347fe7 xfrm: add TX datapath support for IPsec packet offload mode
cfcc50f xfrm: add an interface to offload policy
2f7e5f7 xfrm: propagate extack to all netlink doit handlers
8d459bb xfrm: add extack to verify_policy_type
3563725 xfrm: allow state packet offload mode
207abea xfrm: add extack support to xfrm_dev_state_add
facf282 xfrm: add new packet offload flag
6f12533 xfrm: Remove not-used total variable
46bd9eb xfrm: drop not needed flags variable in XFRM offload struct
bbadbe7 xfrm: store and rely on direction to construct offload flags
c01b278 xfrm: rename xfrm_state_offload struct to allow reuse
f337706 xfrm: delete not used number of external headers
db0cee8 Revert "UBUNTU: SAUCE: net/xfrm: Fix XFRM flags validity check"
57995bb Revert "UBUNTU: SAUCE: net/xfrm: IPsec full offload support for lifetime limit"
244050a Revert "UBUNTU: SAUCE: net/xfrm: Add support for xfrm full offload" |
|
| 2023-10-03 22:47:31 |
Ubuntu Kernel Bot |
tags |
verification-needed-jammy |
kernel-spammed-jammy-linux-bluefield-v2 verification-needed-jammy verification-needed-jammy-linux-bluefield |
|
| 2023-10-03 23:37:23 |
Feysel Mohammed |
tags |
kernel-spammed-jammy-linux-bluefield-v2 verification-needed-jammy verification-needed-jammy-linux-bluefield |
verification-done-jammy |
|
| 2023-10-04 00:49:51 |
Ubuntu Kernel Bot |
tags |
verification-done-jammy |
kernel-spammed-jammy-linux-bluefield-v2 verification-done-jammy verification-needed-jammy-linux-bluefield |
|
| 2023-10-04 00:58:33 |
Andy Whitcroft |
tags |
kernel-spammed-jammy-linux-bluefield-v2 verification-done-jammy verification-needed-jammy-linux-bluefield |
kernel-spammed-jammy-linux-bluefield-v2 verification-done-jammy verification-done-jammy-linux-bluefield |
|
| 2023-10-04 01:07:28 |
Tony Duan |
tags |
kernel-spammed-jammy-linux-bluefield-v2 verification-done-jammy verification-done-jammy-linux-bluefield |
verification-done-jammy verification-done-jammy-linux-bluefield |
|
| 2023-10-04 01:34:18 |
Ubuntu Kernel Bot |
tags |
verification-done-jammy verification-done-jammy-linux-bluefield |
kernel-spammed-jammy-linux-bluefield-v2 verification-done-jammy verification-needed-jammy-linux-bluefield |
|
| 2023-10-04 01:53:12 |
Tony Duan |
tags |
kernel-spammed-jammy-linux-bluefield-v2 verification-done-jammy verification-needed-jammy-linux-bluefield |
verification-done-jammy verification-done-jammy-linux-bluefield |
|
| 2023-10-04 06:16:02 |
Ubuntu Kernel Bot |
tags |
verification-done-jammy verification-done-jammy-linux-bluefield |
kernel-spammed-jammy-linux-bluefield-v2 verification-done-jammy verification-needed-jammy-linux-bluefield |
|
| 2023-10-04 06:42:58 |
Tony Duan |
tags |
kernel-spammed-jammy-linux-bluefield-v2 verification-done-jammy verification-needed-jammy-linux-bluefield |
kernel-spammed-jammy-linux-bluefield-v2 verification-done-jammy verification-done-jammy-linux-bluefield |
|
