Regression for net:vrf-xfrm-tests.sh with 5.15 kernel on ARM64 node scobee-kernel

[Po-Hsu Lin]

Issue found on ARM64 node scobee-kernel with:
 * J-5.15.0-86.95 lowlatency
 * 5.15.0-85.95~20.04.2 generic
 * F-5.15.0-86.95~20.04.1 lowlatency
 * F-5.15.0-87.96~20.04.1 lowlatency-64k

Test failed with:
$ sudo ./vrf-xfrm-tests.sh

No qdisc on VRF device
TEST: IPv4 no xfrm policy [ OK ]
TEST: IPv6 no xfrm policy [ OK ]
TEST: IPv4 xfrm policy based on address [FAIL]
TEST: IPv6 xfrm policy based on address [FAIL]
TEST: IPv6 xfrm policy with VRF in selector [ OK ]
TEST: IPv4 xfrm policy with xfrm device [FAIL]
TEST: IPv6 xfrm policy with xfrm device [FAIL]

netem qdisc on VRF device
TEST: IPv4 no xfrm policy [ OK ]
TEST: IPv6 no xfrm policy [ OK ]
TEST: IPv4 xfrm policy based on address [FAIL]
TEST: IPv6 xfrm policy based on address [FAIL]
TEST: IPv6 xfrm policy with VRF in selector [ OK ]
TEST: IPv4 xfrm policy with xfrm device [FAIL]
TEST: IPv6 xfrm policy with xfrm device [FAIL]

Tests passed: 6
Tests failed: 8

And this issue does not exist with the following combination:
* F-generic-5.15.0-86.96~20.04.1 howzit-kernel
* F-generic-5.15.0-86.96~20.04.1 wright-kernel
* F-generic-64k-5.15.0-85.95~20.04.2 kopter-kernel
* F-lowlatency-5.15.0-88.98~20.04.1 howzit-kernel
* F-lowlatency-5.15.0-85.94 starmie-kernel
* F-lowlatency-64k-5.15.0-85.94~20.04.1 howzit-kernel
* J-lowlatency-64k-5.15.0-85.94 starmie-kernel
* J-lowlatency-64k-5.15.0-86.95 howzit-kernel

So it looks like this is hardware related.

And the cause seems to be commit cb43c60 (" selftests: net: vrf-xfrm-tests: change authentication and encryption algos"), which lands on the Jammy tree since:
 * Ubuntu-5.15.0-85.95
 * Ubuntu-lowlatency-5.15.0-85.94

With this commit reverted, this test can pass on node scobee-kernel with 5.15.0-87-lowlatency-64k

[Po-Hsu Lin (cypressyew)]

With 5.15.0-84-generic-64k on scobee-kernel, with test case patched with commit cb43c60 it is still failing. Without that commit the test will pass.

It shows this is related to commit cb43c60, not the kernel function itself.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2039816

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Ike Panhc (ikepanhc)]

Interested. I can reproduce this issue with latest vrf-xfrm-tests.sh on 5.15.0-87.97 but can not reproduce with linux v6.3 vrf-xfrm-tests.sh.

This might be a testcase issue.

[Ike Panhc (ikepanhc)]

Checked with vrf-xfrm-tests.sh in mainline kernel. With 5.15.0-87.97 kernel v6.3 passes and v6.4 failed.

[Ike Panhc (ikepanhc)]

Between v6.3 and v6.4 some algorithms has been replaced.

commit cb43c60e64ca67fcc9d23bd08f51d2ab8209d9d7
Author: Magali Lemes <email address hidden>
Date: Tue Jun 13 09:32:21 2023 -0300

    selftests: net: vrf-xfrm-tests: change authentication and encryption algos

    The vrf-xfrm-tests tests use the hmac(md5) and cbc(des3_ede)
    algorithms for performing authentication and encryption, respectively.
    This causes the tests to fail when fips=1 is set, since these algorithms
    are not allowed in FIPS mode. Therefore, switch from hmac(md5) and
    cbc(des3_ede) to hmac(sha1) and cbc(aes), which are FIPS compliant.

[Ike Panhc (ikepanhc)]

Using 5.15.0-25.25 and the results are the same, vrf-xfrm-tests.sh in linux v6.3 passes and v6.4 fails. Good news is this is not because of any kernel upgrade, and the next step is to look if any function missing/broken in Ubuntu 5.15 kernels.

[Ike Panhc (ikepanhc)]

After `rmmod hisi_sec2`, 5.15.0-87.97 kernel passes all tests. In conclusion, after vrf-xfrm-tests algorithms switched since v6.4 kernel, it fails with hisi_sec2 loaded.

[Ike Panhc (ikepanhc)]

Try ‘s/sha1/md5’ and ‘s/aes/des3_ede’ on vrf-xfrm-tests.sh, either way it can passes. I will read more about this test and find out how it fails.

Bug against kunpeng920 is also filed. https://bugs.launchpad.net/kunpeng920/+bug/2043365

[Magali Lemes do Sacramento (magalilemes)]

This seems to be related to the way hisi_sec2 implements aead, specifically with authenc(hmac(sha1),cbc(aes)), authenc(hmac(sha256),cbc(aes)) and authenc(hmac(sha512),cbc(aes)). Other aead algorithms seem to work fine as they don't rely on this module.

Notice that net:xfrm_policy.sh also fails similarly on this node, as it also uses aes and sha1. See LP #2011414.
When running either net:vrf-xfrm-tests.sh or net:xfrm_policy.sh, we get the same output from dmesg:
`hisi_sec2 0000:76:00.0: flag[3], icv[2]`, showing that something is off with the integrity check value.