ubuntu_kernel_selftests:net:fcnal-test.sh: 69 failed test cases on jammy/fips

[Magali Lemes do Sacramento]

Apart from the failed test cases already reported in Bug #2006716, when setting fips=1 we have even more failed test cases. These failures happen on all supported architectures. When fips=0, we have less failed test cases, which are the same failures that happen on the generic kernel. The failed test cases are shown below:

# TEST: ping local, VRF bind - VRF IP [FAIL]
# TEST: ping local, device bind - ns-A IP [FAIL]
# TEST: ping local, VRF bind - VRF IP [FAIL]
# TEST: ping local, device bind - ns-A IP [FAIL]
# TEST: MD5: Single address config [FAIL]
# TEST: MD5: Server no config, client uses password [FAIL]
# TEST: MD5: Client uses wrong password [FAIL]
# TEST: MD5: Client address does not match address configured with password [FAIL]
# TEST: MD5: Prefix config [FAIL]
# TEST: MD5: Prefix config, client uses wrong password [FAIL]
# TEST: MD5: Prefix config, client address not in configured prefix [FAIL]
# TEST: MD5: Single address config [FAIL]
# TEST: MD5: Server no config, client uses password [FAIL]
# TEST: MD5: Client uses wrong password [FAIL]
# TEST: MD5: Client address does not match address configured with password [FAIL]
# TEST: MD5: Prefix config [FAIL]
# TEST: MD5: Prefix config, client uses wrong password [FAIL]
# TEST: MD5: Prefix config, client address not in configured prefix [FAIL]
# TEST: MD5: VRF: Single address config [FAIL]
# TEST: MD5: VRF: Server no config, client uses password [FAIL]
# TEST: MD5: VRF: Client uses wrong password [FAIL]
# TEST: MD5: VRF: Client address does not match address configured with password [FAIL]
# TEST: MD5: VRF: Prefix config [FAIL]
# TEST: MD5: VRF: Prefix config, client uses wrong password [FAIL]
# TEST: MD5: VRF: Prefix config, client address not in configured prefix [FAIL]
# TEST: MD5: VRF: Single address config in default VRF and VRF, conn in VRF [FAIL]
# TEST: MD5: VRF: Single address config in default VRF and VRF, conn in default VRF [FAIL]
# TEST: MD5: VRF: Single address config in default VRF and VRF, conn in default VRF with VRF pw [FAIL]
# TEST: MD5: VRF: Single address config in default VRF and VRF, conn in VRF with default VRF pw [FAIL]
# TEST: MD5: VRF: Prefix config in default VRF and VRF, conn in VRF [FAIL]
# TEST: MD5: VRF: Prefix config in default VRF and VRF, conn in default VRF [FAIL]
# TEST: MD5: VRF: Prefix config in default VRF and VRF, conn in default VRF with VRF pw [FAIL]
# TEST: MD5: VRF: Prefix config in default VRF and VRF, conn in VRF with default VRF pw [FAIL]
# TEST: MD5: VRF: VRF-bound server, unbound key accepts connection [FAIL]
# TEST: MD5: VRF: VRF-bound server, bound key accepts connection [FAIL]
# TEST: MD5: VRF: Global server, Key bound to ifindex=0 rejects VRF connection [FAIL]
# TEST: MD5: VRF: Global server, key bound to ifindex=0 accepts non-VRF connection [FAIL]
# TEST: MD5: VRF: Global server, key not bound to ifindex accepts VRF connection [FAIL]
# TEST: MD5: VRF: Global server, key not bound to ifindex accepts non-VRF connection [FAIL]
# TEST: ping out, vrf device+address bind - ns-B IPv6 LLA [FAIL]
# TEST: MD5: Single address config [FAIL]
# TEST: MD5: Server no config, client uses password [FAIL]
# TEST: MD5: Client uses wrong password [FAIL]
# TEST: MD5: Client address does not match address configured with password [FAIL]
# TEST: MD5: Prefix config [FAIL]
# TEST: MD5: Prefix config, client uses wrong password [FAIL]
# TEST: MD5: Prefix config, client address not in configured prefix [FAIL]
# TEST: MD5: Single address config [FAIL]
# TEST: MD5: Server no config, client uses password [FAIL]
# TEST: MD5: Client uses wrong password [FAIL]
# TEST: MD5: Client address does not match address configured with password [FAIL]
# TEST: MD5: Prefix config [FAIL]
# TEST: MD5: Prefix config, client uses wrong password [FAIL]
# TEST: MD5: Prefix config, client address not in configured prefix [FAIL]
# TEST: MD5: VRF: Single address config [FAIL]
# TEST: MD5: VRF: Server no config, client uses password [FAIL]
# TEST: MD5: VRF: Client uses wrong password [FAIL]
# TEST: MD5: VRF: Client address does not match address configured with password [FAIL]
# TEST: MD5: VRF: Prefix config [FAIL]
# TEST: MD5: VRF: Prefix config, client uses wrong password [FAIL]
# TEST: MD5: VRF: Prefix config, client address not in configured prefix [FAIL]
# TEST: MD5: VRF: Single address config in default VRF and VRF, conn in VRF [FAIL]
# TEST: MD5: VRF: Single address config in default VRF and VRF, conn in default VRF [FAIL]
# TEST: MD5: VRF: Single address config in default VRF and VRF, conn in default VRF with VRF pw [FAIL]
# TEST: MD5: VRF: Single address config in default VRF and VRF, conn in VRF with default VRF pw [FAIL]
# TEST: MD5: VRF: Prefix config in default VRF and VRF, conn in VRF [FAIL]
# TEST: MD5: VRF: Prefix config in default VRF and VRF, conn in default VRF [FAIL]
# TEST: MD5: VRF: Prefix config in default VRF and VRF, conn in default VRF with VRF pw [FAIL]
# TEST: MD5: VRF: Prefix config in default VRF and VRF, conn in VRF with default VRF pw [FAIL]

[Po-Hsu Lin (cypressyew)]

Interesting finding on J-aws-fips 5.15.0-1046.51+fips3, this is not failing with a huge amount of sub-tests like in 5.15.0-1042.47+fips1. Just 4 or 5 of them this time.

[Magali Lemes do Sacramento (magalilemes)]

Commit f2290f518f ("selftests: net: fcnal-test: check if FIPS mode is enabled") fixes this issue and was added to the Jammy kernels through upstream stable rebase (v5.15.119).
I verified the results of the following FIPS kernels from the 2023.09.04-1 cycle:
- j:fips 5.15.0-85.95+fips1
- j:aws-fips 5.15.0-1046.51+fips3
- j:ibm-gt-fips 5.15.0-1045.48+fips1
Therefore, I'm changing this issue to 'Fix released', as the FIPS kernels are now only failing with Bug #2006716.