posix_ipc test from test_regression_testsuite of ubuntu_qrt_apparmor failed with F-aws-5.15 T3A (could not mq_notify: Connection timed out)

[Po-Hsu Lin]

Issue found on AWS AMD 64 instance t3a.2xlarge with:
 * F-aws-5.15.0-1053.58~20.04.1
 * J-aws-5.15.0-1053.58

This is reproducible, 4 out of 4 attempts on F-aws-5.15.0-1053.58~20.04.1

There is no such issue on t3a.2xlarge with F-aws-5.15.0-1052.57~20.04.1 nor J-aws-5.15.0-1052.57. Therefore it appears to be a possible regression.

The posix_ipc test in test_regression_testsuite will fail with:
  Error: posix_mq_rcv failed. Test 'POSIX MQUEUE (confined receiver root - unconfined sender : mq_notify)' was expected to 'pass'. Reason for failure 'FAIL - could not mq_notify: Connection timed out'

Test log:
ubuntu@t3a:~/testlibp8PXk0/source/focal/apparmor-2.13.3/tests/regression/apparmor$ sudo bash posix_ipc.sh
Error: posix_mq_rcv failed. Test 'POSIX MQUEUE (confined root - mqueue : mq_notify)' was expected to 'pass'. Reason for failure 'FAIL - could not mq_notify: Connection timed out'
xpass: POSIX MQUEUE (confined root - mqueue label 1)
xpass: POSIX MQUEUE (confined root - mqueue label 1 : mq_notify)
xpass: POSIX MQUEUE (confined root - mqueue label 1 : select)
xpass: POSIX MQUEUE (confined root - mqueue label 1 : poll)
xpass: POSIX MQUEUE (confined root - mqueue label 1 : epoll)
xpass: POSIX MQUEUE (confined root - mqueue label 2)
xpass: POSIX MQUEUE (confined root - mqueue label 2 : mq_notify)
xpass: POSIX MQUEUE (confined root - mqueue label 2 : select)
xpass: POSIX MQUEUE (confined root - mqueue label 2 : poll)
xpass: POSIX MQUEUE (confined root - mqueue label 2 : epoll)
Error: posix_mq_rcv failed. Test 'POSIX MQUEUE (confined 1001 - mqueue : mq_notify)' was expected to 'pass'. Reason for failure 'FAIL - could not mq_notify: Connection timed out'
Error: posix_mq_rcv failed. Test 'POSIX MQUEUE (confined receiver 1001 - unconfined sender : mq_notify)' was expected to 'pass'. Reason for failure 'FAIL - could not mq_notify: Connection timed out'
xpass: POSIX MQUEUE (confined 1001 - mqueue label 1)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 1 : mq_notify)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 1 : select)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 1 : poll)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 1 : epoll)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 2)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 2 : mq_notify)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 2 : select)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 2 : poll)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 2 : epoll)

[Georgia Garcia (georgiag)]

I tried reproducing the issue on a local VM with the same kernel (not a AWS instance) but I was not able to reproduce it.
If possible, could you provide me with a AWS instance for me to test for myself? If not, could you run "sudo bash posix_mq.sh -r", paste the contents of the test directory it leaves behind (Files retained in: /tmp/sdtest.*) and the dmesg messages?

georgia@sec-focal-amd64:/tmp/apparmor-2.13.3/tests/regression/apparmor$ sudo bash posix_mq.sh -r
xpass: POSIX MQUEUE (confined root - mqueue label 1)
xpass: POSIX MQUEUE (confined root - mqueue label 1 : mq_notify)
xpass: POSIX MQUEUE (confined root - mqueue label 1 : select)
xpass: POSIX MQUEUE (confined root - mqueue label 1 : poll)
xpass: POSIX MQUEUE (confined root - mqueue label 1 : epoll)
xpass: POSIX MQUEUE (confined root - mqueue label 2)
xpass: POSIX MQUEUE (confined root - mqueue label 2 : mq_notify)
xpass: POSIX MQUEUE (confined root - mqueue label 2 : select)
xpass: POSIX MQUEUE (confined root - mqueue label 2 : poll)
xpass: POSIX MQUEUE (confined root - mqueue label 2 : epoll)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 1)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 1 : mq_notify)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 1 : select)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 1 : poll)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 1 : epoll)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 2)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 2 : mq_notify)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 2 : select)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 2 : poll)
xpass: POSIX MQUEUE (confined 1001 - mqueue label 2 : epoll)
Files retained in: /tmp/sdtest.9123-15605-WZwoQU

georgia@sec-focal-amd64:/tmp/apparmor-2.13.3/tests/regression/apparmor$ uname -a
Linux sec-focal-amd64 5.15.0-1053-aws #58~20.04.1-Ubuntu SMP Mon Jan 22 17:15:01 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[Georgia Garcia (georgiag)]

I was able to reproduce it in the AWS instances for both Focal and Jammy.
In focal, I was able to reproduce the issue up until version 5.4.0-1009-aws, so it's not a regression.
We didn't catch this before because the AppArmor tests only run when the feature is present, which only happened now for mqueue.

To reproduce on earlier kernel versions I ran the following commands
$ sudo apt install dpkg-dev pkg-config libapparmor-dev
$ apt source apparmor
$ cd apparmor-*/tests/regression/apparmor
$ USE_SYSTEM=1 make
$ result=PASS; while [ "$result" == "PASS" ]; do result=""; result=$(sudo ~/apparmor-*/tests/regression/apparmor/posix_mq_rcv -c ~/apparmor-*/tests/regression/apparmor/posix_mq_snd -k /queuename -n mq_notify); done

The last command only returns if we are able to reproduce the issue. It usually reproduces in less than 10 seconds.
Note that the test is running unconfined, so it's unlikely an issue with apparmor.

[Manuel Diewald (diewald)]

Found on Jammy linux 5.15.0-97.107, security cycle s2024.01.08.

[Manuel Diewald (diewald)]

Found on Mantic linux 6.5.0-21.21, security cycle s2024.01.08 (AWS + Azure Standard_A2_v2 instance).

[Bethany Jamison (bjamison)]

Same issue found on Focal azure-5.15 5.15.0-1063.72~20.04.1, SRU cycle 2024.04.01

[Po-Hsu Lin (cypressyew)]

Found on AWS t3a.2xlarge with Jammy 6.8.0-1010.10~22.04.2