ubuntu_qrt_kernel_security: "test_060_nx" fails on focal/linux-oracle

[Khaled El Mously]

Example failure:

http://10.246.72.9/5.4.0-1045.49-oracle/focal-linux-oracle-oracle-amd64-5.4.0-VM.Standard2.1-ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security/results/ubuntu_qrt_kernel_security.test-kernel-security.py/debug/ubuntu_qrt_kernel_security.test-kernel-security.py.DEBUG.html

Failure output:

585. FAIL: test_060_nx (__main__.KernelSecurityTest)
586. NX bit is working
587. ----------------------------------------------------------------------
588. Traceback (most recent call last):
589. File "./test-kernel-security.py", line 554, in test_060_nx
590. self.assertShellExitEquals(rie_expected, ["./nx-test-rie", "data"])
591. File "/home/ubuntu/autotest/client/tmp/ubuntu_qrt_kernel_security/src/qa-regression-testing/scripts/testlib.py", line 1166, in assertShellExitEquals
592. self.assertEqual(expected, rc, msg + result + report)
593. AssertionError: Got exit code -11, expected 0
594. Command: './nx-test-rie', 'data'
595. Output:
596. rodata:0x559730ee5008
597. data: 0x559730ee7010
598. bss: 0x559730ef7040
599. brk: 0x5597313a62b0
600. rw: 0x7fa963894000
601. rwx: 0x7fa963867000
602. stack: 0x7ffe099e9eb0
603. Dump of /proc/self/maps:
604. 559730ee3000-559730ee4000 r--p 00000000 08:01 1562993 /home/ubuntu/autotest/client/tmp/ubuntu_qrt_kernel_security/src/qa-regression-testing/scripts/kernel-security/nx/nx-test-rie
605. 559730ee4000-559730ee5000 r-xp 00001000 08:01 1562993 /home/ubuntu/autotest/client/tmp/ubuntu_qrt_kernel_security/src/qa-regression-testing/scripts/kernel-security/nx/nx-test-rie
606. 559730ee5000-559730ee6000 r--p 00002000 08:01 1562993 /home/ubuntu/autotest/client/tmp/ubuntu_qrt_kernel_security/src/qa-regression-testing/scripts/kernel-security/nx/nx-test-rie
607. 559730ee6000-559730ee7000 r--p 00002000 08:01 1562993 /home/ubuntu/autotest/client/tmp/ubuntu_qrt_kernel_security/src/qa-regression-testing/scripts/kernel-security/nx/nx-test-rie
608. 559730ee7000-559730ee8000 rw-p 00003000 08:01 1562993 /home/ubuntu/autotest/client/tmp/ubuntu_qrt_kernel_security/src/qa-regression-testing/scripts/kernel-security/nx/nx-test-rie
609. 559730ee8000-559730ef8000 rw-p 00000000 00:00 0
610. 5597313a5000-5597313c6000 rw-p 00000000 00:00 0 [heap]
611. 7fa96366b000-7fa963690000 r--p 00000000 08:01 45790 /usr/lib/x86_64-linux-gnu/libc-2.31.so
612. 7fa963690000-7fa963808000 r-xp 00025000 08:01 45790 /usr/lib/x86_64-linux-gnu/libc-2.31.so
613. 7fa963808000-7fa963852000 r--p 0019d000 08:01 45790 /usr/lib/x86_64-linux-gnu/libc-2.31.so
614. 7fa963852000-7fa963853000 ---p 001e7000 08:01 45790 /usr/lib/x86_64-linux-gnu/libc-2.31.so
615. 7fa963853000-7fa963856000 r--p 001e7000 08:01 45790 /usr/lib/x86_64-linux-gnu/libc-2.31.so
616. 7fa963856000-7fa963859000 rw-p 001ea000 08:01 45790 /usr/lib/x86_64-linux-gnu/libc-2.31.so
617. 7fa963859000-7fa96385f000 rw-p 00000000 00:00 0
618. 7fa963867000-7fa963868000 rwxp 00000000 00:00 0
619. 7fa963868000-7fa963869000 r--p 00000000 08:01 45783 /usr/lib/x86_64-linux-gnu/ld-2.31.so
620. 7fa963869000-7fa96388c000 r-xp 00001000 08:01 45783 /usr/lib/x86_64-linux-gnu/ld-2.31.so
621. 7fa96388c000-7fa963894000 r--p 00024000 08:01 45783 /usr/lib/x86_64-linux-gnu/ld-2.31.so
622. 7fa963894000-7fa963895000 rw-p 00000000 00:00 0
623. 7fa963895000-7fa963896000 r--p 0002c000 08:01 45783 /usr/lib/x86_64-linux-gnu/ld-2.31.so
624. 7fa963896000-7fa963897000 rw-p 0002d000 08:01 45783 /usr/lib/x86_64-linux-gnu/ld-2.31.so
625. 7fa963897000-7fa963898000 rw-p 00000000 00:00 0
626. 7ffe099cb000-7ffe099ec000 rwxp 00000000 00:00 0 [stack]
627. 7ffe099ee000-7ffe099f1000 r--p 00000000 00:00 0 [vvar]
628. 7ffe099f1000-7ffe099f2000 r-xp 00000000 00:00 0 [vdso]
629. ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
630. Attempting to execute function at 0x559730ee7010
631. If this program seg-faults, the region was enforced as non-executable...
632.
633.
634. ----------------------------------------------------------------------
635. Ran 125 tests in 73.445s
636.
637. FAILED (failures=1)

This failure is expected on the 5.4 oracle kernel, since it no longer assumes READ_IMPLIES_EXEC personality by default, as per this change:

https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-oracle/+git/focal/commit/?id=0e74912698dc008056759c501e7de3c551daedf1

So this test will fail from now on on f/oracle.

The test should treat f/oracle as it treats the 5.8 kernels - it should expect a return output of -11 not 0.

[Steve Beattie (sbeattie)]

Hey Khaled, was there a specific reason this patch was only applied to the linux-oracle deriviative kernel and not all of the 5.4 kernels?

[Steve Beattie (sbeattie)]

Regardless of the why, I've gone ahead and put into place a special case for linux-oracle/5.4 and newer kernels in qa-r-t in commit https://git.launchpad.net/qa-regression-testing/commit/?id=116f104915be98ccf9e27defcf8639d9dcfc2f89 .

I *think* I have the check right for the kernel variant. Please holler if it does not work right for that platform.

Thanks.

[Khaled El Mously (kmously)]

Hello @Steve.

That patch was called out by Ampere in its Altra documentation in https://github.com/AmpereComputing/ampere-centos-kernel/wiki as "fixing a soft lockup during heavy stress test", so I included in the arm64 bring-up.

It is possible that this is a general soft lockup and not one specific to Altra or arm64 (that seems likely to me) but I didn't look into it deeply.

[Khaled El Mously (kmously)]

And, thanks for quickly taking care of it !

[Khaled El Mously (kmously)]

@Steve, somehow this issue appears to be resolved -- in the amd64 testing only.

For some reason it still fails on arm64 oracle kernel. This is the latest Oracle kernel 1046:

http://10.246.72.9/5.4.0-1046.50-oracle/focal-linux-oracle-oracle-arm64-5.4.0-VM.Standard.A1.Flex-4c.8m-ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security/results/ubuntu_qrt_kernel_security.test-kernel-security.py/debug/ubuntu_qrt_kernel_security.test-kernel-security.py.DEBUG.html

Note that the same issue does not happen on amd64 with the 1046 kernel:

http://10.246.72.9/5.4.0-1046.50-oracle/focal-linux-oracle-oracle-amd64-5.4.0-VM.Standard2.1-ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security/results/ubuntu_qrt_kernel_security.test-kernel-security.py/debug/ubuntu_qrt_kernel_security.test-kernel-security.py.DEBUG.html

[Khaled El Mously (kmously)]

Will re-set the status back to "New" - I could create a new bug if you prefer.

[Steve Beattie (sbeattie)]

Nope, that's fine. The reason it fails on arm64 that despite the mentions of improved arm support, I didn't actually include arm64 as one of the arches for linux-oracle. I've fixed that now with https://git.launchpad.net/qa-regression-testing/commit/?id=3c077df7a17f1d3e9e82cd09b8e583916efb5394

Sorry for not catching that in the initial commit. Am closing but please re-open if this change is not sufficient.

Thanks!