failing ftrace self tests from 5.7+ onwards

[Colin Ian King]

Ftrace regression tests in tools/testing/selftests/ftrace are picking up regressions in 5.7+ - not sure if these are kernel regressions or bugs in ftrace selftests.

I'm seeing the following ftrace test regresions:

5.9-rc2 test failures with ARM64 in a 4GM VM + 8 CPUs

Test 12, "Generic dynamic event - selective clear (compatibility)"
Test 39, "Kprobe event with comm arguments"
Test 40, "Kprobe event string type argument"
Test 41, "Kprobe event symbol argument"
Test 43, "Kprobe dynamic event with arguments"
Test 44, "Kprobes event arguments with types"
Test 45, "Kprobe event user-memory access"
Test 47, "Kprobe dynamic event with function tracer"
Test 48, "Kprobe dynamic event - probing module"
Test 51, "Kretprobe dynamic event with arguments"

5.8.3, 5.7.17 test failures:
Test 45, "Kprobe event user-memory access"
Test 48, "Kprobe dynamic event - probing module"
Test 51, "Kretprobe dynamic event with arguments"

These tests were working fine when I tested with 5.4.60, 5.5.19, 5.6.19, so these are definite regressions

Attached is a spreadsheet of the full set of test results

[colin.king (colin.king-linux-kernel-bugs)]

Ftrace regression tests in tools/testing/selftests/ftrace are picking up regressions in 5.7+ - not sure if these are kernel regressions or bugs in ftrace selftests.

I'm seeing the following ftrace test regresions:

5.9-rc2 test failures with ARM64 in a 4GM VM + 8 CPUs

Test 12, "Generic dynamic event - selective clear (compatibility)"
Test 39, "Kprobe event with comm arguments"
Test 40, "Kprobe event string type argument"
Test 41, "Kprobe event symbol argument"
Test 43, "Kprobe dynamic event with arguments"
Test 44, "Kprobes event arguments with types"
Test 45, "Kprobe event user-memory access"
Test 47, "Kprobe dynamic event with function tracer"
Test 48, "Kprobe dynamic event - probing module"
Test 51, "Kretprobe dynamic event with arguments"

5.8.3, 5.7.17 test failures:
Test 45, "Kprobe event user-memory access"
Test 48, "Kprobe dynamic event - probing module"
Test 51, "Kretprobe dynamic event with arguments"

These tests were working fine when I tested with 5.4.60, 5.5.19, 5.6.19, so these are definite regressions

[colin.king (colin.king-linux-kernel-bugs)]

Urgh, typo. It should be "AMD64", in a QEMU VM.

[colin.king (colin.king-linux-kernel-bugs)]

I suspect I've got some config settings disabled. I wonder if the require config settings are described somewhere for these tests to work correctly?

[colin.king (colin.king-linux-kernel-bugs)]

Created attachment 292173
config-5.8.3-050803-generic

[colin.king (colin.king-linux-kernel-bugs)]

Created attachment 292175
config-5.9.0-rc2+

[colin.king (colin.king-linux-kernel-bugs)]

Is it possible to have a copy of a config where all these tests pass?

[colin.king (colin.king-linux-kernel-bugs)]

Make alldefconfig isn't producing passing results on amd64 for me either.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Kelsey Steele (kelsey-steele)]

ftrace failure in Focal 5.4.0-57.63 & 5.4.0-59.65:

Kprobe event user-memory access [FAIL]

Andrea found the problem was introduced into focal with this commit: 22555d066ef2c0a0e6ecba928896ec0c08b53ad7 ("selftests/ftrace: check for do_sys_openat2 in user-memory test")

Testing failed on:
    amd64: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/amd64/l/linux/20201209_050128_d377a@/log.gz
    arm64: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/arm64/l/linux/20201209_061300_0aada@/log.gz

[Stefan Bader (smb)]

According to bug 1907523 which has been duplicated against this one, the do_sys_openat2 change was made via stable to 5.4/Focal as well. Lowering the importance since failing self-tests is nothing that affects users.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Kleber Sacilotto de Souza (kleber-souza)]

ftrace testcase completed successfully with focal/linux 5.4.0-66.74 in -proposed:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/amd64/l/linux/20210205_135222_29af6@/log.gz

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.4.0-66.74

---------------
linux (5.4.0-66.74) focal; urgency=medium

  * focal/linux: 5.4.0-66.74 -proposed tracker (LP: #1913152)

  * Add support for selective build of special drivers (LP: #1912789)
    - [Packaging] Add support for ODM drivers
    - [Packaging] Turn on ODM support for amd64

  * Packaging resync (LP: #1786013)
    - update dkms package versions
    - update dkms package versions

  * Introduce the new NVIDIA 460-server series and update the 460 series
    (LP: #1913200)
    - [Config] dkms-versions -- drop NVIDIA 435 455 and 440-server
    - [Config] dkms-versions -- add the 460-server nvidia driver

  * Enable mute and micmute LED on HP EliteBook 850 G7 (LP: #1910102)
    - ALSA: hda/realtek: Enable mute and micmute LED on HP EliteBook 850 G7

  * SYNA30B4:00 06CB:CE09 Mouse on HP EliteBook 850 G7 not working at all
    (LP: #1908992)
    - HID: multitouch: Enable multi-input for Synaptics pointstick/touchpad device

  * HD Audio Device PCI ID for the Intel Cometlake-R platform (LP: #1912427)
    - SAUCE: ALSA: hda: Add Cometlake-R PCI ID

  * switch to an autogenerated nvidia series based core via dkms-versions
    (LP: #1912803)
    - [Packaging] nvidia -- use dkms-versions to define versions built
    - [Packaging] update-version-dkms -- maintain flags fields
    - [Config] dkms-versions -- add transitional/skip information for nvidia
      packages

  * udpgro.sh in net from ubuntu_kernel_selftests seems not reflecting sub-test
    result (LP: #1908499)
    - selftests: fix the return value for UDP GRO test

  * qede: Kubernetes Internal DNS Failure due to QL41xxx NIC not supporting IPIP
    tx csum offload (LP: #1909062)
    - qede: fix offload for IPIP tunnel packets

  * Use DCPD to control HP DreamColor panel (LP: #1911001)
    - SAUCE: drm/dp: Another HP DreamColor panel brigntness fix

  * kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor
    requests on cascadelake based kvm hosts (LP: #1911848)
    - KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set

  * Ubuntu 20.10 four needed fixes to 'Add driver for Mellanox Connect-IB
    adapters' (LP: #1905574)
    - net/mlx5: Fix a race when moving command interface to polling mode

  * Fix right sounds and mute/micmute LEDs for HP ZBook Fury 15/17 G7 Mobile
    Workstation (LP: #1910561)
    - ALSA: hda/realtek: fix right sounds and mute/micmute LEDs for HP machines

  * Ubuntu 20.04 - multicast counter is not increased in ip -s (LP: #1901842)
    - net/mlx5e: Fix multicast counter not up-to-date in "ip -s"

  * eeh-basic.sh in powerpc from ubuntu_kernel_selftests timeout with 5.4 P8 /
    P9 (LP: #1882503)
    - selftests/powerpc/eeh: disable kselftest timeout setting for eeh-basic

  * DMI entry syntax fix for Pegatron / ByteSpeed C15B (LP: #1910639)
    - Input: i8042 - unbreak Pegatron C15B

  * CVE-2020-29372
    - mm: check that mm is still valid in madvise()

  * update ENA driver, incl. new ethtool stats (LP: #1910291)
    - net: ena: Change WARN_ON expression in ena_del_napi_in_range()
    - net: ena: ethtool: convert stat_offset to 64 bit resolution
    - net: ena: eth...

[a.kawashiro (a.kawashiro-linux-kernel-bugs)]

I confirmed all ftrace related tests passed with `LOCALVERSION=-dev-master sudo make kselftest KCFLAGS="-W" CC="ccache clang-12" -j 12 TARGETS="ftrace"`. Commit is
0457e5153e0e8420134f60921349099e907264ca Merge: fc93310c67ff f3136c4ce7ac
Author: Linus Torvalds <email address hidden>
Date: Fri Feb 4 16:28:11 2022 -0800
.

[rostedt (rostedt-linux-kernel-bugs)]

(In reply to Akira Kawata from comment #7)
> I confirmed all ftrace related tests passed with `LOCALVERSION=-dev-master
> sudo make kselftest KCFLAGS="-W" CC="ccache clang-12" -j 12
> TARGETS="ftrace"`. Commit is
> 0457e5153e0e8420134f60921349099e907264ca Merge: fc93310c67ff f3136c4ce7ac
> Author: Linus Torvalds <email address hidden>
> Date: Fri Feb 4 16:28:11 2022 -0800
> .

Does this mean that we can close this bug?

-- Steve

[a.kawashiro (a.kawashiro-linux-kernel-bugs)]

Created attachment 300400
ftrace passed config

[a.kawashiro (a.kawashiro-linux-kernel-bugs)]

> Does this mean that we can close this bug?

I think so. I attached the succeded config.

[a.kawashiro (a.kawashiro-linux-kernel-bugs)]

Created attachment 300401
Succeeded test log

Result of test here.
`LOCALVERSION=-dev-master sudo make kselftest KCFLAGS="-W" CC="ccache clang-12" -j 12 TARGETS="ftrace" 2>& | tee kselftest.log

[a.kawashiro (a.kawashiro-linux-kernel-bugs)]

Ooops, I ran kselftest with old kernel. I will upload the correct log soon. Sorry.

[a.kawashiro (a.kawashiro-linux-kernel-bugs)]

Created attachment 300402
Result of `LOCALVERSION=-dev-master sudo make kselftest KCFLAGS="-W" CC="ccache gcc" -j 12 TARGETS="ftrace" 2>&1 | tee kselftest.log`

I attached the correct one. All tests passed also.