Change in trace file leads to test timeout in ftrace tests on 5.15 ARM64

[Roxana Nicolescu]

[Impact]
On ARM64 systems, tests like subsystem-enable.tc, event-enable.tc,
event-pid.tc in linux/tools/testing/selftests/ftrace will take too
much time to run and consequently leads to test timeout.

The culprit is the `cat` command on the changing trace file.

[Fix]
* 25b9513872 selftests/ftrace: Stop tracing while reading the trace
  file by default

This patch can be cherry-picked into Jammy. I didn't see this issue in
our 5.4 kernels. And it's already landed in newer kernels.

[Test]
On an ARM64 testing node, apply this patch to the kernel tree and run
event-pid.tc test in linux/tools/testing/selftests/ftrace with:

  sudo ./ftracetest -vvv test.d/event/event-pid.tc

The test should finish within a few minutes.

[Regression Potential]
Change limited to testing tools, it should not have any actual impact
to kernel functions.

== Original Bug Report ==
First time seen this since ftrace refactor out from ubuntu_kernel_selftests to ubuntu-kselftests-ftrace

Issue found on ARM64 nodes with both generic / generic-64k kernel.

Partial test log:
 Running './ftracetest -vvv test.d/event/subsystem-enable.tc'
 === Ftrace unit tests ===
 ....
 [1] event tracing - enable/disable with event level files
 + [ 2 -eq 1 ]
 + [ -f set_event_pid ]
 + echo
 + [ -f set_ftrace_pid ]
 + echo
 + [ -f set_ftrace_notrace ]
 + echo
 + [ -f set_graph_function ]
 + echo
 + tee set_graph_function set_graph_notrace
 + [ -f stack_trace_filter ]
 + echo
 + [ -f kprobe_events ]
 + echo
 + [ -f uprobe_events ]
 + echo
 + [ -f synthetic_events ]
 + echo
 + [ -f snapshot ]
 + echo 0
 + clear_trace
 + echo
 + enable_tracing
 + echo 1
 + . /home/ubuntu/autotest/client/tmp/ubuntu_kselftests_ftrace/src/linux/tools/testing/selftests/ftrace/test.d/event/event-enable.tc
 + echo sched:sched_switch
 + yield
 + ping 127.0.0.1 -c 1
 PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.096 ms

 --- 127.0.0.1 ping statistics ---
 1 packets transmitted, 1 received, 0% packet loss, time 0ms
 rtt min/avg/max/mdev = 0.096/0.096/0.096/0.000 ms
 + cat trace
 + grep sched_switch
 + wc -l
 Timer expired (600 sec.), nuking pid 20982

From the code (tools/testing/selftests/ftrace/test.d/event/event-enable.tc) this test is trying to run:

    count=`cat trace | grep sched_switch | wc -l`

This command caused the timeout as `cat trace` seems to be never ending.

[Po-Hsu Lin (cypressyew)]

This can be found failing with:
  * J generic ARM64 - scobee-kernel
  * J generic ARM64 - wright-kernel
  * J generic-64k ARM64 - scobee-kernel
  * J-realtime ARM64 - kopter-kernel
  * J-lowlatency ARM64 - starmie-kernel
  * F-hwe-5.15 ppc64el - dryden
  * F-hwe-5.15 gneric-64k ARM64 - scobee-kernel
  * F-hwe-5.15 generic ARM64 - howzit-kernel

I am now testing a longer timeout on maas@obruchev

[Po-Hsu Lin (cypressyew)]

https://lists.ubuntu.com/archives/kernel-team/2023-November/147192.html

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/5.15.0-93.103 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux' to 'verification-done-jammy-linux'. If the problem still exists, change the tag 'verification-needed-jammy-linux' to 'verification-failed-jammy-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Po-Hsu Lin (cypressyew)]

Verified with Jammy 5.15.0-93.103 ARM64, this test is not failing anymore.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-intel-iotg/5.15.0-1047.53 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-intel-iotg' to 'verification-done-jammy-linux-intel-iotg'. If the problem still exists, change the tag 'verification-needed-jammy-linux-intel-iotg' to 'verification-failed-jammy-linux-intel-iotg'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Po-Hsu Lin (cypressyew)]

There is no ARM64 for intel iotg kernel.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-ibm-gt-fips/5.15.0-1052.55+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-ibm-gt-fips' to 'verification-done-jammy-linux-ibm-gt-fips'. If the problem still exists, change the tag 'verification-needed-jammy-linux-ibm-gt-fips' to 'verification-failed-jammy-linux-ibm-gt-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Po-Hsu Lin (cypressyew)]

There is no ARM64 for jammy-linux-ibm-gt-fips.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.15.0-94.104

---------------
linux (5.15.0-94.104) jammy; urgency=medium

  * jammy/linux: 5.15.0-94.104 -proposed tracker (LP: #2048777)

  * [SRU] Duplicate Device_dax ids Created and hence Probing is Failing.
    (LP: #2028158)
    - device-dax: Fix duplicate 'hmem' device registration

  * Add ODM driver f81604 usb-can (LP: #2045387)
    - can: usb: f81604: add Fintek F81604 support
    - [Config] updateconfigs for ODM drivers CONFIG_CAN_F81604

  * Add ODM driver gpio-m058ssan (LP: #2045386)
    - SAUCE: ODM: gpio: add M058SSAN gpio driver
    - [Config] updateconfigs for ODM drivers CONFIG_GPIO_M058SSAN

  * Add ODM driver rtc-pcf85263 (LP: #2045385)
    - SAUCE: ODM: rtc: add PCF85263 RTC driver
    - [Config] updateconfigs for ODM drivers CONFIG_RTC_DRV_PCF85263

  * AppArmor patch for mq-posix interface is missing in jammy (LP: #2045384)
    - SAUCE: (no-up) apparmor: reserve mediation classes
    - SAUCE: (no-up) apparmor: Add fine grained mediation of posix mqueues

  * Packaging resync (LP: #1786013)
    - [Packaging] update annotations scripts

linux (5.15.0-93.103) jammy; urgency=medium

  * jammy/linux: 5.15.0-93.103 -proposed tracker (LP: #2048330)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log
    - [Packaging] resync update-dkms-versions helper
    - [Packaging] remove helper scripts
    - [Packaging] update annotations scripts
    - debian/dkms-versions -- update from kernel-versions (main/2024.01.08)

  * Hotplugging SCSI disk in QEMU VM fails (LP: #2047382)
    - Revert "PCI: acpiphp: Reassign resources on bridge if necessary"

  * CVE-2023-6622
    - netfilter: nf_tables: bail out on mismatching dynset and set expressions

  * CVE-2024-0193
    - netfilter: nf_tables: skip set commit for deleted/destroyed sets

  * CVE-2023-6040
    - netfilter: nf_tables: Reject tables of unsupported family

  * Patches needed for AmpereOne (arm64) (LP: #2044192)
    - clocksource/arm_arch_timer: Add build-time guards for unhandled register
      accesses
    - clocksource/drivers/arm_arch_timer: Drop CNT*_TVAL read accessors
    - clocksource/drivers/arm_arch_timer: Extend write side of timer register
      accessors to u64
    - clocksource/drivers/arm_arch_timer: Move system register timer programming
      over to CVAL
    - clocksource/drivers/arm_arch_timer: Move drop _tval from erratum function
      names
    - clocksource/drivers/arm_arch_timer: Fix MMIO base address vs callback
      ordering issue
    - clocksource/drivers/arm_arch_timer: Move MMIO timer programming over to CVAL
    - clocksource/drivers/arm_arch_timer: Advertise 56bit timer to the core code
    - clocksource/drivers/arm_arch_timer: Work around broken CVAL implementations
    - clocksource/drivers/arm_arch_timer: Remove any trace of the TVAL programming
      interface
    - clocksource/drivers/arm_arch_timer: Drop unnecessary ISB on CVAL programming
    - clocksource/drivers/arm_arch_timer: Fix masking for high freq counters
    - clocksource/drivers/arch_arm_timer: Move workaround synchronisation around

  * Add quirk to disable i915 fastboot on B&R PC (LP: #2047630)
    - SAUCE: i915...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-bluefield/5.15.0-1035.37 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-bluefield' to 'verification-done-jammy-linux-bluefield'. If the problem still exists, change the tag 'verification-needed-jammy-linux-bluefield' to 'verification-failed-jammy-linux-bluefield'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-raspi/5.15.0-1046.49 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-raspi' to 'verification-done-jammy-linux-raspi'. If the problem still exists, change the tag 'verification-needed-jammy-linux-raspi' to 'verification-failed-jammy-linux-raspi'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure/5.15.0-1056.64 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-azure' to 'verification-done-jammy-linux-azure'. If the problem still exists, change the tag 'verification-needed-jammy-linux-azure' to 'verification-failed-jammy-linux-azure'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra/5.15.0-1022.22 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-tegra' to 'verification-done-jammy-linux-nvidia-tegra'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-tegra' to 'verification-failed-jammy-linux-nvidia-tegra'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-xilinx-zynqmp/5.15.0-1027.31 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-xilinx-zynqmp' to 'verification-done-jammy-linux-xilinx-zynqmp'. If the problem still exists, change the tag 'verification-needed-jammy-linux-xilinx-zynqmp' to 'verification-failed-jammy-linux-xilinx-zynqmp'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure-fips/5.15.0-1058.66+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-azure-fips' to 'verification-done-jammy-linux-azure-fips'. If the problem still exists, change the tag 'verification-needed-jammy-linux-azure-fips' to 'verification-failed-jammy-linux-azure-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-mtk/5.15.0-1030.34 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-done-jammy-linux-mtk'. If the problem still exists, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-failed-jammy-linux-mtk'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gcp-fips/5.15.0-1055.63+fips2 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-gcp-fips' to 'verification-done-jammy-linux-gcp-fips'. If the problem still exists, change the tag 'verification-needed-jammy-linux-gcp-fips' to 'verification-failed-jammy-linux-gcp-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Po-Hsu Lin (cypressyew)]

Hints removed. Closing this bug.

This issue cannot be found on J-azure, J-azure-fips, J-gcp-fips, J-mtk, J-nvidia-tegra, J-xilinx.

We don't run this test on J-raspi, J-bluefield.