6 sub-tests from ubuntu_kselftests_ftrace failed with ARM64k

Bug #2029389 reported by Po-Hsu Lin
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ubuntu-kernel-tests
New
Undecided
Unassigned

Bug Description

Before running the ftrace sub-tests one-by-one with -vvv flag (https://git.launchpad.net/~canonical-kernel-team/+git/autotest-client-tests/commit/?id=ced9e7378a360a0d2ef8ffd9c34a22e2b47418d5) the ARM64k instance will hang with step 16 (bug 1959424) or 19 (bug 2023381), which is the following event tracing tests respectively:
* [16] event tracing - enable/disable with event level files
* [19] event tracing - enable/disable with subsystem level files
The whole test will be interrupted on this point.

By running tests one-by-one with -vvv flag, it will have a chance to run though all sub-tests, with following failures:
(step 16)
* ftrace:test.d--event--event-enable.tc FAIL
* ftrace:test.d--event--event-pid.tc FAIL
(step 19)
* ftrace:test.d--event--subsystem-enable.tc FAIL
* ftrace:test.d--ftrace--fgraph-filter-stack.tc FAIL
* ftrace:test.d--ftrace--fgraph-filter.tc FAIL
* ftrace:test.d--ftrace--func_traceonoff_triggers.tc FAIL

So if it hung with step 16, you will see 6 failures reported. But if it hung with step 19 before the test improvement, you will see just 4 failures.

This is not a regression but some new issues revealed with this test case improvement.

They all timeout with the 600 seconds threshold. I did test these manually and it seems the culprit is the never ending `cat trace | grep sched_switch | wc -l` command for /sys/kernel/tracing/trace. Not sure how long it will take but it didn't finish even after a few hours.

I will paste the ending part of each tests below, you will find a more complete output in the attachments:
* ftrace:test.d--event--event-enable.tc
 [stderr] + . /home/ubuntu/autotest/client/tmp/ubuntu_kselftests_ftrace/src/linux/tools/testing/selftests/ftrace/test.d/event/event-enable.tc
 [stderr] + echo sched:sched_switch
 [stderr] + yield
 [stderr] + ping 127.0.0.1 -c 1
 [stdout] PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
 [stdout] 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.058 ms
 [stdout]
 [stdout] --- 127.0.0.1 ping statistics ---
 [stdout] 1 packets transmitted, 1 received, 0% packet loss, time 0ms
 [stdout] rtt min/avg/max/mdev = 0.058/0.058/0.058/0.000 ms
 [stderr] + cat trace
 [stderr] + grep sched_switch
 [stderr] + wc -l
 Timer expired (600 sec.), nuking pid 23930

* ftrace:test.d--event--event-pid.tc
 [stderr] + . /home/ubuntu/autotest/client/tmp/ubuntu_kselftests_ftrace/src/linux/tools/testing/selftests/ftrace/test.d/event/event-pid.tc
 [stderr] + echo 0
 [stderr] + echo 1
 [stderr] + yield
 [stderr] + ping 127.0.0.1 -c 1
 [stdout] PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
 [stdout] 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.055 ms
 [stdout]
 [stdout] --- 127.0.0.1 ping statistics ---
 [stdout] 1 packets transmitted, 1 received, 0% packet loss, time 0ms
 [stdout] rtt min/avg/max/mdev = 0.055/0.055/0.055/0.000 ms
 [stderr] + cat trace
 [stderr] + grep sched_switch
 [stderr] + wc -l
 Timer expired (600 sec.), nuking pid 24304

* ftrace:test.d--event--subsystem-enable.tc
 [stderr] + . /home/ubuntu/autotest/client/tmp/ubuntu_kselftests_ftrace/src/linux/tools/testing/selftests/ftrace/test.d/event/subsystem-enable.tc
 [stderr] + echo sched:*
 [stderr] + yield
 [stderr] + ping 127.0.0.1 -c 1
 [stdout] PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
 [stdout] 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.064 ms
 [stdout]
 [stdout] --- 127.0.0.1 ping statistics ---
 [stdout] 1 packets transmitted, 1 received, 0% packet loss, time 0ms
 [stdout] rtt min/avg/max/mdev = 0.064/0.064/0.064/0.000 ms
 [stderr] + cat trace
 [stderr] + grep -v ^#
 [stderr] + awk { print $5 }
 [stderr] + sort -u
 [stderr] + wc -l
 Timer expired (600 sec.), nuking pid 24412

* ftrace:test.d--ftrace--fgraph-filter-stack.tc
 [stderr] + . /home/ubuntu/autotest/client/tmp/ubuntu_kselftests_ftrace/src/linux/tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter-stack.tc
 [stderr] + disable_tracing
 [stderr] + echo 0
 [stderr] + clear_trace
 [stderr] + echo
 [stderr] + echo schedule
 [stderr] + echo function_graph
 [stdout] Now testing with stack tracer
 [stderr] + echo Now testing with stack tracer
 [stderr] + echo 1
 [stderr] + disable_tracing
 [stderr] + echo 0
 [stderr] + clear_trace
 [stderr] + echo
 [stderr] + enable_tracing
 [stderr] + echo 1
 [stderr] + sleep 1
 [stderr] + cat trace
 [stderr] + grep ()
 [stderr] + grep -v schedule
 [stderr] + wc -l
 [stderr] + count=0
 [stderr] + [ 0 -ne 0 ]
 [stderr] + cat trace
 [stderr] + grep schedule()
 [stderr] + wc -l
 [stderr] + count=46070
 [stderr] + [ 46070 -eq 0 ]
 [stderr] + echo 0
 [stderr] + clear_trace
 [stderr] + echo
 [stderr] + sleep 1
 [stderr] + cat trace
 [stderr] + grep ()
 [stderr] + grep -v schedule
 [stderr] + wc -l
 Timer expired (600 sec.), nuking pid 24930

* ftrace:test.d--ftrace--fgraph-filter.tc
 [stderr] + . /home/ubuntu/autotest/client/tmp/ubuntu_kselftests_ftrace/src/linux/tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc
 [stderr] + disable_tracing
 [stderr] + echo 0
 [stderr] + clear_trace
 [stderr] + echo
 [stderr] + echo schedule
 [stderr] + echo function_graph
 [stderr] + enable_tracing
 [stderr] + echo 1
 [stderr] + sleep 1
 [stderr] + cat trace
 [stderr] + grep ()
 [stderr] + grep -v schedule
 [stderr] + wc -l
 [stderr] + count=0
 [stderr] + [ 0 -ne 0 ]
 [stderr] + cat trace
 [stderr] + grep schedule()
 [stderr] + wc -l
 Timer expired (600 sec.), nuking pid 25115

* ftrace:test.d--ftrace--func_traceonoff_triggers.tc
 [stderr] + . /home/ubuntu/autotest/client/tmp/ubuntu_kselftests_ftrace/src/linux/tools/testing/selftests/ftrace/test.d/ftrace/func_traceonoff_triggers.tc
 [stdout] Testing function probes with enabling disabling tracing:
 [stdout] ** DISABLE TRACING
 [stderr] + SLEEP_TIME=.1
 [stderr] + echo Testing function probes with enabling disabling tracing:
 [stderr] + echo ** DISABLE TRACING
 [stderr] + disable_tracing
 [stderr] + echo 0
 [stderr] + clear_trace
 [stderr] + echo
 [stderr] + cnt_trace
 [stderr] + grep -v ^# trace
 [stderr] + wc -l
 [stderr] + cnt=0
 [stderr] + [ 0 -ne 0 ]
 [stderr] + echo ** ENABLE EVENTS
 [stdout] ** ENABLE EVENTS
 [stdout] ** ENABLE TRACING
 [stderr] + echo 1
 [stderr] + echo ** ENABLE TRACING
 [stderr] + enable_tracing
 [stderr] + echo 1
 [stderr] + cnt_trace
 [stderr] + grep -v ^# trace
 [stderr] + wc -l
 Timer expired (600 sec.), nuking pid 27469

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Complete test log for ftrace:test.d--event--event-enable.tc

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Complete test log for ftrace:test.d--event--event-pid.tc

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Complete test log for ftrace:test.d--event--subsystem-enable.tc

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Complete test log for ftrace:test.d--ftrace--fgraph-filter-stack.tc

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Complete test log for ftrace:test.d--ftrace--fgraph-filter.tc

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Complete test log for ftrace:test.d--ftrace--func_traceonoff_triggers.tc

Po-Hsu Lin (cypressyew)
description: updated
Revision history for this message
Jacob Martin (jacobmartin) wrote :

Noticing the following failures with j:linux-nvidia-tegra 5.15.0-1016.16 on kopter-kernel with 4k pages:
* ftrace:test.d--event--event-enable.tc
* ftrace:test.d--event--event-pid.tc
* ftrace:test.d--event--subsystem-enable.tc
* ftrace:test.d--ftrace--fgraph-filter-stack.tc
* ftrace:test.d--ftrace--fgraph-filter.tc

All failures appear to have the same output as initially reported.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.