[Impact]
The perf tool use scandir() to iterate threads and sort in alphabetical order when synthesizing PERF_RECORD_ events.
If the process ID is 9999 and it has one thread (tid = 10000), the thread will be processed before the process.
It results in PERF_RECORD_FORK events that come before PERF_RECORD_MMAP2 events.
The callstack will have missing symbols for threads where `PERF_RECORD_FORK` events are processed before the `PERF_RECORD_MMAP2` event for the corresponding process.
[Fix]
Do not use alphasort when calling scandir()
363afa3aef24f5e08df6a539f5dc3aae4cddcc1a (perf synthetic-events: Don't sort the task scan result from /proc)
[Test Plan]
<test.lua>
function update_last_pid()
local file <close> = io.open("/proc/sys/kernel/ns_last_pid", "w")
file:write(9997)
end
update_last_pid()
os.execute("~/reproducer")
<reproducer.cpp>
#include <iostream>
#include <thread>
#include <unistd.h>
constexpr int kThreadNum = 10;
void thread_job() { sleep(30); }
int main(void) {
std::thread threads[kThreadNum];
std::cout << "Parent process with pid " << getpid() << std::endl;
for (int i = 0; i < kThreadNum; ++i) {
threads[i] = std::thread(thread_job);
}
for (int i = 0; i < kThreadNum; ++i) {
threads[i].join();
}
std::cout << "All threads have finished" << std::endl;
return 0;
}
The flow is to set /proc/sys/kernel/ns_last_pid first, which represents the last pid allocated in the current pid namespace.
The script (test.lua) sets ns_last_pid to 9997 and executes the reproducer (reproducer.cpp).
After the reproducer creates ten threads, we execute the perf command as follows: perf record -F 49 -e cpu-clock -a -g sleep 20.
Here is the result of command: perf report -f --tasks --mmaps -D | egrep -i 'perf_record_fork|perf_record_mmap' | grep 9999
Before applying the patch, the output of the perf command was as follows:
0 0 0x34910 [0x40]: PERF_RECORD_FORK(9999:10000):(9999:9999)
0 0 0x34990 [0x40]: PERF_RECORD_FORK(9999:10001):(9999:9999)
0 0 0x34a10 [0x40]: PERF_RECORD_FORK(9999:10002):(9999:9999)
0 0 0x34a90 [0x40]: PERF_RECORD_FORK(9999:10003):(9999:9999)
0 0 0x34b10 [0x40]: PERF_RECORD_FORK(9999:10004):(9999:9999)
0 0 0x34b90 [0x40]: PERF_RECORD_FORK(9999:10005):(9999:9999)
0 0 0x34c10 [0x40]: PERF_RECORD_FORK(9999:10006):(9999:9999)
0 0 0x34c90 [0x40]: PERF_RECORD_FORK(9999:10007):(9999:9999)
0 0 0x34d10 [0x40]: PERF_RECORD_FORK(9999:10008):(9999:9999)
0 0 0x34d90 [0x40]: PERF_RECORD_FORK(9999:10009):(9999:9999)
0 0 0x34e10 [0x40]: PERF_RECORD_FORK(9999:9999):(9998:9998)
0 0 0x34e90 [0x80]: PERF_RECORD_MMAP2 9999/9999: [0x555de0159000(0x1000) @ 0x1000 fd:00 2097758 0]: r-xp /root/reproducer
0 0 0x34f10 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd17f8000(0x7c000) @ 0xe000 fd:00 1055098 0]: r-xp /usr/lib/x86_64-linux-gnu/libm.so.6
0 0 0x34fa0 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd18f9000(0x195000) @ 0x28000 fd:00 1054988 0]: r-xp /usr/lib/x86_64-linux-gnu/libc.so.6
0 0 0x35030 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1afc000(0x17000) @ 0x3000 fd:00 1054745 0]: r-xp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
0 0 0x350c0 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1bb3000(0x110000) @ 0x9a000 fd:00 1068050 0]: r-xp /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
0 0 0x35158 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1d4e000(0x2a000) @ 0x2000 fd:00 1054949 0]: r-xp /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0 0 0x351f0 [0x70]: PERF_RECORD_MMAP2 9999/9999: [0x7ffc983bb000(0x2000) @ 0 00:00 0 0]: r-xp [vdso]
0 0 0x35260 [0x78]: PERF_RECORD_MMAP2 9999/9999: [0xffffffffff600000(0x1000) @ 0 00:00 0 0]: --xp [vsyscall]
After applying the patch, the output of the perf command is as follows:
0 0 0x30c28 [0x40]: PERF_RECORD_FORK(9999:9999):(9998:9998)
0 0 0x30ca8 [0x80]: PERF_RECORD_MMAP2 9999/9999: [0x5642c7635000(0x1000) @ 0x1000 fd:00 2097758 0]: r-xp /root/reproducer
0 0 0x30d28 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada03d000(0x7c000) @ 0xe000 fd:00 1055098 0]: r-xp /usr/lib/x86_64-linux-gnu/libm.so.6
0 0 0x30db8 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada13e000(0x195000) @ 0x28000 fd:00 1054988 0]: r-xp /usr/lib/x86_64-linux-gnu/libc.so.6
0 0 0x30e48 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada341000(0x17000) @ 0x3000 fd:00 1054745 0]: r-xp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
0 0 0x30ed8 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada3f8000(0x110000) @ 0x9a000 fd:00 1068050 0]: r-xp /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30
0 0 0x30f70 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada593000(0x2a000) @ 0x2000 fd:00 1054949 0]: r-xp /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0 0 0x31008 [0x70]: PERF_RECORD_MMAP2 9999/9999: [0x7ffcb2116000(0x2000) @ 0 00:00 0 0]: r-xp [vdso]
0 0 0x31078 [0x78]: PERF_RECORD_MMAP2 9999/9999: [0xffffffffff600000(0x1000) @ 0 00:00 0 0]: --xp [vsyscall]
0 0 0x310f0 [0x40]: PERF_RECORD_FORK(9999:10000):(9999:9999)
0 0 0x31170 [0x40]: PERF_RECORD_FORK(9999:10001):(9999:9999)
0 0 0x311f0 [0x40]: PERF_RECORD_FORK(9999:10002):(9999:9999)
0 0 0x31270 [0x40]: PERF_RECORD_FORK(9999:10003):(9999:9999)
0 0 0x312f0 [0x40]: PERF_RECORD_FORK(9999:10004):(9999:9999)
0 0 0x31370 [0x40]: PERF_RECORD_FORK(9999:10005):(9999:9999)
0 0 0x313f0 [0x40]: PERF_RECORD_FORK(9999:10006):(9999:9999)
0 0 0x31470 [0x40]: PERF_RECORD_FORK(9999:10007):(9999:9999)
0 0 0x314f0 [0x40]: PERF_RECORD_FORK(9999:10008):(9999:9999)
0 0 0x31570 [0x40]: PERF_RECORD_FORK(9999:10009):(9999:9999)
[Where problems could occur]
The fix will apply upstream commits, so the regression can be considered as low.
This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:
apport-collect 2008971
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.