perf report can't annotate kernel and dbgsym package has wrong addresses

[Ali]

Impact:

Under some condition, 'perf report' is unable to show the assembly code of a kernel function (perf report -> select a kernel function -> annotate).

This is not an issue of the kernel itself or the perf tool, instead it's your environment that is missing the objdump binary (and perf, unfortunately, doesn't complain about it).

Make sure binutils is not installed, or move away the objdump binary:

# mv /usr/bin/objdump /usr/bin/objdump.foo

Run a perf session and write down all events:

# perf record -a -- sleep 10

# perf report

selct a kernel function, press 'Annotate', a blank screen will appears (instead of the assembly code).

Now put the objdump binary back (or install binutils):

# mv /usr/bin/objdump.foo /usr/bin/objdump

and run again perf report:

# perf report

select a kernel function, annotate, the disassembly will show up.

Fix:

Make binutils a Depends in SRCPKGNAME-tools-PKGVER-ABINUM - see the attached patch.

How to test:

Install a patched linux-tools-PKGVER-ABINUM.deb package: it will require binutils.

Regression:

We are adding a new dependency on a package, so code wise there's no regression potential - on the other hand, image creation, or development environment will experience a slight increase in size (1.8MB on x86-64 and 2.1M on arm64) if they didn't install binutils already.

--

Running zesty 4.10.0-13-generic kernel and running:
$ perf record

followed by

$ perf report

results in output that can't be annotated (even when run as root). The following message appears:
Couldn't annotate do_io_submit:
No vmlinux file with build id 81ba79d482fa9e3ea58486de8f119b27fe6db55e
was found in the path.

Note that annotation using /proc/kcore requires CAP_SYS_RAWIO capability.

Please use:

perf buildid-cache -vu vmlinux

Note this is being run as root and /proc/kcore is accessible with dd from the same shell:
$ dd if=/proc/kcore bs=8 count=16 | hexdump
0000000 457f 464c 0102 0001 0000 0000 0000 0000
0000010 0004 00b7 0001 0000 0000 0000 0000 0000
0000020 0040 0000 0000 0000 0000 0000 0000 0000
0000030 0000 0000 0040 0038 0003 0000 0000 0000
0000040 0004 0000 0000 0000 00e8 0000 0000 0000

Additionally, installing the linux-image-4.10.0-13-generic-dbgsym actually hurts the situation because the symbol addresses in the dbgsym don't match the kernel addresses.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1675949

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Seth Forshee (sforshee)]

I tried this with 4.10.0-14-generic and could not reproduce your issue. Please try updating and see if you still experience this issue.

Run-time addresses are expected to be unpredictable due to ASLR.

[Ali (asaidi)]

Hi Seth,

Thanks for taking a look. After upgrading to 4.10.0-14-generic I'm still seeing the same issue. The build-id has changed, but the symptoms are the same:

[Seth Forshee (sforshee)]

I'm going to need more information. Can you give me the output of these commands?

$ uname -a
$ dpkg -S $(which perf)
$ dpkg --get-selections | grep linux-tools

Can you proived the exact commands you're running to produce this behavior? For me, running "perf record" with no additional arguments only generates help text.

Is this only with one specific way of invoking "perf record"? Have you tried sampling other events to see if you get the same problem? For example, does this produce a similar error?

# perf record -F 99 -g -- sleep 10

[Ali (asaidi)]

root@host1:~/fio# uname -a
Linux host1 4.10.0-14-generic #16-Ubuntu SMP Fri Mar 17 15:20:17 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux

root@host1:~/fio# dpkg -S $(which perf)
linux-tools-common: /usr/bin/perf

root@host1:~/fio# dpkg --get-selections | grep linux-tools
linux-tools-4.10.0-11 install
linux-tools-4.10.0-13 install
linux-tools-4.10.0-13-generic install
linux-tools-4.10.0-14 install
linux-tools-4.10.0-14-generic install
linux-tools-4.10.0-9 install
linux-tools-4.10.0-9-generic install
linux-tools-common install
linux-tools-generic install

Running perf like: perf record -F 99 -g -- sleep 10 also has the same issue

I've been running: perf record -a -- sleep 10

Annotation works on kernel modules and user code, but non-module kernel code has the issue where annotation doesn't work on it.

After running record, run perf report, select one of the kernel symbols, hit enter, select Annotate <symbol> and hit enter again.

Thanks,
Ali

[Seth Forshee (sforshee)]

Everything continues to work fine for me, but I see now that you are on aarch64.

I don't have any 64-bit ARM platforms in my posession, but I should be able to get access to something. I'll see what I can find out.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Yangzheng Bai (zoy)]

Have we fixed this issue for aarch64?

We experienced the same thing with 4.10, 4.13 and 4.15 kernels on xenial, artful and bionic

uname -a
Linux 4.13.0-46-generic #51-Ubuntu SMP Tue Jun 12 12:39:10 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

dpkg --get-selections | grep linux-tools
linux-tools-4.13.0-46 install
linux-tools-4.13.0-46-generic install
linux-tools-common install
linux-tools-generic install

[Seth Forshee (sforshee)]

This issue fell off my radar, and I never followed up on it. I'll see if we can get someone to take a look.

[Paolo Pisati (p-pisati)]

On a fresh Xenial/arm64 installation:

$ uname -r
4.4.0-141-generic
$ uname -m
aarch64

$ dpkg -S $(which perf)
linux-tools-common: /usr/bin/perf

$ dpkg --get-selections | grep linux-tools
linux-tools-4.4.0-141 install
linux-tools-4.4.0-141-generic install
linux-tools-common install
linux-tools-generic install

$ sudo perf record -a -- sleep 10
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.131 MB perf.data (626 samples) ]

followed by:

$ sudo perf report

i see the captured samples:

  42.22% swapper [kernel.kallsyms] [k] arch_cpu_idle ▒
   5.10% tmux [kernel.kallsyms] [k] copy_page ◆
   1.37% tmux libc-2.23.so [.] 0x000000000006e680 ▒
   1.37% tmux tmux [.] 0x0000000000028684 ▒
   1.37% tmux libc-2.23.so [.] strcmp ▒
   1.35% swapper [kernel.kallsyms] [k] tick_nohz_idle_enter ▒
   1.23% tmux tmux [.] 0x00000000000355dc ▒
   1.05% sh [kernel.kallsyms] [k] handle_mm_fault ▒
   1.01% tmux [kernel.kallsyms] [k] filemap_map_pages ▒
   1.01% swapper [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore ▒
   0.96% tmux libc-2.23.so [.] free

but when i try to inspect a kernel function:

Annotate filemap_map_pages ◆
Zoom into sh(5304) thread ▒
Zoom into [kernel.kallsyms] DSO ▒Annotate filemap_map_pages ◆
Zoom into sh(5304) thread ▒
Zoom into [kernel.kallsyms] DSO ...