Missing libunwind support in perf

[Mark Nelson]

Hi,

I'm trying to use perf's libunwind support in 13.04 with kernel 3.11.0-12-generic on amd64. Typically you can do this by using the "-g dwarf" flag, but it appears the perf command included in linux-tools-3.11.0-12-generic was not compiled with libunwind. This may be as simple as the build machine didn't have libunwind installed. Having dwarf support in perf with default Ubuntu kernels would be very useful as it provides a way to get profiling information when -fomitframepointer is used.

Relevant upstream code:

https://github.com/torvalds/linux/blob/v3.11/tools/perf/Makefile#L459
https://github.com/torvalds/linux/blob/v3.11/tools/perf/builtin-record.c#L758

Let me know if I can provide any other information.

[Julian Taylor (jtaylor)]

kernel 3.11 is a saucy kernel not raring
are you on saucy or using a backported kernel?

on saucy perf its '-g fp' instead of '-g dwarf'

[Julian Taylor (jtaylor)]

oh '-g fp' is without unwind support
right having dwarf back would be nice.

[Mark Nelson (mark-a-nelson)]

jtaylor:

Yep, this is saucy. Just upgraded in fact for the newer stock kernel and hoping that dwarf support would be there. :)

I work on Ceph at Inktank and there are times when we are debugging customer clusters with stock kernels where having unwind support in perf would be very useful. It would be especially nice if we could get it working before 14.04 lands as I expect a lot of people will be standardizing on that in the future.

[Tim Gardner (timg-tpi)]

Mark - as far as I can tell, libunwind _is_ used during the compilation of perf. I assume what you are doing worked prior to Saucy ?

[Julian Taylor (jtaylor)]

yes in raring perf -g dwarf worked. It stopped working for me in saucy.

[Mark Nelson (mark-a-nelson)]

Hi Tim,

Sorry to respond so late to this, hadn't seen your question earlier and have been bogged down with holidays. I've not seen -g dwarf work yet in a stock ubuntu kernel, but have seen it work in custom kernel compiles. Unfortunately I don't have any raring machines handy at the moment to verify jtaylor's observation that it worked in the stock distribution there.

[tacy (tacy-lee)]

Presics + trusty lts kernel (3.13.0-27)

~$ sudo perf record -g dwarf -a
dwarf: No such file or directory

strace output:

11268 10647 execve("/usr/local/sbin/dwarf", ["dwarf", "-a"], [/* 28 vars */]) = -1 ENOENT (No such file or directory)
11269 10647 execve("/usr/local/bin/dwarf", ["dwarf", "-a"], [/* 28 vars */]) = -1 ENOENT (No such file or directory)
11270 10647 execve("/usr/sbin/dwarf", ["dwarf", "-a"], [/* 28 vars */]) = -1 ENOENT (No such file or directory)
11271 10647 execve("/usr/bin/dwarf", ["dwarf", "-a"], [/* 28 vars */]) = -1 ENOENT (No such file or directory)
11272 10647 execve("/sbin/dwarf", ["dwarf", "-a"], [/* 28 vars */]) = -1 ENOENT (No such file or directory)
11273 10647 execve("/bin/dwarf", ["dwarf", "-a"], [/* 28 vars */]) = -1 ENOENT (No such file or directory)
11274 10647 dup(2) = 4

can't find dwarf ?

[Julian Taylor (jtaylor)]

in trusty the command line is: perf record --call-graph dwarf
compared to saucy dwarf support seems to work fine in trusty so this
issue can be closed.

[tacy (tacy-lee)]

Hi Julian

Below is my test.

tacy@tacy:~$ sudo perf record --call-graph dwarf -a
callchain: Unknown --call-graph option value: dwarf

 usage: perf record [<options>] [<command>]
    or: perf record [<options>] -- <command> [<options>]

        --call-graph <mode[,dump_size]>
                          setup and enables call-graph (stack chain/backtrace) recording: fp
tacy@tacy:~$ uname -a
Linux tacy 3.13.0-27-generic #50~precise1-Ubuntu SMP Fri May 16 20:47:56 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

tacy@tacy:~$ cd -
/home/tacy/linux-lts-trusty-3.13.0/tools/perf
tacy@tacy:~/linux-lts-trusty-3.13.0/tools/perf$ sudo ./perf record --call-graph dwarf -a
^C[ perf record: Woken up 63 times to write data ]
[ perf record: Captured and wrote 17.352 MB perf.data (~758127 samples) ]

tacy@tacy:~/linux-lts-trusty-3.13.0/tools/perf$ ./perf --version
perf version 3.13.11
tacy@tacy:~/linux-lts-trusty-3.13.0/tools/perf$ perf --version
perf version 3.13.11

Yes, After manually compiling can support, but default perf don't support.

[mbel (mbel)]

looks like this issue is rather tricky. adding libunwind to build dependecies actually gives nothing, there is still no dwarf support. Digging deeper I found that following test, which detects libunwind support for perf is not passed:

"gcc -o test-libunwind test-libunwind.c -lunwind -lunwind-x86 -lelf" fails with
/usr/lib/gcc/i686-linux-gnu/4.8/../../../i386-linux-gnu/libunwind-x86.so: undefined reference to `lzma_stream_buffer_decode'
/usr/lib/gcc/i686-linux-gnu/4.8/../../../i386-linux-gnu/libunwind-x86.so: undefined reference to `lzma_index_uncompressed_size'
/usr/lib/gcc/i686-linux-gnu/4.8/../../../i386-linux-gnu/libunwind-x86.so: undefined reference to `lzma_index_end'
/usr/lib/gcc/i686-linux-gnu/4.8/../../../i386-linux-gnu/libunwind-x86.so: undefined reference to `lzma_index_buffer_decode'
/usr/lib/gcc/i686-linux-gnu/4.8/../../../i386-linux-gnu/libunwind-x86.so: undefined reference to `lzma_stream_footer_decode'
/usr/lib/gcc/i686-linux-gnu/4.8/../../../i386-linux-gnu/libunwind-x86.so: undefined reference to `lzma_index_size'
collect2: error: ld returned 1 exit status

adding -llzma to the test above doesn't solve the problem. During further investigation I found that there are problem with libunwind8-dev - its libraries lacks dependant libs. I filed bug on libunwind and proposed solution for x86 platform. Link to the bug is https://bugs.launchpad.net/ubuntu/+source/libunwind/+bug/1336912

[Tim Gardner (timg-tpi)]

This appears to be fixed in Wily linux 4.2-7.7

[Julian Taylor (jtaylor)]

can the fix be applied at least to 14.04 lts kernels as well?
It should just be adding a build depends as I can build a correct perf just fine on a stock 14.04 with the right dependencies (libbfd-dev I think is the missing one)

[Tim Gardner (timg-tpi)]

The Wily kernel will be released as an LTS HWE kernel in 14.04.4

[Julian Taylor (jtaylor)]

just tested the 4.2 kernel, the issue has not been fixed
it can also be seen in the buildlogs

 52311 config/Makefile:527: No bfd.h/libbfd found, please install binutils-dev[el]/zlib-static/libiberty-dev to gain sym bol demangling
 52312 config/Makefile:556: No liblzma found, disables xz kernel module decompression, please install xz-devel/liblzma-d ev

[Alex Cornejo (acornejoc)]

This seems like a packaging problem, since in the debian world (not ubuntu) perf happily supports dwarf symbols.

What is the current status? I would be happy to contribute a patch to fix it, but don't want to redo anyone's work.

Also, is there a reason that the fix could not get backported to older distros? It doesn't require changing the kernel, just the user space perf binaries with an update.

Anyone that doesn't have perf tools installed won't notice the change. And everyone that has perf tools installed (and therefore uses it) will be very happy; I honestly can't see a reason NOT to do it. perf tools is an essential component for performance tuning server applications.

[Tim Gardner (timg-tpi)]

Alex - please send patches to <email address hidden>. I'm pretty sure nobody is looking at this particular problem right now.

[Julian Taylor (jtaylor)]

with current linux git master you also need this so libunwind is detected on trusty (probably due to the --as-needed linker setting):

--- a/tools/perf/config/Makefile
+++ b/tools/perf/config/Makefile
@@ -29,7 +29,7 @@ ifeq ($(ARCH),x86)
   ifeq (${IS_64_BIT}, 1)
     CFLAGS += -DHAVE_ARCH_X86_64_SUPPORT
     ARCH_INCLUDE = ../../arch/x86/lib/memcpy_64.S ../../arch/x86/lib/memset_64.S
- LIBUNWIND_LIBS = -lunwind -lunwind-x86_64
+ LIBUNWIND_LIBS = -lunwind -lunwind-x86_64 -llzma
     $(call detected,CONFIG_X86_64)
   else
     LIBUNWIND_LIBS = -lunwind-x86 -llzma -lunwind

[Jerry Quinn (jlquinn)]

Bug 1396654 c++ demangle support missing reported as a duplicate of this one.

Is anyone working on it? I'm having difficulty profiling c++ on ubuntu.

[Benjamin.M (benjaminm)]

This problem affects me too (on Xenial):

> uname -a
Linux Nyx 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

> perf --version
perf version 4.4.6

Adding --call-graph dwarf doesn't change that the symbols aren't demangled.

[Dmitry Rekman (rcl)]

Does this problem not affect Canonical developers working on, say, Mir (which is written in C++)? It is way more painful to profile applications on stock Ubuntu.

This can mean that Mir developers are not interested in performance of their server, which I hope is a wrong conclusion to draw.

[Kamal Mostafa (kamalmostafa)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Julian Taylor (jtaylor)]

if you check the buildlog you see its not fixed, its still missing libbfd-dev, have you applied the posted patch?

[Andy Whitcroft (apw)]

For clarity the patch applied to Xenial was the moral equivalent to that in comment#17. This does enable libunwind in the perf build, but we also need the libbfd support for it to be useful.

[Andy Whitcroft (apw)]

I have spun a new patch for the additional Build-Depends: and will submit it once testing is complete.

From an SRU testing standpoint the application of this patch is benign so if we do not have a respin (when we should consider including this additional patch) we can consider this "verification-done" for the purposes of letting this bit of the fix out. We can then reopen the bug for the next piece.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-23.41

---------------
linux (4.4.0-23.41) xenial; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1582431

  * zfs: disable module checks for zfs when cross-compiling (LP: #1581127)
    - [Packaging] disable zfs module checks when cross-compiling

  * Xenial update to v4.4.10 stable release (LP: #1580754)
    - Revert "UBUNTU: SAUCE: (no-up) ACPICA: Dispatcher: Update thread ID for
      recursive method calls"
    - Revert "UBUNTU: SAUCE: nbd: ratelimit error msgs after socket close"
    - Revert: "powerpc/tm: Check for already reclaimed tasks"
    - RDMA/iw_cxgb4: Fix bar2 virt addr calculation for T4 chips
    - ipvs: handle ip_vs_fill_iph_skb_off failure
    - ipvs: correct initial offset of Call-ID header search in SIP persistence
      engine
    - ipvs: drop first packet to redirect conntrack
    - mfd: intel-lpss: Remove clock tree on error path
    - nbd: ratelimit error msgs after socket close
    - ata: ahci_xgene: dereferencing uninitialized pointer in probe
    - mwifiex: fix corner case association failure
    - CNS3xxx: Fix PCI cns3xxx_write_config()
    - clk-divider: make sure read-only dividers do not write to their register
    - soc: rockchip: power-domain: fix err handle while probing
    - clk: rockchip: free memory in error cases when registering clock branches
    - clk: meson: Fix meson_clk_register_clks() signature type mismatch
    - clk: qcom: msm8960: fix ce3_core clk enable register
    - clk: versatile: sp810: support reentrance
    - clk: qcom: msm8960: Fix ce3_src register offset
    - lpfc: fix misleading indentation
    - ath9k: ar5008_hw_cmn_spur_mitigate: add missing mask_m & mask_p
      initialisation
    - mac80211: fix statistics leak if dev_alloc_name() fails
    - tracing: Don't display trigger file for events that can't be enabled
    - MD: make bio mergeable
    - Minimal fix-up of bad hashing behavior of hash_64()
    - mm, cma: prevent nr_isolated_* counters from going negative
    - mm/zswap: provide unique zpool name
    - ARM: EXYNOS: Properly skip unitialized parent clock in power domain on
    - ARM: SoCFPGA: Fix secondary CPU startup in thumb2 kernel
    - xen: Fix page <-> pfn conversion on 32 bit systems
    - xen/balloon: Fix crash when ballooning on x86 32 bit PAE
    - xen/evtchn: fix ring resize when binding new events
    - HID: wacom: Add support for DTK-1651
    - HID: Fix boot delay for Creative SB Omni Surround 5.1 with quirk
    - Input: zforce_ts - fix dual touch recognition
    - proc: prevent accessing /proc/<PID>/environ until it's ready
    - mm: update min_free_kbytes from khugepaged after core initialization
    - batman-adv: fix DAT candidate selection (must use vid)
    - batman-adv: Check skb size before using encapsulated ETH+VLAN header
    - batman-adv: Fix broadcast/ogm queue limit on a removed interface
    - batman-adv: Reduce refcnt of removed router when updating route
    - writeback: Fix performance regression in wb_over_bg_thresh()
    - MAINTAINERS: Remove asterisk from EFI directory names
    - x86/tsc: Read all ratio bits from MSR_PLATFORM_INFO
    - ARM: cpuidle: Pass on arm_cpuidle_s...

[Shikhar Bhardwaj (shikhar-c)]

Hi,

When will this bugfix be released for Xenial?

Thanks

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-24.43

---------------
linux (4.4.0-24.43) xenial; urgency=low

  [ Kamal Mostafa ]

  * CVE-2016-1583 (LP: #1588871)
    - ecryptfs: fix handling of directory opening
    - SAUCE: proc: prevent stacking filesystems on top
    - SAUCE: ecryptfs: forbid opening files without mmap handler
    - SAUCE: sched: panic on corrupted stack end

  * arm64: statically link rtc-efi (LP: #1583738)
    - [Config] Link rtc-efi statically on arm64

 -- Kamal Mostafa <email address hidden> Fri, 03 Jun 2016 10:02:16 -0700

[Reimar Döffinger (reimar-doeffinger)]

This either was never fixed or broke again, perf links neither against libunwind or libdw:
$ ldd /usr/lib/linux-tools/4.8.0-37-generic/perf
        linux-vdso.so.1 => (0x00007ffee7766000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efec8bb9000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007efec89b1000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007efec86a8000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007efec84a4000)
        libelf.so.1 => /usr/lib/x86_64-linux-gnu/libelf.so.1 (0x00007efec828c000)
        libaudit.so.1 => /lib/x86_64-linux-gnu/libaudit.so.1 (0x00007efec8062000)
        libslang.so.2 => /lib/x86_64-linux-gnu/libslang.so.2 (0x00007efec7cd7000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007efec7abd000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007efec789a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efec74d3000)
        /lib64/ld-linux-x86-64.so.2 (0x0000564e0db72000)
        libcap-ng.so.0 => /lib/x86_64-linux-gnu/libcap-ng.so.0 (0x00007efec72ce000)

[Jon Gjengset (jonhoo)]

In fact, on Yakkety with 4.8.0-34, the results of running perf --call-graph are even more bizarre:

$ perf record --call-graph dwarf sleep 1
Error:
The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (cycles).
/bin/dmesg may provide additional information.
No CONFIG_PERF_EVENTS=y kernel support configured?
$ grep -i config_perf_events /boot/config-$(uname -r)
CONFIG_PERF_EVENTS=y
CONFIG_PERF_EVENTS_INTEL_UNCORE=y
CONFIG_PERF_EVENTS_INTEL_RAPL=m
CONFIG_PERF_EVENTS_INTEL_CSTATE=m
# CONFIG_PERF_EVENTS_AMD_POWER is not set

[Jon Gjengset (jonhoo)]

For what it's worth, recompiling `perf` from source fixes the problem.

[Mark Sonnabaum (msonnabaum)]

I also just ran into this using hwe on xenial 16.04.2.

perf in the stock 16.04.2 is linked to libdw, libunwind, libbfd, and libbz2, all of which are missing after switching to hwe with the 4.8 kernel.