tools/perf: Symbol loading problems in perf report

Bug #608775 reported by Dave Martin
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Low
Andy Whitcroft
Maverick
Invalid
Low
Unassigned

Bug Description

Affected: linux 2.6.35-10.15
Architecture: all

 * When perf report loads symbols from an image that is in the
current directory or a subdirectory of the current directory, perf
report abbreviates the current directory part of the path to "./".
This happens independently of the directory perf record was run in.

 * For Ubuntu, perf report searches for debug symbols in
/usr/lib/debug<path>. This works when <path> is absolute, but fails
to work if the path of an image has been abbreviated, as above.

So, for example:

If I have the debug symbols for bash in /usr/lib/debug/bin/bash:

/bin$ perf record bash -c 'while :; do :; done& sleep 5; kill $!'
/bin$ perf report

...then perf report fails to find the debug symbols for bash (because
it searches in /usr/lib/debug./bash)

A simple workaround is to change to any directory which is not a
direct parent of the affected images before running perf report:

/bin$ cd /tmp
/tmp$ perf report -i /bin/perf.data

The affected code is at:

tools/perf/util/map.c:map__new():55

tools/perf/util/symbol.c:dso__load():1334

One possible fix is simply to remove the current directory
abbreviation for names of loaded images - this makes some sense,
because the absolute path of each image is needed to form the correct
path under /usr/lib/debug for the separated debug image (possible patch attached).
 Otherwise, the absolute path would need to be tracked separately, in
addition to the abbreviated path used for displaying in the report.

Revision history for this message
Dave Martin (dave-martin-arm) wrote :
Revision history for this message
Dave Martin (dave-martin-arm) wrote :

Exchanged mails with Arnaldo Carvalho de Melo, who is involved in upstream maintenance of the perf tools:

Arnaldo Carvalho de Melo <acme at redhat.com> wrote:

Em Thu, Jul 22, 2010 at 09:07:04AM +0100, Dave Martin escreveu:

[...]

Your analysis seems correct, I guess we should store the absolute path
and only at report time strip the current directory, will look into
that.

[...]

tags: added: patch
tags: added: kj-triage
Revision history for this message
Dave Martin (dave-martin-arm) wrote :

Some patches merged into Arnaldo's perf/core branch at git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux-2.6.git:

perf report: Don't abbreviate file paths relative to the cwd
http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commitdiff;h=361d13462585474267a0c41e956f1a1c19a93f17

perf tools: Remove unneeded code for tracking the cwd in perf sessions
http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commitdiff;h=88ca895dd4e0e64ebd942adb7925fa60ca5b2a98

There is still a problem with the build-id based caching done in new versions of the tools.

This can be disabled by passing --no-buildid-cache to perf record (bit if there are stale files in ~/.debug you may still need to delete them in order for perf report to work properly).

I'm currently looking into a more complete fix for this.

Revision history for this message
Dave Martin (dave-martin-arm) wrote :

Additional set of patches which allows symbol search to bypass the buildid cache if there are better debug images available elsewhere.

perf tools: Factor out buildid reading and make it implicit in dso__load
http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commitdiff;h=21916c380d93ab59d6d07ee198fb31c8f1338e26

perf tools: remove extra build-id check factored into dso__load
http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commitdiff;h=8b1389ef93b36621c6acdeb623bd85aee3c405c9

perf symbols: Improve debug image search when loading symbols
http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commitdiff;h=6da80ce8c43ddda153208cbb46b75290cf566fac

The caching mechanism needs updating to put the debug images into the cache also, but these patches implement an interim solution which appears to work for Ubuntu.

Revision history for this message
Dave Martin (dave-martin-arm) wrote :

The above patches are now merged into the master branch of linux-2.6-tip:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git

Revision history for this message
Dave Martin (dave-martin-arm) wrote :

... and an aditional bug that needs fixing in order for perf report to produce sensible output is here:

http://lkml.org/lkml/2010/8/3/198 (perf: symbol offset breakage with separated debug)

Revision history for this message
Dave Martin (dave-martin-arm) wrote :

Update: most of the patches are now in linux-2.6-tip:

   * http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=361d13462585474267a0c41e956f1a1c19a93f17 (perf report: Don't abbreviate file paths relative to the cwd)
   * http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=88ca895dd4e0e64ebd942adb7925fa60ca5b2a98 (perf tools: Remove unneeded code for tracking the cwd in perf sessions)
   * http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=21916c380d93ab59d6d07ee198fb31c8f1338e26 (perf tools: Factor out buildid reading and make it implicit in dso__load)
   * http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=8b1389ef93b36621c6acdeb623bd85aee3c405c9 (perf tools: remove extra build-id check factored into dso__load)
   * http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=6da80ce8c43ddda153208cbb46b75290cf566fac (perf symbols: Improve debug image search when loading symbols)
   * http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=b5a6325464b700c4bdac8799c495970516eed41c (perf events: Fix mmap offset determination)

The final patch is not upstreamable yet - strictly speaking it's broken, but it does make loading symbols from Ubuntu packages work in the common case (which is otherwise broken) - I'm working on a better version of this.

The interim, "broken" patch can be found:
   * here: http://lkml.org/lkml/2010/8/3/197, or
   * here: http://repo.or.cz/w/linux-2.6/linux-wd.git/commitdiff/224a2106f63943b3194b2080bdf938ea94fe8d22 (Will Deacon's tree)

Note that some of these patches may already be merged in linaro kernel tree.

Revision history for this message
Dave Martin (dave-martin-arm) wrote :

The following, ARM-specific fix is also now in Arnaldo's perf-core branch, here:
   * http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=commitdiff;h=f83d9ca2ed2f0ba6380cca5af008ce1dfbbf33d6 (perf symbols: Ignore mapping symbols on ARM)

This is non-essential, but makes the profile output less confusing in some cases.

Revision history for this message
Dave Martin (dave-martin-arm) wrote :

> The interim, "broken" patch can be found:
> * here: http://lkml.org/lkml/2010/8/3/197, or
> * here: http://repo.or.cz/w/linux-2.6/linux-wd.git/commitdiff/224a2106f63943b3194b2080bdf938ea94fe8d22 (Will Deacon's tree)

The above patch should be backed out (if present) and replaced by the following patch:

http://lkml.org/lkml/2010/8/12/127 (perf symbols: fix symbol offset breakage with separated debug)

Still waiting for upstream feedback on this.

Revision history for this message
Andy Whitcroft (apw) wrote :

I believe all of this lot is in the v2.6.38-rc2 kernel, and therefore will be in the Natty kernel. Moving Fix Committed there.

Andy Whitcroft (apw)
Changed in linux (Ubuntu):
status: New → Fix Committed
assignee: nobody → Andy Whitcroft (apw)
Changed in linux (Ubuntu Maverick):
importance: Undecided → Low
Changed in linux (Ubuntu):
importance: Undecided → Low
Changed in linux (Ubuntu Maverick):
status: New → Triaged
Revision history for this message
Dave Martin (dave-martin-arm) wrote :

Note: the patch http://lkml.org/lkml/2010/8/12/127 isn't merged yet, but I need to rework this upstream. It's not a top priority, so it may take a while.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.2 KiB)

This bug was fixed in the package linux - 2.6.38-1.27

---------------
linux (2.6.38-1.27) natty; urgency=low

  [ Andy Whitcroft ]

  * ubuntu: AUFS -- update aufs-update to track new locations of headers
  * ubuntu: AUFS -- update to c5021514085a5d96364e096dbd34cadb2251abfd
  * SAUCE: ensure root is ready before running usermodehelpers in it
  * correct the Vcs linkage to point to natty
  * rebase to linux tip e78bf5e6cbe837daa6ab628a5f679548742994d3
  * [Config] update configs following rebase
    e78bf5e6cbe837daa6ab628a5f679548742994d3
  * SAUCE: Yama: follow changes to generic_permission
  * ubuntu: compcache -- follow changes to bd_claim/bd_release
  * ubuntu: iscsitarget -- follow changes to open_bdev_exclusive
  * ubuntu: ndiswrapper -- fix interaction between __packed and packed
  * ubuntu: AUFS -- update to 806051bcbeec27748aae2b7957726a4e63ff308e
  * update package version to match payload version
  * rebase to e6f597a1425b5af64917be3448b29e2d5a585ac8
  * rebase to v2.6.38-rc1
  * [Config] updateconfigs following rebase to v2.6.38-rc1
  * SAUCE: x86 fix up jiffies/jiffies_64 handling
  * rebase to linus tip 2b1caf6ed7b888c95a1909d343799672731651a5
  * [Config] updateconfigs following rebase to
    2b1caf6ed7b888c95a1909d343799672731651a5
  * [Config] disable CONFIG_TRANSPARENT_HUGEPAGE to fix i386 boot crashes
  * ubuntu: AUFS -- suppress benign plink warning messages
    - LP: #621195
  * [Config] CONFIG_NR_CPUS=256 for amd64 -server flavour
  * rebase to v2.6.38-rc2
  * rebase to mainline d315777b32a4696feb86f2a0c9e9f39c94683649
  * rebase to c723fdab8aa728dc2bf0da6a0de8bb9c3f588d84
  * [Config] update configs following rebase to
    c723fdab8aa728dc2bf0da6a0de8bb9c3f588d84
  * [Config] disable CONFIG_AD7152 to fix FTBS on armel versatile
  * [Config] disable CONFIG_AD7150 to fix FTBS on armel versatile
  * [Config] disable CONFIG_RTL8192CE to fix FTBS on armel omap
  * [Config] disable CONFIG_MANTIS_CORE to fix FTBS on armel versatile

  [ Kees Cook ]

  * SAUCE: kernel: make /proc/kallsyms mode 400 to reduce ease of attacking

  [ Stefan Bader ]

  * Temporarily disable RODATA for virtual i386
    - LP: #699828

  [ Tim Gardner ]

  * [Config] CONFIG_NLS_DEFAULT=utf8
    - LP: #683690
  * [Config] CONFIG_HIBERNATION=n
  * update bnx2 firmware files in d-i/firmware/nic-modules

  [ Upstream Kernel Changes ]

  * Revert "drm/radeon/bo: add some fallback placements for VRAM only
    objects."
  * packaging: make System.map mode 0600
  * thinkpad_acpi: Always report scancodes for hotkeys
    - LP: #702407
  * sched: tg->se->load should be initialised to tg->shares
  * Input: sysrq -- ensure sysrq_enabled and __sysrq_enabled are consistent
  * brcm80211: include linux/slab.h for kfree
  * pch_dma: add include/slab.h for kfree
  * i2c-eg20t: include linux/slab.h for kfree
  * gpio/ml_ioh_gpio: include linux/slab.h for kfree
  * tty: include linux/slab.h for kfree
  * winbond: include linux/delay.h for mdelay et al

  [ Upstream Kernel Changes ]

  * mark the start of v2.6.38 versioning
  * rebase v2.6.37 to v2.6.38-rc2 + c723fdab8aa728dc2bf0da6a0de8bb9c3f588d84
    - LP: #689886
    - LP: #702125
    - LP: #608775
    - LP: #215802
...

Read more...

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Julian Wiedmann (jwiedmann) wrote :

This release has reached end-of-life [0].

[0] https://wiki.ubuntu.com/Releases

Changed in linux (Ubuntu Maverick):
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.