Systemtap should require kernel debug information

Bug #106957 reported by Sylvain Pasche on 2007-04-16
78
This bug affects 10 people
Affects Status Importance Assigned to Milestone
systemtap (Ubuntu)
Medium
Unassigned

Bug Description

Binary package hint: systemtap

If you have no kernel debug image package installed, systemtap won't work as expetced. It should require or at least recommend installing a linux-debug dependency for the running kernel.

Another issue I experienced is that systemtap is not looking at the right kernel debug file, event if you have it installed. I had to manually create a symbolic link from /boot/vmlinux-2.6.20-15-generic pointing to /boot/vmlinux-dbg-2.6.20-15-generic.

I'm not sure if the *-dbg-* naming scheme is a Debian convention, but maybe it should try to look at these files too.

Jordan Kanter (jikanter) wrote :

Thank you for your bug report. Could you please upload the log files and/or tell me exactly what happened when you tried to run this package. Thanks.

Changed in systemtap:
assignee: nobody → jikanter
status: Unconfirmed → Needs Info
Sylvain Pasche (sylvain-pasche) wrote :

I installed systemtap, and tried the sample from the tutorial: http://sources.redhat.com/cgi-bin/cvsweb.cgi/~checkout~/doc/tutorial/strace-open.stp?rev=1.2&content-type=text/plain&cvsroot=systemtap

# stap strace-open.stp
semantic error: libdwfl failure (dwfl_linux_kernel_report_offline): No such file or directory while resolving probe point kernel.function("sys_open")?
semantic error: cannot find kernel debuginfo while resolving probe point kernel.function("sys32_open")?
semantic error: no match for probe point while resolving probe point syscall.open
Pass 2: analysis failed. Try again with more '-v' (verbose) options.

Then with the help of strace, I saw that the stap process wanted to access /boot/vmlinux-2.6.20-15-generic

So I installed the kernel debug package and made the symlink:

# apt-get install linux-image-debug-2.6.20-15-generic
# cd /boot
# ln -s vmlinux-dbg-2.6.20-15-generic vmlinux-2.6.20-15-generic

And afterwards, the strace-open.stp sample works as expected.

Of course, systemtap still can run if you don't have the kernel debug package installed, but you won't be able to use the probes referring to symbol names.

Miek Gieben (miek) wrote :

This is on powerpc - to get systemtap working I followed the instruction from the previous post, but I still get errors:

 ./keyhack.stp -v
Pass 1: parsed user script and 52 library script(s) in 1010usr/20sys/1120real ms.
WARNING: cannot find module kernel debuginfo: No DWARF information found
WARNING: cannot find module kernel debuginfo: No DWARF information found
semantic error: no match for probe point while resolving probe point kernel.function("kbd_event")
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) in 380usr/150sys/592real ms.
Pass 2: analysis failed. Try again with more '-v' (verbose) options.

% ./top.stp -vv
SystemTap translator/driver (version 0.5.12 built 2007-02-02)
(Using Red Hat elfutils 0.123 libraries.)
Copyright (C) 2005-2006 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
Created temporary directory "/tmp/stap5oOiKJ"
Searched '/usr/share/systemtap/tapset/2.6.20-16-powerpc/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6.20-16-powerpc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6.20/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6.20/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/*.stp', match count 33
Searched '/usr/share/systemtap/tapset/LKET/2.6.20-16-powerpc/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6.20-16-powerpc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6.20/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6.20/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/ppc/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/*.stp', match count 19
Pass 1: parsed user script and 52 library script(s) in 990usr/30sys/1184real ms.
WARNING: cannot find module kernel debuginfo: No DWARF information found
WARNING: cannot find module kernel debuginfo: No DWARF information found
semantic error: no match for probe point while resolving probe point kernel.function("sys_*")
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 1 global(s) in 360usr/190sys/609real ms.
Pass 2: analysis failed. Try again with more '-v' (verbose) options.
Running rm -rf /tmp/stap5oOiKJ

Sylvain Pasche (sylvain-pasche) wrote :

Hi Miek,

It may be possible that my instructions above are out of date. You need the linux-image-debug-* package that matches your running kernel. You can get your kernel version with running "uname -a". As of now, I had to install linux-image-debug-2.6.20-16-generic. Of course, the symlink name has to be adapted.

If that doesn't help, you could try running strace, to find what kernel symbol file name stap tries to access:

strace -f -e trace=file -o/tmp/log stap /tmp/your_test.stp
grep vmlinux /tmp/log

Miek Gieben (miek) wrote :

I did all that, I've installed the debug-* package and fixed the symlink - but still no go. I might try to run this on my AMD64 system. But on power-pc it looks like a no-go.

Sylvain Pasche (sylvain-pasche) wrote :

Did you try with strace? What information did you get? You can also attach the interesting information there.

Miek Gieben (miek) wrote :

Okay, I was typing this when systemtap decided to issue a illegal opcode and crash my server :(

Anyhow. Looks like I got it working by:

apt-get install linux-source-2.6.20
compiling a kernel in /usr/src
setting /lib/modules/`uname -r`/build -> /usr/src/linux-source-2.6.20

and then running top.stp, but then my system crashed...

Frank Ch. Eigler (fche) wrote :

Miek, it is possible that this is a systemtap or kernel problem
(in which case we'd appreciate hearing more details about the
crash - see http://sourceware.org/systemtap/wiki/HowToReportBugs).

But it is also possible that your rebuilt kernel does not exactly
match the one you're *running*. This situation is sometimes
hard to catch in software, though we plan to improve on this.

It is also possible that the ubuntu snapshot of systemtap is
too old - many bugs have certainly been fixed in the last six
months.

Jeff Schroeder (sejeff) wrote :

Jordan, symlinking the debug kernel to the running kernel name doesn't work for me. Am I doing something wrong? See bug 115830

Thanks

Jeff Schroeder (sejeff) on 2007-09-20
Changed in systemtap:
status: Incomplete → Confirmed

I confirm that the proposed fix does work under gutsy x86_64.

# ln -s /boot/vmlinux-debug-2.6.22-14-generic /boot/vmlinux-2.6.22-14-generic

oh (oystein-homelien) wrote :

I am unable to resolve module functions with systemtap, like:

probe module("hidp").function("hidp_input_report") {

fails with:

WARNING: cannot find module hidp debuginfo: No DWARF information found
..etc.

I have linux-image-debug installed, but it only contains the _kernel_ with debug info, not the _modules_, it seems. Am I off base here, or are the modules with DWARF information simply .. not available in ubuntu?

Steven Harms (sharms) wrote :

Jordan,

This is still an issue in Hardy, are you still the correct person for this to be assigned to?

Michael R. Head (burner) wrote :

Can the systemtap package effectively do what is described in the http://sourceware.org/systemtap/wiki/SystemtapOnUbuntu page?

$ sudo apt-get install linux-image-debug-generic
$ sudo ln -s /boot/vmlinux-debug-$(uname -r) /lib/modules/$(uname -r)/vmlinux
$ sudo apt-get install linux-headers-generic gcc libcap-dev

After doing that, stap seems to be functioning correctly on hardy now.

Steven Harms (sharms) on 2008-06-05
Changed in systemtap:
assignee: jikanter → nobody
Elliot Hughes (elliot-hughes) wrote :

The package attached depends upon the mentioned required packages - give it a go. I can't promise it won't break anything - but it would be nice to have it tested. This package is for amd64.

I have the same problem as OH mentioned in <a href="https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/106957/comments/11">comment 11</a>.

What packages contains the symbols for the loadable modules?

jlouis (jesper-louis-andersen) wrote :

Just an addition: There seems to be no linux-image-debug-generic package in Intrepid. Will such a package be added later or should we be using something else for Intrepid?

John Kozyrakis (ikoz) wrote :

There is one, but it's very well hidden..... (dont ask why, i dont know too).

The packages containing the vmlinux files are here: http://ddebs.ubuntu.com/pool/main/l/linux/

Daniel Hahler (blueyed) wrote :

See also bug 289087 - which is about the missing packages, e.g. "linux-image-debug-generic".
This is documented at http://sourceware.org/systemtap/wiki/SystemtapOnUbuntu

Changed in systemtap (Ubuntu):
status: Confirmed → Triaged
importance: Undecided → Medium

systemtap 2.1-1~experimental1 in debian now instructs users to see README.Debian every time stap fails. README.Debian then asks them to run the "stap-prep" command that installs all the required headers and debugging symbols. This has been tested on both Debian and Ubuntu.

Michael Terry (mterry) wrote :

This seems like it can be closed, now that stap-prep is the documented way to ensure all the debug packages are installed.

Changed in systemtap (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers