Missing linux-image-debug packages and metapackages since Intrepid

[f3a97]

Hi,
         it seems that the debug package is missing from repositories!

I want it in order to run Systemtap.

It seems that the repos contains a debug image for 2.6.25, is that correct?

Thanks

[Martin Pitt (pitti)]

In Intrepid it is now marked as a "debug symbol" package and lands at

  http://ddebs.ubuntu.com/pool/main/l/linux/

However, I agree that you aren't the first one who complains about
this, and it isn't all that obvious how to find it.

Maybe some kernel package should at least ship a README to explain how
to find it?

[f3a97 (f3a97)]

Thanks Martin,

        but what does this mean?

Why is this package not present into ubuntu repositories?

Should we add some apt source entriy to get them?

I think this should be definitely made clearer.

[f3a97 (f3a97)]

For example,
        with the current kernel package:

ii linux-image-generic 2.6.27.7.11

The previous kernel package seems to have a different naming convention:

ii linux-image-2.6.27-7-generic 2.6.27-7.15

Why is that? What kind of debug image I have to install?

I don't see any package with this particular version in the ddeb site you mentioned.

BTW if I issue a uname, I get:

2.6.27-7-generic

Thanks

[Matt Ginzton (matt-ginzton)]

Duplicate of https://bugs.launchpad.net/ubuntu/intrepid/+source/linux-meta/+bug/253904?

[Emanuele Rocca (ema)]

Some linux-image-debug packages are available on ddebs.ubuntu.com, but according to https://wiki.ubuntu.com/DebuggingProgramCrash one should be able to install them via apt-get, and this is not the case.

The ddebs are also not "in sync" with kernels available in intrepid. On my machine the most recent kernel available is linux-image-2.6.27-11-generic, and on ddebs.u.c the only 2.6.27 is -14.

Debug images for jaunty are correct, though (2.6.28-9 and -10).

[Max Bowsher (maxb)]

During the Intrepid cycle, the 'linux' source package ceased to build explicit debug .debs. These were apparently "replaced" by ddebs - however, http://ddebs.ubuntu.com/pool/main/l/linux/ seems to be consistently carrying only ddebs for the current *development* release, thus leaving users of Intrepid and Jaunty stable releases without a feature that was offered in Hardy and previous.

This should ideally be fixed - if it's not going to be, then at least some explanation for removing this facility needs to be published.

[Rumpeltux (rumpeltux)]

Is there any known workaround? Is there any way to build linux-image.2.x.x yourself and enable the debug-image explicitly?

The bug is really annoying as the ddeb packages from the site mentioned are for kernel which either render my system completely unusable or are not available at all.

[Rumpeltux (rumpeltux)]

Ok building the ddeb yourself is apperently simple. Editing debian/rules.d/0-common-vars.mk and replacing the two (2) instances of skipdbg=true with skipdbg=false will produce a linux-image-debug when compiled as described there:
https://help.ubuntu.com/community/Kernel/Compile

[Leann Ogasawara (leannogasawara)]

This has apparently been discussed on the Ubuntu kernel-team mailing list:

https://lists.ubuntu.com/archives/kernel-team/2009-June/005931.html

"Debug packages are so large that they are only built on the buildd's.
You can simulate the same build environment as the buildd by

sudo mkdir /CurrentlyBuilding
debuild -b"

[Max Bowsher (maxb)]

Regardless of whether this bug eventually ends up as "Won't Fix" or not, I don't think it's valid to set it as such on the basis of the mail quoted above. That mail concerns how to build the debug packages locally, not why they are no longer in the archive.

Looking through the kernel-team mail archives, I found this thread:
https://lists.ubuntu.com/archives/kernel-team/2009-March/004570.html

which demonstrates that the rationale for no longer publishing debug packages in the archive and the validity of not doing so is unresolved.

[Domas Mituzas (domas-mituzas)]

Dear sirs, jaunty (and intrepid, it seems) does not have linux-image-debug packages, and ddebs.ubuntu.com don't carry anything pre-karmic at the moment.
As systemtap, oprofile and plethora of other tools depend on this, this is quite a show stopper.

[Lennart Hansen (lahansen)]

I can confirm this is also the case with Kamic beta.

[Martin Olsson (mnemo)]

These packages are really really really useful and lots of other things are large in size but still remain in repos, why can't these packages be available on ddebs just like the other -dbgsym packages?

This bug makes it very hard to get started with oprofile and systemtap on Ubuntu; I think many issues (both bugs and performance problems) could be resolved if it wasn't such a pita to get started with these tools.

[Martin Olsson (mnemo)]

If the file size is really unacceptably large, then maybe as a workaround you can provide a package called linux-image-debugsymbols-installer which contains a script that re-creates the debug symbols for the current kernel locally on the users machine?

[Max Bowsher (maxb)]

IIUC they are on ddebs initially but expire quickly as newer kernels are uploaded to the development release.

*However*, I don't understand why they can't be added to the normal archive, which is the purpose of having *-dbg | *-dbgsym alternated dependencies AFAIK.

[Stefan Sauer (ensonic)]

Replying to comment #9. I read that email you are pointing to, but the instructions are a bit sparse.What should I use instead of "CurrentlyBuilding". Please note that obviously I don't indent to rebuild the kernel myself.

[Bob Stine (bob-waltonstine)]

Just to reiterate the problem with no linux-image-debug-generic... installing SystemTap is quick and easy in Hardy Heron. This is NOT the case with Karmic Koala.

I'll investigate other Linux platforms before wasting time building my own debug kernel.

In my opinion, this is one hell of a regression.

[Lucian Adrian Grijincu (lucian.grijincu)]

No need to build your kernel
Already built kernels are here: http://ddebs.ubuntu.com/pool/main/l/linux/

I'm able to use Systemtap with these kernels.
You can also add it to you apt/sources.list

The only problem: there is no automated update of the .ddeb when the
coresponding .deb gets updated.

[Max Bowsher (maxb)]

Odd... there seem to be more kernel debug packages on ddebs than I remember there being in the past - they seemed to quickly expire after new versions were built in newer distroseries last time I looked.

[Dan Kegel (dank)]

http://ddebs.ubuntu.com/pool/main/l/linux/ does not have the kernel I need (2.6.31-17-generic),
it only has 2.6.31-18.
This is annoying when trying to use oprofile.

[Brandon Black (blblack)]

As above, the ddebs link is useless for me. I'm running the current latest release of Ubuntu server, fully updated. My kernel is "2.6.31-17-server". There is no vmlinux shipped, there doesn't appear to be one available via apt, and even the ddebs link noted above only contains packages for kernel 2.6.32-11.

This is pretty simple stuff guys. When a user decides to break out oprofile to solve problems, there should be *some* mechanism by which they can obtain a valid vmlinux for the vendor-built kernel. One that doesn't involve jumping through the hoops of "figure out the vendor's kernel build process and rebuild a new kernel exactly like the one they shipped you, pointlessly, just to get a corresponding copy of vmlinux".

Given the current situation, I don't see why you even bothered putting the oprofile tools in apt or enabling oprofile in your kernel builds...

[jambo (jdramer)]

I'd like to echo Brandon's sentiments. None of the kernel images available through apt have corresponding linux-image-debug packages on http://ddebs.ubuntu.com/pool/main/l/linux/

Is there any location where these might be archived?

[Thierry Carrez (ttx)]

No clear solution, more a kernel issue than a server issue. Invalidated in 20100210 meeting.

[Henning Moll (drscott)]

It seems that

http://ddebs.ubuntu.com/pool/main/l/linux/

only contains the current proposed kernel. Unfortunately, this problem makes apport-retrace in an productive environment useless. The description on

https://wiki.ubuntu.com/KernelTeam/CrashdumpRecipe

is not working.

[Chris Conway (cconway)]

Add my voice to the chorus frustrated at the difficulty of getting a debug image corresponding to my current linux image (linux-image-2.6.31-19-generic=2.6.31-19.56) for use with oprofile. Can anyone point to a step-by-step guide to building a debug image from source?

[Chris Conway (cconway)]

I seem to have successfully built a debug image. Here's how:

apt-get source linux
apt-get build-dep linux
cd linux-2.6.31/
fakeroot make -f debian/rules binary-generic skipdbg=false
sudo dpkg -i ../linux-image-debug-2.6.31-19-generic_2.6.31-19.56_amd64.ddeb

Replace "generic" with whichever flavor of the kernel you want to build and, obviously, the version number and amd64 with the version you are building and your host architecture.

Building the image takes a long time (20-30 minutes) and the resulting package is more than 400MB.

[ilia (ilia)]

Hey, Debian now builds dbg packages for the kernel
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=365349#194
Can Ubuntu do the same?
As pointed here and in dups debugging info for released kernels is a dependency of many profiling/debugging/troubleshooting tools, and it's not just for kernel developers, so you cannot assume that anyone needing, say, systemtap is ready to deploy custom kernel build. This is especially important for LTS release as targeting enterprise. Enterprise admins just can't deploy non-official packages in production.
Please fix this in Lucid!

[Andy Whitcroft (apw)]

The debug packages are already being built, our buld system is predicated on building the ddebs first and stripping them for the main archive. As pointed out by pitti they should be available at the URL below, on ddebs.ubuntu.com where all debug .debs are located. One or two have been missing recently due to an over active reaper, but in general this is where they are:

    http://ddebs.ubuntu.com/pool/main/l/linux/

[Andy Whitcroft (apw)]

Ok, investigation seems to indicate that the hacks to maintain the older kernel debug images (which had different filename formats) was incorrectly triggering expirey of those with the new form. Lucid and later. It is believed that these are now fixed and future images should correctly remain on ddebs until the kernel they apply to is reaped from the main archive.

Unfortuanatly there is no way to recover the lost images but we will be monitoring the new images to ensure they remain correctly. We expect an upload shortly which will create new debug images to match the new kernel.

As this bug referrs to fixing the kernel packaging to produce debug debs, which it does correctly do I will close this one off Fix Released.

[Jeremy Foshee (jeremyfoshee)]

Per Andy's comment above and Pitti's from 2008, I am marking this Fix Released.

~JFo

[Henning Moll (drscott)]

This has not just been an problem for ludid and later. It always happend also for karmic kernels. Maybe this is solved by the same fix. I just want to let you now in case there needs to be something else fixed.

[Alain Kalker (miki4242)]

I would like to make a few updates, improvements to Chris's very helpful howto/workaround for building your own debug symbols (post #23):
- doing just `apt-get linux` will get 'linux-meta' package which we don't need: let apt figure out which source package to get.
- also don't install recommended packages: this saves you installing ~ 800MB worth of unneeded packages (mostly texlive), you might need that space after all for building the ddeb ...

So I'm using:

apt-get source linux-image-$(uname -r)
apt-get build-dep -no-install-recommends linux-image-$(uname -r)
cd linux-2.6.32/
fakeroot make -f debian/rules binary-generic skipdbg=false

[Alain Kalker (miki4242)]

Fix small typo:

apt-get source linux-image-$(uname -r)
apt-get build-dep --no-install-recommends linux-image-$(uname -r)
cd linux-2.6.32/
fakeroot make -f debian/rules binary-generic skipdbg=false

[Andy Whitcroft (apw)]

@Henning -- this has actually been fixed twice, once for the older releases (intrepid to karmic), and now again for newer releases (lucid onwards).

For the sake of clarity here is the history. There are three different classes of debug debs:

1) in the very oldest release (only Dapper and Hardy now) the debug packages are real .deb's in the archive and are correctly available.
2) in intrepid through karmic they are ddebs but named <image>-debug
3) from lucid onwards they are ddebs names <image>-dbgsym

When ddebs were introduced we started building our kernel ddebs as <image>-debug. This appeared to work as the ddebs appeared in the right place. However they were not being matched correctly to the master .deb in the main archive and were being cleaned up after 14 days. When this was discovered we were requested to rename them to <image>-dbgsym to match the archive standard. In parallel a hack was put in place in the reaper to keep the <image>-debug in place. This sorted out the Intrepid->Karmic ddebs being incorrectly deleted but inadvertently introduced the issue for Lucid and later.

We believe that all releases should now be correctly recognised and the the appropriate .ddebs retained. However, we cannot be 100% sure until we have a new upload in Lucid. We therefore believe the issue should be fixed going forward.

[adi (lp465)]

@apw -- currently http://ddebs.ubuntu.com/pool/main/l/linux/ contains debug images for:

linux-image-2.6.35-3
linux-image-2.6.35-4
linux-image-2.6.35-5
linux-image-2.6.35-6
linux-image-debug-2.6.28-18
linux-image-debug-2.6.28-19
linux-image-debug-2.6.31-20
linux-image-debug-2.6.31-21
linux-image-debug-2.6.31-22

My up-to-date Lucid box has

 linux-image-2.6.32-22-generic 2.6.32-22.36

As far as I can see, the relevant -debug or -dbgsym package is not available.

[joe williams (joetify)]

I see the same sort of thing, I have 2.6.32-23-server which is the latest according to apt yet http://ddebs.ubuntu.com/pool/main/l/linux/ has 2.6.32-24. crash (gdb) fails with:

crash: invalid kernel virtual address: f type: "possible"
WARNING: cannot read cpu_possible_map
crash: /usr/lib/debug/boot/vmlinux-2.6.32-24-server and /tmp/unpacked/VmCore do not match!

Additionally something like the following fails too, which I assume is related since it's trying to install debug packages.

root@ubuntu:~# apport-retrace --stdout --rebuild-package-info /var/crash/linux-image-2.6.32-23-server.0.crash
Traceback (most recent call last):
  File "/usr/bin/apport-retrace", line 261, in <module>
    options.extra_packages)
  File "/usr/lib/python2.6/dist-packages/apport/packaging_impl.py", line 407, in install_retracing_packages
    return self._install_debug_kernel(report)
  File "/usr/lib/python2.6/dist-packages/apport/packaging_impl.py", line 363, in _install_debug_kernel
    ver = report['Package'].split()[1]
IndexError: list index out of range

[nutznboltz (nutznboltz-deactivatedaccount)]

$ uname -srvm
Linux 2.6.32-24-generic #38-Ubuntu SMP Mon Jul 5 09:20:59 UTC 2010 x86_64

$ lsb_release -ds
Ubuntu 10.04.1 LTS

$ dpkg -l | grep linux-image-2
ii linux-image-2.6.32-24-generic 2.6.32-24.39 Linux kernel image for version 2.6.32 on x86
ii linux-image-2.6.32-24-generic-dbgsym 2.6.32-24.39 Linux kernel debug image for version 2.6.32

$ pwd
/usr/share/doc/systemtap-doc/examples/io

$ sudo stap disktop.stp
ERROR: Build-id mismatch: "kernel" vs. "vmlinux-2.6.32-24-generic"

Suggestions?

[Michael (michaeljt)]

This happened to me after I updated the kernel package (and the kernel symbols) recently. A reboot fixed it.

[Michael (michaeljt)]

I might mention that as of the last Lucid kernel update bar one the symbols in the repository match the kernel, so that this issue is solved for me.