[Ubuntu18.04] linux-image-4.13.0-25-generic-dbgsym is not available in repo which is required for crash analysis of vmcore

[bugproxy]

---Problem Description---
No debug kernel deb packages available for 4.13.0-25-generic kernel [Ubuntu 18.04 ]
Unable to analyze vmcore generated via crash.

---uname output---
Linux spoon2 4.13.0-25-generic #29-Ubuntu SMP Mon Jan 8 21:15:55 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = 8335-GTC

---Steps to Reproduce---
Install 18.04 ppc64el
upgrade kernel to 4.13.0-25-generic or 4.13.0-17-generic
try configuring debug repo
https://wiki.ubuntu.com/Kernel/CrashdumpRecipe
http://ddebs.ubuntu.com/pool/main/l/linux/
sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-proposed main restricted universe multiverse
EOF

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C01
sudo apt-get update
sudo apt-get install linux-image-$(uname -r)-dbgsym

Warning /!\ Be aware that those packages are huge! (~600 MB)

When installed, the debug kernel can be found under /usr/lib/debug/boot/ and crash is started by:

crash <debug kernel> <crash dump>

Crash command to analyze vmcore will not work as kernel debuginfo deb package is not found in repo .
root@spoon2:~# sudo apt-get install linux-image-$(uname -r)-dbgsym
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package linux-image-4.13.0-25-generic-dbgsym
E: Couldn't find any package by glob 'linux-image-4.13.0-25-generic-dbgsym'
E: Couldn't find any package by regex 'linux-image-4.13.0-25-generic-dbgsym'
root@spoon2:~#

I was also not able to find this debug symbol for 18.04 kernel. I understand that the current 18.04 kernel is old and was removed from the archive. Is it correct?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-08 16:07 EDT-------
Repeating the same question about the 4.13.0-25-generic kernel debug symbols:

I was also not able to find this debug symbol for 18.04 kernel. I understand that the current 18.04 kernel is old and was removed from the archive. Is it correct?

[Joseph Salisbury (jsalisbury)]

When trying to debug a bug, it's probably best to use the latest Ubuntu kernel version, in case it's already fixed.

Are you specifically looking for older versions of kernels and their dbgsym package?

All prior kernel versions can be found here:
https://launchpad.net/ubuntu/artful/+source/linux

From that page, select the version. On the next page, select ppc64el under the "Builds" section. The linux-image-VERSION-generic-dbgsym kernel can be downloaded from that last page.

[Joseph Salisbury (jsalisbury)]

Specifically, the 4.13.0-25 dbgsym package is available from here:
https://launchpad.net/~canonical-kernel-security-team/+archive/ubuntu/ppa/+build/14223090

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-19 02:22 EDT-------
I am looking for debug symbols package linux-image-4.15.0-generic-dbgsym

Any link where I can get the packages for bionic kernel ?

[Joseph Salisbury (jsalisbury)]

That is available here:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/bootstrap/+build/14347013

If you want a different arch, you can select that arch from here, under 'Builds':
https://launchpad.net/ubuntu/+source/linux/4.15.0-10.11

[Andrew Cloke (andrew-cloke)]

After providing a reference to the requested debug symbols, awaiting information as to whether this issue is reproducible with the latest Ubuntu kernel.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-27 06:10 EDT-------
(In reply to comment #14)
> After providing a reference to the requested debug symbols, awaiting
> information as to whether this issue is reproducible with the latest Ubuntu
> kernel.

Comment by Abdul, on LP1746088
On Ubuntu 1804 Bionic, with 4.15.0-10 kernel. crash analysis failed. I think the fixes are yet to pushed to bionic

# crash /share/10.10.10.43-201802210423/dump.201802210423 /usr/lib/debug/boot/vmlinux-4.15.0-10-generic

crash 7.2.0
Copyright (C) 2002-2017 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu"...

please wait... (gathering task table data)
crash: cannot resolve "init_task_union"

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-27 06:22 EDT-------
Is crash fix available?

[Thadeu Lima de Souza Cascardo (cascardo)]

Did you upgrade? crash 7.2.1 is already available on bionic.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-27 07:39 EDT-------
(In reply to comment #17)
> Did you upgrade? crash 7.2.1 is already available on bionic.

Sorry for the noise,

I verified this now,
=============
# dpkg -s crash
Package: crash
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 10698
Maintainer: Ubuntu Developers <email address hidden>
Architecture: ppc64el
Version: 7.2.1-1
Depends: libc6 (>= 2.17), liblzo2-2, libncurses5 (>= 6), libsnappy1v5, libtinfo5 (>= 6), zlib1g (>= 1:1.2.0), binutils
Suggests: kexec-tools, makedumpfile
Description: kernel debugging utility, allowing gdb like syntax
The core analysis suite is a self-contained tool that can be used to
investigate either live systems, or multiple different core dump formats
including kdump, LKCD, netdump and diskdump.
.
o The tool is loosely based on the SVR4 crash command, but has been
completely integrated with gdb in order to be able to display
formatted kernel data structures, disassemble source code, etc.
.
o The current set of available commands consist of common kernel core
analysis tools such as a context-specific stack traces, source code
disassembly, kernel variable displays, memory display, dumps of
linked-lists, etc. In addition, any gdb command may be entered,
which in turn will be passed onto the gdb module for execution.
.
o There are several commands that delve deeper into specific kernel
subsystems, which also serve as templates for kernel developers
to create new commands for analysis of a specific area of interest.
Adding a new command is a simple affair, and a quick recompile
adds it to the command menu.
.
o The intent is to make the tool independent of Linux version dependencies,
building in recognition of major kernel code changes so as to adapt to
new kernel versions, while maintaining backwards compatibility.
Original-Maintainer: Troy Heber <email address hidden>

root@<hostname>/var/crash/201802220222# crash dump.201802220222 /usr/lib/debug/boot/vmlinux-4.15.0-10-generic

crash 7.2.1
Copyright (C) 2002-2017 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu"...

KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-10-generic
DUMP...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-27 07:41 EDT-------
since 4.15.0-11-generic is released, can we get dbgsym for linux-image-4.15.0-11-generic-dbgsym_4.15.0-11.xx_ppc64el.ddeb ?

[Dimitri John Ledkov (xnox)]

On 27 February 2018 at 12:50, bugproxy <email address hidden> wrote:
> ------- Comment From <email address hidden> 2018-02-27 07:41 EDT-------
> since 4.15.0-11-generic is released, can we get dbgsym for linux-image-4.15.0-11-generic-dbgsym_4.15.0-11.xx_ppc64el.ddeb ?

This question is odd. How do you normally "get dbgsym" packages and
install them?

Have you followed the instructions at
https://wiki.ubuntu.com/Debug%20Symbol%20Packages to enable the debug
symbols repository, and thus have all -dbgsym packages available for
installation with "apt install" for any package in Ubuntu?

--
Regards,

Dimitri.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-27 08:46 EDT-------
(In reply to comment #20)
> On 27 February 2018 at 12:50, bugproxy <email address hidden> wrote:
> > since 4.15.0-11-generic is released, can we get dbgsym for linux-image-4.15.0-11-generic-dbgsym_4.15.0-11.xx_ppc64el.ddeb ?
>
> This question is odd. How do you normally "get dbgsym" packages and
> install them?
>
> Have you followed the instructions at
> https://wiki.ubuntu.com/Debug%20Symbol%20Packages to enable the debug
> symbols repository, and thus have all -dbgsym packages available for
> installation with "apt install" for any package in Ubuntu?
>
> --
> Regards,
>
> Dimitri.

Excellent,
we were referring to https://wiki.ubuntu.com/Kernel/CrashdumpRecipe
which has obsolete keys?
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C0 ??

With https://wiki.ubuntu.com/Debug%20Symbol%20Packages we are able to install linux-image-4.15.0-11-generic-dbgsym ppc64el 4.15.0-11.12

With regards,
Naveed

[Andrew Cloke (andrew-cloke)]

Thanks for confirming that you are now able to download the debug symbols package. Are you ok for us to close?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-03-05 14:01 EDT-------
Yes we can close this bug as fixed.

Thank you for the support and help.