crash tool support for the ppc64el arch

[Mauricio Faria de Oliveira]

SRU Justification:

Impact: crash doesn't work on ppc64el architectures
Test Case: run crash on a ppc64el crash dump
Regression Potential: This only affects ppc64el architectures, and is currently in utopic.

--

Hi,

Is it possible to (discuss how to) enable the crash package to on the ppc64el arch?

Recently the kernel received the config changes
(CONFIG_CRASH_DUMP=y, CONFIG_RELOCATABLE=y, currently in the -proposed pocket [1])

Is it hard to move to GDB 7.7? (ppc64el GDB package starts on that version).

Thanks!

[1] https://launchpad.net/ubuntu/trusty/+source/linux/3.13.0-26.48

For testing the trusty SRU:

 - the package builds on ppc64el
 - follow test procedure from comment #6

[Mauricio Faria de Oliveira (mfo)]

I got news we have the patch on top of Ubuntu's src pkg available, and that it will be attached here.

[Matthias Klose (doko)]

patch is still missing

[Bharata B Rao (bharata)]

This patchset is for enabling ppc64le on gdb-7.6. These patches need to be applied to gdb-7.6 that is included in crash-7.0.3 to make crash work on ppc64le. Most of these patches (0001 till 0008) are direct backport of gdb upstream commits. 0009 is one patch that pulls in pieces from multiple gdb upstream commits.

[Mauricio Faria de Oliveira (mfo)]

Hi Matthias,

I built a test version of the crash package for internal testing here.
It's probably going to have its round of testing on Monday (due to timezone differences).

It's just
  - the patches above in gdb-7.6-ppc64el.patch (filterdiff --strip=1 --addprefix=gdb-7.6/ patches/*.patch)
  - changes in debian/rules to incorporate that into gdb-7.6.patch (I didn't try to make the build system pick more than 1 patch file in Makefile)
   - trivial change in debian/control.

Attaching the debdiff if it's of any help.

Thanks,

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "crash_7.0.3-3ubuntu2.1+lp1318809.c3_ppc64el.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Kalpana S Shetty (kalshett)]

I have tested the ported "crash" tool from development team(Mauricio and Bharata). It is working fine. Here are the steps I followed to do basic testing.

1) dump Ubuntu 14.04 LE VM dump using virsh dump command
Ex: virsh dump --memory-only kal_ubu_at kal_ubu_at.dump --verbose
VM: kal_ubu_at
dump_file: kal_ubu_at.dump

2) scp "kal_ubu_at.dump" to Ubuntu VM

3) Install crash tool from development
  (http://poc1.ltc.br.ibm.com/~mauricfo/for_kalpana/lp1318809/crash_7.0.3-3ubuntu2.1+lp1318809.c3_ppc64el.deb)

4) Run crash =============> PASS

5) Open dump file using crash (for analysis purpose....)
crash /usr/lib/debug/boot/vmlinux-3.13.0-24-generic /root/kal_ubu_at.dump

Could able to open the dump file =======================> PASS

crash 7.0.3
Copyright (C) 2002-2013 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu"...

      KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-24-generic
    DUMPFILE: /root/kal_ubu_at.dump
        CPUS: 1
        DATE: Mon May 19 11:58:51 2014
      UPTIME: 2 days, 13:26:42
LOAD AVERAGE: 0.00, 0.03, 0.05
       TASKS: 139
    NODENAME: ubuk61
     RELEASE: 3.13.0-24-generic
     VERSION: #47-Ubuntu SMP Fri May 2 23:29:18 UTC 2014
     MACHINE: ppc64le (3026 Mhz)
      MEMORY: 2 GB
       PANIC: ""
         PID: 0
     COMMAND: "swapper/0"
        TASK: c000000000ea59f0 [THREAD_INFO: c000000000fc8000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

crash>

few of crash commands tried:

crash> bt
PID: 0 TASK: c000000000ea59f0 CPU: 0 COMMAND: "swapper/0"

 R0: 0000000022002084 R1: c000000000fcba70 R2: c000000000fccf88
 R3: 0000000000000000 R4: c000000000e24e80 R5: 0000000000000000

crash> ipcs
SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS
c000000075b2c380 00000000 0 0 644 80 2
c000000075b2d280 00000000 32769 0 644 16384 2
c000000075b22d80 00000000 65538 0 644 280 2
c0000000778ba980 00000000 98307 0 644 64 2 dest
c0000000778bd280 00000000 131076 0 644 16384 2 dest
c0000000778b9f80 00000000 163845 0 644 280 2 ...

[Matthias Klose (doko)]

now uploaded to utopic. is it needed/wanted to fix for trusty too?

[Mauricio Faria de Oliveira (mfo)]

Hi Matthias,

Yes, we need this for trusty.

Thanks.

[Kalpana S Shetty (kalshett)]

We need to make sure ported crash packages should be part of upcoming SRU cycle. I'm not sure when would be the next SRU patches coming in. Need help from Ubuntu/Canonical.

[Matthias Klose (doko)]

fixed in utopic

[Kalpana S Shetty (kalshett)]

Don't we get this as part of 14.04 SRU ?

[Chris J Arges (arges)]

Hello Mauricio, or anyone else affected,

Accepted crash into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/crash/7.0.3-3ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Please also submit this to the upstream project https://github.com/crash-utility/crash.
In addition it would be helpful for this to make it into debian as well.

[Mauricio Faria de Oliveira (mfo)]

Hi Chris,

Thanks, I see the new version in trusty-proposed.

$ echo 'deb http://ports.ubuntu.com/ubuntu-ports/ trusty-proposed restricted main multiverse universe' | sudo tee /etc/apt/sources.list.d/trusty-proposed.list
deb http://ports.ubuntu.com/ubuntu-ports/ trusty-proposed restricted main multiverse universe

$ sudo apt-get update

$ apt-cache policy crash
crash:
  Installed: (none)
  Candidate: 7.0.3-3ubuntu3
  Version table:
     7.0.3-3ubuntu3 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty-proposed/main ppc64el Packages

[Mauricio Faria de Oliveira (mfo)]

Hi Kalpana,

Can you please test the version in -proposed?

I dont have virsh handy in my environment, and kdump-tools is not quite working here (I'll follow up internally).

(add the trusty-proposed repo as in comment #14, then reinstall the package, if the dev version is still there)

$ sudo apt-get install --reinstall crash
[...]
Get:1 http://ports.ubuntu.com/ubuntu-ports/ trusty-proposed/main crash ppc64el 7.0.3-3ubuntu3 [2,601 kB]
[...]
Setting up crash (7.0.3-3ubuntu3) ...
$

Thank you.

[Mauricio Faria de Oliveira (mfo)]

Chris,

One of the bugs in the way for in-guest generated crash dumps is [1].
Is it possible for you to please help there, just to make it gets assigned quickly?

Thank you very much,

[1] https://bugs.launchpad.net/ubuntu/+source/makedumpfile/+bug/1324544

[Kalpana S Shetty (kalshett)]

I tested with "trusty-proposed" packages and crash tool is working fine.

Steps to install trusty-proposed crash tool:

1) Added below entry in /etc/apt/source.list
deb http://ports.ubuntu.com/ubuntu-ports/ trusty-proposed restricted main multiverse universe

2) Did apt-get update

3) Check for crash tool
root@ubuk60:~# apt-cache policy crash
crash:
  Installed: (none)
  Candidate: 7.0.3-3ubuntu3
  Version table:
     7.0.3-3ubuntu3 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty-proposed/main ppc64el Packages
root@ubuk60:~#

4) apt-get install crash

5) crash /usr/lib/debug/boot/vmlinux-3.13.0-27-generic /root/kal_ubu_stress.dump

I could able to invoke crash with debug linux and able to see the bt etc....

We can push this package to SRU.

[Mauricio Faria de Oliveira (mfo)]

Thanks, Kalpana.

Changing the tag to verification-done, as per Kalpana's comment #17, as requested on comment #12.

> [...] If this package fixes the bug for you, please add a comment to this bug,
> mentioning the version of the package you tested, and change the tag from verification-needed to verification-done.

Thanks for the upload, Chris.

[Launchpad Janitor (janitor)]

This bug was fixed in the package crash - 7.0.3-3ubuntu3

---------------
crash (7.0.3-3ubuntu3) trusty-proposed; urgency=medium

  * Build crash for ppc64el. LP: #1318809.
  * Add ppc64el support (debian/control, debian/rules (for patches),
    gdb-7.6-ppc64el.patch); patch-set provided by Bharata Rao.
 -- Mauricio Faria de Oliveira <email address hidden> Fri, 16 May 2014 08:52:26 -0300

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for crash has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.