| 2016-07-29 17:59:41 |
bugproxy |
bug |
|
|
added bug |
| 2016-07-29 17:59:43 |
bugproxy |
tags |
|
architecture-ppc64le bugnameltc-143307 severity-critical targetmilestone-inin14045 |
|
| 2016-07-29 17:59:51 |
bugproxy |
attachment added |
|
Compressed working crash binary - compiled with crash_7.0.3-3ubuntu4.4 along with commits 6f1f78 & 098cda https://bugs.launchpad.net/bugs/1607894/+attachment/4709811/+files/crash.gz |
|
| 2016-07-29 17:59:52 |
bugproxy |
ubuntu: assignee |
|
Taco Screen team (taco-screen-team) |
|
| 2016-07-29 17:59:55 |
bugproxy |
affects |
ubuntu |
linux (Ubuntu) |
|
| 2016-07-29 20:18:37 |
Tim Gardner |
affects |
linux (Ubuntu) |
crash (Ubuntu) |
|
| 2016-12-01 16:58:32 |
Robert Hooker |
crash (Ubuntu): assignee |
Taco Screen team (taco-screen-team) |
Robert Hooker (sarvatt) |
|
| 2016-12-07 18:28:59 |
Robert Hooker |
attachment added |
|
debdiff containing the fix https://bugs.launchpad.net/ubuntu/+source/crash/+bug/1607894/+attachment/4788691/+files/crash_7.0.3-3ubuntu4.5.debdiff |
|
| 2016-12-07 20:28:10 |
Ubuntu Foundations Team Bug Bot |
tags |
architecture-ppc64le bugnameltc-143307 severity-critical targetmilestone-inin14045 |
architecture-ppc64le bugnameltc-143307 patch severity-critical targetmilestone-inin14045 |
|
| 2016-12-07 20:28:18 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2016-12-08 15:24:43 |
Dimitri John Ledkov |
description |
== Comment: #0 - NAVEED A. UPPINANGADY SALIH - 2016-07-01 02:41:07 ==
---Problem Description---
Unable to analyse vmcore/kernel dumpfile generated by 4.4.0-29-generic kernel with ddeb kernel debuginfo available in http://ddebs.ubuntu.com/pool/main/l/linux/
http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-4.4.0-29-generic-dbgsym_4.4.0-29.48_ppc64el.ddeb
---uname output---
4.4.0-29-generic #48~14.04.1-Ubuntu SMP Wed Jun 29 19:55:03 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
Machine Type = model : 8247-22L machine : PowerNV 8247-22L
---Steps to Reproduce---
# crash /usr/lib/debug/boot/vmlinux-4.4.0-29-generic /var/crash/201606300840/dump.201606300840
crash 7.0.3
...
This GDB was configured as "powerpc64le-unknown-linux-gnu"...
crash: invalid kernel virtual address: 61c2555c09c17994 type: "possible"
WARNING: cannot read cpu_possible_map
crash: invalid kernel virtual address: 8a0f6ddee20c4116 type: "present"
WARNING: cannot read cpu_present_map
crash: invalid kernel virtual address: 5a09344e320a1886 type: "online"
WARNING: cannot read cpu_online_map
WARNING: cannot read linux_banner string
crash: /usr/lib/debug/boot/vmlinux-4.4.0-29-generic and /var/crash/201606300840/dump.201606300840 do not match!
Userspace rpm: linux-image-4.4.0-29-generic-dbgsym_4.4.0-29.48_ppc64el.ddeb
== Comment: #5 - Naresh Bannoth - 2016-07-27 04:12:09 ==
The Latest kernel I am having is as follows
root@ltcalpine-lp6:~# uname -a
Linux ltcalpine-lp6 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:03:56 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
root@ltcalpine-lp6:~#
====>>> the issue same as posted originally.
...
This GDB was configured as "powerpc64le-unknown-linux-gnu"...
crash: invalid kernel virtual address: 61c2555c09c17994 type: "possible"
WARNING: cannot read cpu_possible_map
crash: invalid kernel virtual address: 8a0f6ddee20c4116 type: "present"
WARNING: cannot read cpu_present_map
crash: invalid kernel virtual address: 5a09344e320a1886 type: "online"
WARNING: cannot read cpu_online_map
WARNING: cannot read linux_banner string
crash: /usr/lib/debug/boot/vmlinux-4.4.0-31-generic and /var/crash/201607270248/dump.201607270248 do not match!
== Comment: #6 - Hari Krishna Bathini - 2016-07-28 11:49:31 ==
The kernel is compiled with gcc version 4.8.4 while the vmlinux pulled
from http://ddebs.ubuntu.com/pool/main/l/linux/ is compiled with gcc
version 5.3.1 which is the source of the issue.
Resolved this by pulling the kernel debug symbols package from:
deb http://ddebs.ubuntu.com/ -updates main
which indeed has the vmlinux compiled with gcc version 4.8.4.
But I got this error:
crash: invalid structure member offset: module_num_symtab
FILE: kernel.c LINE: 3049 FUNCTION: module_init()
after the update owning to couple of missing patches listed below:
commit 6f1f78e33474d00d5f261d7ed9d835c558b34d61
Author: Dave Anderson <anderson@redhat.com>
Date: Wed Jan 20 09:56:36 2016 -0500
Fix for the changes made to the kernel module structure introduced by
this kernel commit for Linux 4.5 and later kernels:
commit 7523e4dc5057e157212b4741abd6256e03404cf1
module: use a structure to encapsulate layout.
Without the patch, the crash session fails during initialization
with the error message: "crash: invalid structure member offset:
module_init_text_size".
(sebott@linux.vnet.ibm.com)
commit 098cdab16dfa6a85e9dad2cad604dee14ee15f66
Author: Dave Anderson <anderson@redhat.com>
Date: Fri Feb 12 14:32:53 2016 -0500
Fix for the changes made to the kernel module structure introduced by
this kernel commit for Linux 4.5 and later kernels:
commit 8244062ef1e54502ef55f54cced659913f244c3e
modules: fix longstanding /proc/kallsyms vs module insertion race.
Without the patch, the crash session fails during initialization
with the error message: "crash: invalid structure member offset:
module_num_symtab".
(anderson@redhat.com)
Applying the above patches on top of crash tool version 7.0.3-3ubuntu4.4,
was able to analyze the dump. The patches apply cleanly..
Thanks
Hari
== Comment: #8 - Naresh Bannoth - 2016-07-29 06:59:23 ==
Verified with 4.4.0-31-generic kernel.
on Applying the patches mentioned in the #C 6, it is working fine.
I took the Compiled Binary attached in the BUG and able to analyse the Dumpfile without any issues.
Thanks
Naresh
== Comment: #9 - Hari Krishna Bathini - 2016-07-29 07:47:25 ==
To sum this up, two things to make this work:
1. Install debug symbols package from here
deb http://ddebs.ubuntu.com/ trusty main restricted universe multiverse
deb http://ddebs.ubuntu.com/ trusty-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ trusty-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ trusty-proposed main restricted universe multiverse
instead of http://ddebs.ubuntu.com/pool/main/l/linux/ as the gcc versions don't match.
2. Use a crash binary that has the upstream patches 6f1f78 & 098cda
Thanks
Hari |
[ Test case ]
* trigger a kernel crash dump
* use crash tool to analyse it
* ... on ppc64el architecture
== Comment: #0 - NAVEED A. UPPINANGADY SALIH - 2016-07-01 02:41:07 ==
---Problem Description---
Unable to analyse vmcore/kernel dumpfile generated by 4.4.0-29-generic kernel with ddeb kernel debuginfo available in http://ddebs.ubuntu.com/pool/main/l/linux/
http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-4.4.0-29-generic-dbgsym_4.4.0-29.48_ppc64el.ddeb
---uname output---
4.4.0-29-generic #48~14.04.1-Ubuntu SMP Wed Jun 29 19:55:03 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
Machine Type = model : 8247-22L machine : PowerNV 8247-22L
---Steps to Reproduce---
# crash /usr/lib/debug/boot/vmlinux-4.4.0-29-generic /var/crash/201606300840/dump.201606300840
crash 7.0.3
...
This GDB was configured as "powerpc64le-unknown-linux-gnu"...
crash: invalid kernel virtual address: 61c2555c09c17994 type: "possible"
WARNING: cannot read cpu_possible_map
crash: invalid kernel virtual address: 8a0f6ddee20c4116 type: "present"
WARNING: cannot read cpu_present_map
crash: invalid kernel virtual address: 5a09344e320a1886 type: "online"
WARNING: cannot read cpu_online_map
WARNING: cannot read linux_banner string
crash: /usr/lib/debug/boot/vmlinux-4.4.0-29-generic and /var/crash/201606300840/dump.201606300840 do not match!
Userspace rpm: linux-image-4.4.0-29-generic-dbgsym_4.4.0-29.48_ppc64el.ddeb
== Comment: #5 - Naresh Bannoth - 2016-07-27 04:12:09 ==
The Latest kernel I am having is as follows
root@ltcalpine-lp6:~# uname -a
Linux ltcalpine-lp6 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:03:56 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
root@ltcalpine-lp6:~#
====>>> the issue same as posted originally.
...
This GDB was configured as "powerpc64le-unknown-linux-gnu"...
crash: invalid kernel virtual address: 61c2555c09c17994 type: "possible"
WARNING: cannot read cpu_possible_map
crash: invalid kernel virtual address: 8a0f6ddee20c4116 type: "present"
WARNING: cannot read cpu_present_map
crash: invalid kernel virtual address: 5a09344e320a1886 type: "online"
WARNING: cannot read cpu_online_map
WARNING: cannot read linux_banner string
crash: /usr/lib/debug/boot/vmlinux-4.4.0-31-generic and /var/crash/201607270248/dump.201607270248 do not match!
== Comment: #6 - Hari Krishna Bathini - 2016-07-28 11:49:31 ==
The kernel is compiled with gcc version 4.8.4 while the vmlinux pulled
from http://ddebs.ubuntu.com/pool/main/l/linux/ is compiled with gcc
version 5.3.1 which is the source of the issue.
Resolved this by pulling the kernel debug symbols package from:
deb http://ddebs.ubuntu.com/ -updates main
which indeed has the vmlinux compiled with gcc version 4.8.4.
But I got this error:
crash: invalid structure member offset: module_num_symtab
FILE: kernel.c LINE: 3049 FUNCTION: module_init()
after the update owning to couple of missing patches listed below:
commit 6f1f78e33474d00d5f261d7ed9d835c558b34d61
Author: Dave Anderson <anderson@redhat.com>
Date: Wed Jan 20 09:56:36 2016 -0500
Fix for the changes made to the kernel module structure introduced by
this kernel commit for Linux 4.5 and later kernels:
commit 7523e4dc5057e157212b4741abd6256e03404cf1
module: use a structure to encapsulate layout.
Without the patch, the crash session fails during initialization
with the error message: "crash: invalid structure member offset:
module_init_text_size".
(sebott@linux.vnet.ibm.com)
commit 098cdab16dfa6a85e9dad2cad604dee14ee15f66
Author: Dave Anderson <anderson@redhat.com>
Date: Fri Feb 12 14:32:53 2016 -0500
Fix for the changes made to the kernel module structure introduced by
this kernel commit for Linux 4.5 and later kernels:
commit 8244062ef1e54502ef55f54cced659913f244c3e
modules: fix longstanding /proc/kallsyms vs module insertion race.
Without the patch, the crash session fails during initialization
with the error message: "crash: invalid structure member offset:
module_num_symtab".
(anderson@redhat.com)
Applying the above patches on top of crash tool version 7.0.3-3ubuntu4.4,
was able to analyze the dump. The patches apply cleanly..
Thanks
Hari
== Comment: #8 - Naresh Bannoth - 2016-07-29 06:59:23 ==
Verified with 4.4.0-31-generic kernel.
on Applying the patches mentioned in the #C 6, it is working fine.
I took the Compiled Binary attached in the BUG and able to analyse the Dumpfile without any issues.
Thanks
Naresh
== Comment: #9 - Hari Krishna Bathini - 2016-07-29 07:47:25 ==
To sum this up, two things to make this work:
1. Install debug symbols package from here
deb http://ddebs.ubuntu.com/ trusty main restricted universe multiverse
deb http://ddebs.ubuntu.com/ trusty-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ trusty-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ trusty-proposed main restricted universe multiverse
instead of http://ddebs.ubuntu.com/pool/main/l/linux/ as the gcc versions don't match.
2. Use a crash binary that has the upstream patches 6f1f78 & 098cda
Thanks
Hari |
|
| 2016-12-08 15:24:51 |
Dimitri John Ledkov |
nominated for series |
|
Ubuntu Trusty |
|
| 2016-12-08 15:24:51 |
Dimitri John Ledkov |
bug task added |
|
crash (Ubuntu Trusty) |
|
| 2016-12-08 15:25:11 |
Dimitri John Ledkov |
crash (Ubuntu): status |
New |
Fix Released |
|
| 2016-12-08 15:25:14 |
Dimitri John Ledkov |
crash (Ubuntu Trusty): status |
New |
In Progress |
|
| 2016-12-08 15:25:23 |
Dimitri John Ledkov |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2016-12-08 15:25:29 |
Dimitri John Ledkov |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2016-12-08 15:25:43 |
Dimitri John Ledkov |
crash (Ubuntu Trusty): assignee |
|
Robert Hooker (sarvatt) |
|
| 2016-12-08 15:25:47 |
Dimitri John Ledkov |
crash (Ubuntu Trusty): importance |
Undecided |
High |
|
| 2016-12-09 08:23:18 |
Brian Murray |
crash (Ubuntu Trusty): status |
In Progress |
Fix Committed |
|
| 2016-12-09 08:23:22 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2016-12-09 08:23:25 |
Brian Murray |
tags |
architecture-ppc64le bugnameltc-143307 patch severity-critical targetmilestone-inin14045 |
architecture-ppc64le bugnameltc-143307 patch severity-critical targetmilestone-inin14045 verification-needed |
|
| 2017-01-08 20:10:47 |
Mathew Hodson |
crash (Ubuntu): importance |
Undecided |
High |
|
| 2017-03-14 13:55:20 |
Louis Bouchard |
tags |
architecture-ppc64le bugnameltc-143307 patch severity-critical targetmilestone-inin14045 verification-needed |
architecture-ppc64le bugnameltc-143307 patch severity-critical targetmilestone-inin14045 verification-done |
|
| 2017-03-15 17:20:55 |
Launchpad Janitor |
crash (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2017-03-15 17:21:03 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
