Ubuntu 16.04.4: crash can not open dumps: "crash: invalid structure member offset: task_struct_pids"

[bugproxy]

Ubuntu 16.04 on IBM z: crash can not open dumps gathered on 4.13.0-39
crash: invalid structure member offset: task_struct_pids
FILE: task.c LINE: 2102 FUNCTION: refresh_hlist_task_table_v3()

---uname output---
Linux fvtc2i 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:28 UTC 2018 s390x s390x s390x GNU/Linux

---Additional Hardware Info---
none

Machine Type = lpar type 3906, model 400 LM2

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 crash vmlinux-4.13.0-39-generic dump.201806150854
or
crash vmlinux-4.13.0-39-generic

Userspace tool common name: crash

The userspace tool has the following bit modes: 64

Userspace rpm: crash,crash-dbgsym 7.1.4-1ubuntu4.2 s390x

Userspace tool obtained from project website: na

We came across this issue while trying to extract traces from a vmcore gathered on one of our Ubuntu 16.04 nodes running HWE kernel (4.13)

It seems crash cannot open this dump

# crash vmlinux-4.13.0-39-generic dump.201806150854

crash 7.1.4
Copyright (C) 2002-2015 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "s390x-ibm-linux-gnu"...

WARNING: cannot determine MAX_PHYSMEM_BITS
please wait... (gathering task table data)
crash: invalid structure member offset: task_struct_pids
       FILE: task.c LINE: 2102 FUNCTION: refresh_hlist_task_table_v3()

[/usr/bin/crash] error trace: 116ed1ed8 => 116ec4b14 => 116f25eec => 116f25e48

this issue is 100% reproducible. An attempt to analyze the live system is failing with exact the same error.

# crash vmlinux-4.13.0-39-generic

crash 7.1.4
...
WARNING: cannot determine MAX_PHYSMEM_BITS

crash: invalid structure member offset: task_struct_pids
       FILE: task.c LINE: 2102 FUNCTION: refresh_hlist_task_table_v3()

[/usr/bin/crash] error trace: 1260d1ed8 => 1260c4b14 => 126125eec => 126125e48

it seems that crash version 7.2.1 shipped with Ubuntu 18.04 can handle dumps taken on kernel 4.13, so we have at least a workaround.

[Joseph Salisbury (jsalisbury)]

This was opened against the 4.13 kernel, which is now EOL. Do you also see this with other kernel versions?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-08-15 13:02 EDT-------
yes, on recent 16.04.5 crash also can not handle dumps generated with HWE kernel (4.15 in this case)

# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"

# strings dump.201808141812 | grep -m1 OSRELEASE
OSRELEASE=4.15.0-30-generic

# crash /usr/lib/debug/boot/vmlinux-4.15.0-30-generic dump.201808141812

crash 7.1.4
Copyright (C) 2002-2015 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "s390x-ibm-linux-gnu"...

WARNING: cannot determine MAX_PHYSMEM_BITS
please wait... (gathering task table data)
crash: cannot resolve "init_task_union" <<< it fails after this message

[Thadeu Lima de Souza Cascardo (cascardo)]

Can you test the crash package on ppa:cascardo/kdump2 ?

Thanks.
Cascardo.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-09-24 02:57 EDT-------
the crash version from ppa:cascardo/kdump2 works

# apt show crash
Package: crash
Version: 7.2.3+real-1~16.04.1
...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-11-05 03:56 EDT-------
@Canonical: Can we close this bug? The ppa-package works...

[Thadeu Lima de Souza Cascardo (cascardo)]

The version in xenial-updates matches the version from the ppa that was tested. This bug can be closed for xenial.

Thanks.
Cascardo.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-11-05 09:01 EDT-------
IBM Bugzilla status -> closed, Fix available by Canonical