Activity log for bug #1765660

Date Who What changed Old value New value Message
2018-04-20 10:20:41 bugproxy bug added bug
2018-04-20 10:20:43 bugproxy tags architecture-ppc64le bugnameltc-166199 severity-high targetmilestone-inin1804
2018-04-20 10:20:44 bugproxy ubuntu: assignee Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
2018-04-20 10:20:47 bugproxy affects ubuntu crash (Ubuntu)
2018-04-20 10:24:21 Andrew Cloke bug task added ubuntu-power-systems
2018-04-20 10:24:33 Andrew Cloke ubuntu-power-systems: importance Undecided High
2018-04-20 10:26:00 Andrew Cloke ubuntu-power-systems: assignee Canonical Kernel Team (canonical-kernel-team)
2018-04-20 10:26:09 Andrew Cloke tags architecture-ppc64le bugnameltc-166199 severity-high targetmilestone-inin1804 architecture-ppc64le bugnameltc-166199 severity-high targetmilestone-inin1804 triage-g
2018-04-20 17:37:59 Frank Heimes ubuntu-power-systems: status New Triaged
2018-04-23 14:31:40 Manoj Iyer crash (Ubuntu): assignee Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) Canonical Kernel Team (canonical-kernel-team)
2018-04-23 14:31:43 Manoj Iyer crash (Ubuntu): importance Undecided High
2018-04-24 19:10:15 bugproxy bug watch added https://github.com/crash-utility/crash/issues/23
2018-05-15 13:31:25 Thadeu Lima de Souza Cascardo nominated for series Ubuntu Bionic
2018-05-15 13:31:25 Thadeu Lima de Souza Cascardo nominated for series Ubuntu Xenial
2018-05-15 13:31:25 Thadeu Lima de Souza Cascardo nominated for series Ubuntu Cosmic
2018-05-15 13:31:25 Thadeu Lima de Souza Cascardo nominated for series Ubuntu Artful
2018-05-15 13:33:43 Thadeu Lima de Souza Cascardo description == Comment: #0 - PAVITHRA R. PRAKASH <> - 2018-03-29 01:14:47 == ---Problem Description--- Ubuntu 18.04: "ipcs" command fails with error "invalid structure member offset" in crash prompt. ---Environment-- System Name : ltc-briggs2 Model/Type : P8 Platform : BML ---Uname output--- root@ltc-briggs2:~# uname -a Linux ltc-briggs2 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux ---Steps to reproduce-- 1. Configure kdump. 2. Trigger crash 3. run crash on captured dump ---Logs---- root@ltc-briggs2:~# dpkg -l|grep makedumpfile ii makedumpfile 1:1.6.3-1 ppc64el VMcore extraction tool root@ltc-briggs2:~# dpkg -l|grep crash ii apport 2.20.9-0ubuntu1 all automatically generate crash reports for debugging ii crash 7.2.1-1 ppc64el kernel debugging utility, allowing gdb like syntax ii kdump-tools 1:1.6.3-1 ppc64el scripts and tools for automating kdump (Linux crash dumps) ii python3-apport 2.20.9-0ubuntu1 all Python 3 library for Apport crash report handling root@ltc-briggs2:~# dpkg -l|grep kexec ii kexec-tools 1:2.0.16-1ubuntu1 ppc64el tools to support fast kexec reboots root@ltc-briggs2:~# .0-13-generic dump.201803272257 03272257# crash /usr/lib/debug/boot/vmlinux-4.15. crash 7.2.1 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu"... KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-13-generic DUMPFILE: dump.201803272257 [PARTIAL DUMP] CPUS: 160 DATE: Tue Mar 27 22:56:58 2018 UPTIME: 00:04:07 LOAD AVERAGE: 1.06, 0.53, 0.20 TASKS: 1734 NODENAME: ltc-briggs2 RELEASE: 4.15.0-13-generic VERSION: #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 MACHINE: ppc64le (2926 Mhz) MEMORY: 512 GB PANIC: "sysrq: SysRq : Trigger a crash" PID: 7420 COMMAND: "bash" TASK: c000003e56c7c600 [THREAD_INFO: c000003e56cb0000] CPU: 41 STATE: TASK_RUNNING (SYSRQ) crash> ? * files mach repeat timer alias foreach mod runq tree ascii fuser mount search union bt gdb net set vm btop help p sig vtop dev ipcs ps struct waitq dis irq pte swap whatis eval kmem ptob sym wr exit list ptov sys q extend log rd task crash version: 7.2.1 gdb version: 7.6 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 70374748930 => 703747482d8 => 703746e8e98 => 703745b1488 ipcs: invalid structure member offset: idr_top FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> q == Comment: #3 - PAVITHRA R. PRAKASH <> - 2018-04-09 00:36:52 == (In reply to comment #1) > Hi Pavithra, > > Have you checked in older kernel versions, whether "ipcs" command > was working ...I think, it is because there were too many changes > to kernel's IPC code over several kernel versions..possibly crash tool > don't have the updated changes w.r.t new kernel IPC code > > also please check if following "options" are not working: > >>> > char *help_ipcs[] = { > "ipcs", > "System V IPC facilities", > "[-smMq] [-n pid|task] [id | addr]", > > " This command provides information on the System V IPC facilities. With > no", > " arguments, the command will display kernel usage of all three > factilities.", > " ", > " -s show semaphore arrays.", > " -m show shared memory segments.", > " -M show shared memory segments with additional details.", > " -q show message queues.", > " id show the data associated with this resource ID.", > " addr show the data associated with this virtual address of a", > " shmid_kernel, sem_array or msq_queue.", > "", > " For kernels supporting namespaces, the -n option may be used to", > " display the IPC facilities with respect to the namespace of a", > " specified task:\n", > " -n pid a process PID.", > " -n task a hexadecimal task_struct pointer.", > >>> > > Thanks!! Issue is observed even with old kernel. KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-12-generic DUMPFILE: dump.201804090031 [PARTIAL DUMP] CPUS: 160 DATE: Mon Apr 9 00:30:53 2018 UPTIME: 00:04:24 LOAD AVERAGE: 2.75, 1.56, 0.64 TASKS: 1738 NODENAME: ltc-briggs2 RELEASE: 4.15.0-12-generic VERSION: #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018 MACHINE: ppc64le (2926 Mhz) MEMORY: 512 GB PANIC: "sysrq: SysRq : Trigger a crash" PID: 6995 COMMAND: "bash" TASK: c000003f70dcfd00 [THREAD_INFO: c000003f70e78000] CPU: 64 STATE: TASK_RUNNING (SYSRQ) crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 198c7e38930 => 198c7e382d8 => 198c7dd8e98 => 198c7ca1488 ipcs: invalid structure member offset: idr_top FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> --------------------------------------------------------------------------------------------------------------------------------- crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -s SEM_ARRAY KEY SEMID UID PERMS NSEMS ipcs: invalid structure member offset: idr_top FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c2ace0 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -m SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -M ipcs: invalid structure member offset: idr_top FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -q MSG_QUEUE KEY MSQID UID PERMS USED-BYTES MESSAGES (none allocated) [Impact] Some analysis won't be possible on new kernels (hwe-edge on xenial, for example; or the standard bionic kernel). [Test Case] The ipcs command has been run after the fix was applied to crash, on a live amd64 system, with success. [Regression Potential] Analysis of crashed kernels will require that the crash file be moved to a system where crash doesn't have the regression. The patch, however, should touch only the broken command, and has been tested. --------------------------------- == Comment: #0 - PAVITHRA R. PRAKASH <> - 2018-03-29 01:14:47 == ---Problem Description--- Ubuntu 18.04: "ipcs" command fails with error "invalid structure member offset" in crash prompt. ---Environment-- System Name : ltc-briggs2 Model/Type : P8 Platform : BML ---Uname output--- root@ltc-briggs2:~# uname -a Linux ltc-briggs2 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux ---Steps to reproduce-- 1. Configure kdump. 2. Trigger crash 3. run crash on captured dump ---Logs---- root@ltc-briggs2:~# dpkg -l|grep makedumpfile ii makedumpfile 1:1.6.3-1 ppc64el VMcore extraction tool root@ltc-briggs2:~# dpkg -l|grep crash ii apport 2.20.9-0ubuntu1 all automatically generate crash reports for debugging ii crash 7.2.1-1 ppc64el kernel debugging utility, allowing gdb like syntax ii kdump-tools 1:1.6.3-1 ppc64el scripts and tools for automating kdump (Linux crash dumps) ii python3-apport 2.20.9-0ubuntu1 all Python 3 library for Apport crash report handling root@ltc-briggs2:~# dpkg -l|grep kexec ii kexec-tools 1:2.0.16-1ubuntu1 ppc64el tools to support fast kexec reboots root@ltc-briggs2:~# .0-13-generic dump.201803272257 03272257# crash /usr/lib/debug/boot/vmlinux-4.15. crash 7.2.1 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu"...       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-13-generic     DUMPFILE: dump.201803272257 [PARTIAL DUMP]         CPUS: 160         DATE: Tue Mar 27 22:56:58 2018       UPTIME: 00:04:07 LOAD AVERAGE: 1.06, 0.53, 0.20        TASKS: 1734     NODENAME: ltc-briggs2      RELEASE: 4.15.0-13-generic      VERSION: #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 7420      COMMAND: "bash"         TASK: c000003e56c7c600 [THREAD_INFO: c000003e56cb0000]          CPU: 41        STATE: TASK_RUNNING (SYSRQ) crash> ? * files mach repeat timer alias foreach mod runq tree ascii fuser mount search union bt gdb net set vm btop help p sig vtop dev ipcs ps struct waitq dis irq pte swap whatis eval kmem ptob sym wr exit list ptov sys q extend log rd task crash version: 7.2.1 gdb version: 7.6 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 70374748930 => 703747482d8 => 703746e8e98 => 703745b1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> q == Comment: #3 - PAVITHRA R. PRAKASH <> - 2018-04-09 00:36:52 == (In reply to comment #1) > Hi Pavithra, > > Have you checked in older kernel versions, whether "ipcs" command > was working ...I think, it is because there were too many changes > to kernel's IPC code over several kernel versions..possibly crash tool > don't have the updated changes w.r.t new kernel IPC code > > also please check if following "options" are not working: > >>> > char *help_ipcs[] = { > "ipcs", > "System V IPC facilities", > "[-smMq] [-n pid|task] [id | addr]", > > " This command provides information on the System V IPC facilities. With > no", > " arguments, the command will display kernel usage of all three > factilities.", > " ", > " -s show semaphore arrays.", > " -m show shared memory segments.", > " -M show shared memory segments with additional details.", > " -q show message queues.", > " id show the data associated with this resource ID.", > " addr show the data associated with this virtual address of a", > " shmid_kernel, sem_array or msq_queue.", > "", > " For kernels supporting namespaces, the -n option may be used to", > " display the IPC facilities with respect to the namespace of a", > " specified task:\n", > " -n pid a process PID.", > " -n task a hexadecimal task_struct pointer.", > >>> > > Thanks!! Issue is observed even with old kernel.       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-12-generic     DUMPFILE: dump.201804090031 [PARTIAL DUMP]         CPUS: 160         DATE: Mon Apr 9 00:30:53 2018       UPTIME: 00:04:24 LOAD AVERAGE: 2.75, 1.56, 0.64        TASKS: 1738     NODENAME: ltc-briggs2      RELEASE: 4.15.0-12-generic      VERSION: #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 6995      COMMAND: "bash"         TASK: c000003f70dcfd00 [THREAD_INFO: c000003f70e78000]          CPU: 64        STATE: TASK_RUNNING (SYSRQ) crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 198c7e38930 => 198c7e382d8 => 198c7dd8e98 => 198c7ca1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> --------------------------------------------------------------------------------------------------------------------------------- crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -s SEM_ARRAY KEY SEMID UID PERMS NSEMS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c2ace0 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -m SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -M ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -q MSG_QUEUE KEY MSQID UID PERMS USED-BYTES MESSAGES (none allocated)
2018-05-17 19:12:42 Tyler Hicks bug task added crash (Ubuntu Cosmic)
2018-05-17 19:13:28 Tyler Hicks bug task added crash (Ubuntu Artful)
2018-05-17 19:13:34 Tyler Hicks bug task added crash (Ubuntu Bionic)
2018-05-17 19:13:40 Tyler Hicks bug task added crash (Ubuntu Xenial)
2018-05-17 23:03:47 Launchpad Janitor crash (Ubuntu Cosmic): status New Fix Released
2018-06-05 14:02:53 Thadeu Lima de Souza Cascardo crash (Ubuntu Bionic): status New In Progress
2018-06-05 14:02:59 Thadeu Lima de Souza Cascardo crash (Ubuntu Bionic): assignee Thadeu Lima de Souza Cascardo (cascardo)
2018-06-05 14:03:04 Thadeu Lima de Souza Cascardo crash (Ubuntu Bionic): importance Undecided High
2018-06-18 14:02:58 Manoj Iyer crash (Ubuntu Artful): status New Won't Fix
2018-06-18 14:03:02 Manoj Iyer crash (Ubuntu Xenial): status New Won't Fix
2018-06-18 14:03:09 Frank Heimes ubuntu-power-systems: status Triaged In Progress
2018-07-09 14:11:51 Manoj Iyer crash (Ubuntu Bionic): status In Progress Fix Released
2018-07-09 14:11:59 Manoj Iyer ubuntu-power-systems: status In Progress Fix Released
2018-08-02 12:40:51 Thadeu Lima de Souza Cascardo crash (Ubuntu Bionic): status Fix Released In Progress
2018-08-02 13:26:08 Manoj Iyer ubuntu-power-systems: status Fix Released In Progress
2018-08-07 16:01:28 Thadeu Lima de Souza Cascardo description [Impact] Some analysis won't be possible on new kernels (hwe-edge on xenial, for example; or the standard bionic kernel). [Test Case] The ipcs command has been run after the fix was applied to crash, on a live amd64 system, with success. [Regression Potential] Analysis of crashed kernels will require that the crash file be moved to a system where crash doesn't have the regression. The patch, however, should touch only the broken command, and has been tested. --------------------------------- == Comment: #0 - PAVITHRA R. PRAKASH <> - 2018-03-29 01:14:47 == ---Problem Description--- Ubuntu 18.04: "ipcs" command fails with error "invalid structure member offset" in crash prompt. ---Environment-- System Name : ltc-briggs2 Model/Type : P8 Platform : BML ---Uname output--- root@ltc-briggs2:~# uname -a Linux ltc-briggs2 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux ---Steps to reproduce-- 1. Configure kdump. 2. Trigger crash 3. run crash on captured dump ---Logs---- root@ltc-briggs2:~# dpkg -l|grep makedumpfile ii makedumpfile 1:1.6.3-1 ppc64el VMcore extraction tool root@ltc-briggs2:~# dpkg -l|grep crash ii apport 2.20.9-0ubuntu1 all automatically generate crash reports for debugging ii crash 7.2.1-1 ppc64el kernel debugging utility, allowing gdb like syntax ii kdump-tools 1:1.6.3-1 ppc64el scripts and tools for automating kdump (Linux crash dumps) ii python3-apport 2.20.9-0ubuntu1 all Python 3 library for Apport crash report handling root@ltc-briggs2:~# dpkg -l|grep kexec ii kexec-tools 1:2.0.16-1ubuntu1 ppc64el tools to support fast kexec reboots root@ltc-briggs2:~# .0-13-generic dump.201803272257 03272257# crash /usr/lib/debug/boot/vmlinux-4.15. crash 7.2.1 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu"...       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-13-generic     DUMPFILE: dump.201803272257 [PARTIAL DUMP]         CPUS: 160         DATE: Tue Mar 27 22:56:58 2018       UPTIME: 00:04:07 LOAD AVERAGE: 1.06, 0.53, 0.20        TASKS: 1734     NODENAME: ltc-briggs2      RELEASE: 4.15.0-13-generic      VERSION: #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 7420      COMMAND: "bash"         TASK: c000003e56c7c600 [THREAD_INFO: c000003e56cb0000]          CPU: 41        STATE: TASK_RUNNING (SYSRQ) crash> ? * files mach repeat timer alias foreach mod runq tree ascii fuser mount search union bt gdb net set vm btop help p sig vtop dev ipcs ps struct waitq dis irq pte swap whatis eval kmem ptob sym wr exit list ptov sys q extend log rd task crash version: 7.2.1 gdb version: 7.6 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 70374748930 => 703747482d8 => 703746e8e98 => 703745b1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> q == Comment: #3 - PAVITHRA R. PRAKASH <> - 2018-04-09 00:36:52 == (In reply to comment #1) > Hi Pavithra, > > Have you checked in older kernel versions, whether "ipcs" command > was working ...I think, it is because there were too many changes > to kernel's IPC code over several kernel versions..possibly crash tool > don't have the updated changes w.r.t new kernel IPC code > > also please check if following "options" are not working: > >>> > char *help_ipcs[] = { > "ipcs", > "System V IPC facilities", > "[-smMq] [-n pid|task] [id | addr]", > > " This command provides information on the System V IPC facilities. With > no", > " arguments, the command will display kernel usage of all three > factilities.", > " ", > " -s show semaphore arrays.", > " -m show shared memory segments.", > " -M show shared memory segments with additional details.", > " -q show message queues.", > " id show the data associated with this resource ID.", > " addr show the data associated with this virtual address of a", > " shmid_kernel, sem_array or msq_queue.", > "", > " For kernels supporting namespaces, the -n option may be used to", > " display the IPC facilities with respect to the namespace of a", > " specified task:\n", > " -n pid a process PID.", > " -n task a hexadecimal task_struct pointer.", > >>> > > Thanks!! Issue is observed even with old kernel.       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-12-generic     DUMPFILE: dump.201804090031 [PARTIAL DUMP]         CPUS: 160         DATE: Mon Apr 9 00:30:53 2018       UPTIME: 00:04:24 LOAD AVERAGE: 2.75, 1.56, 0.64        TASKS: 1738     NODENAME: ltc-briggs2      RELEASE: 4.15.0-12-generic      VERSION: #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 6995      COMMAND: "bash"         TASK: c000003f70dcfd00 [THREAD_INFO: c000003f70e78000]          CPU: 64        STATE: TASK_RUNNING (SYSRQ) crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 198c7e38930 => 198c7e382d8 => 198c7dd8e98 => 198c7ca1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> --------------------------------------------------------------------------------------------------------------------------------- crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -s SEM_ARRAY KEY SEMID UID PERMS NSEMS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c2ace0 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -m SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -M ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -q MSG_QUEUE KEY MSQID UID PERMS USED-BYTES MESSAGES (none allocated) [Impact] Some analysis won't be possible on new kernels. In this case, ipcs command could not be used with the bionic GA kernel on ppc64le. [Test Case] The ipcs command has been run after the fix was applied to crash, on a live amd64 system, with success. [Regression Potential] New crash versions may have bugs and some commands not work with older kernels. The smoke test helps a little, but more testing may be desirable. In case of regressions, analysis of kernel dumps will require that the crash file be moved to a system where crash doesn't have the regression. But live analysis won't be possible in that case. --------------------------------- == Comment: #0 - PAVITHRA R. PRAKASH <> - 2018-03-29 01:14:47 == ---Problem Description--- Ubuntu 18.04: "ipcs" command fails with error "invalid structure member offset" in crash prompt. ---Environment-- System Name : ltc-briggs2 Model/Type : P8 Platform : BML ---Uname output--- root@ltc-briggs2:~# uname -a Linux ltc-briggs2 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux ---Steps to reproduce-- 1. Configure kdump. 2. Trigger crash 3. run crash on captured dump ---Logs---- root@ltc-briggs2:~# dpkg -l|grep makedumpfile ii makedumpfile 1:1.6.3-1 ppc64el VMcore extraction tool root@ltc-briggs2:~# dpkg -l|grep crash ii apport 2.20.9-0ubuntu1 all automatically generate crash reports for debugging ii crash 7.2.1-1 ppc64el kernel debugging utility, allowing gdb like syntax ii kdump-tools 1:1.6.3-1 ppc64el scripts and tools for automating kdump (Linux crash dumps) ii python3-apport 2.20.9-0ubuntu1 all Python 3 library for Apport crash report handling root@ltc-briggs2:~# dpkg -l|grep kexec ii kexec-tools 1:2.0.16-1ubuntu1 ppc64el tools to support fast kexec reboots root@ltc-briggs2:~# .0-13-generic dump.201803272257 03272257# crash /usr/lib/debug/boot/vmlinux-4.15. crash 7.2.1 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu"...       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-13-generic     DUMPFILE: dump.201803272257 [PARTIAL DUMP]         CPUS: 160         DATE: Tue Mar 27 22:56:58 2018       UPTIME: 00:04:07 LOAD AVERAGE: 1.06, 0.53, 0.20        TASKS: 1734     NODENAME: ltc-briggs2      RELEASE: 4.15.0-13-generic      VERSION: #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 7420      COMMAND: "bash"         TASK: c000003e56c7c600 [THREAD_INFO: c000003e56cb0000]          CPU: 41        STATE: TASK_RUNNING (SYSRQ) crash> ? * files mach repeat timer alias foreach mod runq tree ascii fuser mount search union bt gdb net set vm btop help p sig vtop dev ipcs ps struct waitq dis irq pte swap whatis eval kmem ptob sym wr exit list ptov sys q extend log rd task crash version: 7.2.1 gdb version: 7.6 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 70374748930 => 703747482d8 => 703746e8e98 => 703745b1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> q == Comment: #3 - PAVITHRA R. PRAKASH <> - 2018-04-09 00:36:52 == (In reply to comment #1) > Hi Pavithra, > > Have you checked in older kernel versions, whether "ipcs" command > was working ...I think, it is because there were too many changes > to kernel's IPC code over several kernel versions..possibly crash tool > don't have the updated changes w.r.t new kernel IPC code > > also please check if following "options" are not working: > >>> > char *help_ipcs[] = { > "ipcs", > "System V IPC facilities", > "[-smMq] [-n pid|task] [id | addr]", > > " This command provides information on the System V IPC facilities. With > no", > " arguments, the command will display kernel usage of all three > factilities.", > " ", > " -s show semaphore arrays.", > " -m show shared memory segments.", > " -M show shared memory segments with additional details.", > " -q show message queues.", > " id show the data associated with this resource ID.", > " addr show the data associated with this virtual address of a", > " shmid_kernel, sem_array or msq_queue.", > "", > " For kernels supporting namespaces, the -n option may be used to", > " display the IPC facilities with respect to the namespace of a", > " specified task:\n", > " -n pid a process PID.", > " -n task a hexadecimal task_struct pointer.", > >>> > > Thanks!! Issue is observed even with old kernel.       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-12-generic     DUMPFILE: dump.201804090031 [PARTIAL DUMP]         CPUS: 160         DATE: Mon Apr 9 00:30:53 2018       UPTIME: 00:04:24 LOAD AVERAGE: 2.75, 1.56, 0.64        TASKS: 1738     NODENAME: ltc-briggs2      RELEASE: 4.15.0-12-generic      VERSION: #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 6995      COMMAND: "bash"         TASK: c000003f70dcfd00 [THREAD_INFO: c000003f70e78000]          CPU: 64        STATE: TASK_RUNNING (SYSRQ) crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 198c7e38930 => 198c7e382d8 => 198c7dd8e98 => 198c7ca1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> --------------------------------------------------------------------------------------------------------------------------------- crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -s SEM_ARRAY KEY SEMID UID PERMS NSEMS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c2ace0 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -m SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -M ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -q MSG_QUEUE KEY MSQID UID PERMS USED-BYTES MESSAGES (none allocated)
2018-08-10 18:20:00 Steve Langasek crash (Ubuntu Bionic): status In Progress Incomplete
2018-10-17 12:20:32 Thadeu Lima de Souza Cascardo description [Impact] Some analysis won't be possible on new kernels. In this case, ipcs command could not be used with the bionic GA kernel on ppc64le. [Test Case] The ipcs command has been run after the fix was applied to crash, on a live amd64 system, with success. [Regression Potential] New crash versions may have bugs and some commands not work with older kernels. The smoke test helps a little, but more testing may be desirable. In case of regressions, analysis of kernel dumps will require that the crash file be moved to a system where crash doesn't have the regression. But live analysis won't be possible in that case. --------------------------------- == Comment: #0 - PAVITHRA R. PRAKASH <> - 2018-03-29 01:14:47 == ---Problem Description--- Ubuntu 18.04: "ipcs" command fails with error "invalid structure member offset" in crash prompt. ---Environment-- System Name : ltc-briggs2 Model/Type : P8 Platform : BML ---Uname output--- root@ltc-briggs2:~# uname -a Linux ltc-briggs2 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux ---Steps to reproduce-- 1. Configure kdump. 2. Trigger crash 3. run crash on captured dump ---Logs---- root@ltc-briggs2:~# dpkg -l|grep makedumpfile ii makedumpfile 1:1.6.3-1 ppc64el VMcore extraction tool root@ltc-briggs2:~# dpkg -l|grep crash ii apport 2.20.9-0ubuntu1 all automatically generate crash reports for debugging ii crash 7.2.1-1 ppc64el kernel debugging utility, allowing gdb like syntax ii kdump-tools 1:1.6.3-1 ppc64el scripts and tools for automating kdump (Linux crash dumps) ii python3-apport 2.20.9-0ubuntu1 all Python 3 library for Apport crash report handling root@ltc-briggs2:~# dpkg -l|grep kexec ii kexec-tools 1:2.0.16-1ubuntu1 ppc64el tools to support fast kexec reboots root@ltc-briggs2:~# .0-13-generic dump.201803272257 03272257# crash /usr/lib/debug/boot/vmlinux-4.15. crash 7.2.1 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu"...       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-13-generic     DUMPFILE: dump.201803272257 [PARTIAL DUMP]         CPUS: 160         DATE: Tue Mar 27 22:56:58 2018       UPTIME: 00:04:07 LOAD AVERAGE: 1.06, 0.53, 0.20        TASKS: 1734     NODENAME: ltc-briggs2      RELEASE: 4.15.0-13-generic      VERSION: #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 7420      COMMAND: "bash"         TASK: c000003e56c7c600 [THREAD_INFO: c000003e56cb0000]          CPU: 41        STATE: TASK_RUNNING (SYSRQ) crash> ? * files mach repeat timer alias foreach mod runq tree ascii fuser mount search union bt gdb net set vm btop help p sig vtop dev ipcs ps struct waitq dis irq pte swap whatis eval kmem ptob sym wr exit list ptov sys q extend log rd task crash version: 7.2.1 gdb version: 7.6 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 70374748930 => 703747482d8 => 703746e8e98 => 703745b1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> q == Comment: #3 - PAVITHRA R. PRAKASH <> - 2018-04-09 00:36:52 == (In reply to comment #1) > Hi Pavithra, > > Have you checked in older kernel versions, whether "ipcs" command > was working ...I think, it is because there were too many changes > to kernel's IPC code over several kernel versions..possibly crash tool > don't have the updated changes w.r.t new kernel IPC code > > also please check if following "options" are not working: > >>> > char *help_ipcs[] = { > "ipcs", > "System V IPC facilities", > "[-smMq] [-n pid|task] [id | addr]", > > " This command provides information on the System V IPC facilities. With > no", > " arguments, the command will display kernel usage of all three > factilities.", > " ", > " -s show semaphore arrays.", > " -m show shared memory segments.", > " -M show shared memory segments with additional details.", > " -q show message queues.", > " id show the data associated with this resource ID.", > " addr show the data associated with this virtual address of a", > " shmid_kernel, sem_array or msq_queue.", > "", > " For kernels supporting namespaces, the -n option may be used to", > " display the IPC facilities with respect to the namespace of a", > " specified task:\n", > " -n pid a process PID.", > " -n task a hexadecimal task_struct pointer.", > >>> > > Thanks!! Issue is observed even with old kernel.       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-12-generic     DUMPFILE: dump.201804090031 [PARTIAL DUMP]         CPUS: 160         DATE: Mon Apr 9 00:30:53 2018       UPTIME: 00:04:24 LOAD AVERAGE: 2.75, 1.56, 0.64        TASKS: 1738     NODENAME: ltc-briggs2      RELEASE: 4.15.0-12-generic      VERSION: #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 6995      COMMAND: "bash"         TASK: c000003f70dcfd00 [THREAD_INFO: c000003f70e78000]          CPU: 64        STATE: TASK_RUNNING (SYSRQ) crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 198c7e38930 => 198c7e382d8 => 198c7dd8e98 => 198c7ca1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> --------------------------------------------------------------------------------------------------------------------------------- crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -s SEM_ARRAY KEY SEMID UID PERMS NSEMS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c2ace0 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -m SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -M ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -q MSG_QUEUE KEY MSQID UID PERMS USED-BYTES MESSAGES (none allocated) [Impact] Some analysis won't be possible on new kernels. In this case, ipcs command could not be used with the bionic GA kernel on ppc64le. [Test Case] The ipcs command has been run after the fix was applied to crash, on a live amd64 system, with success. More tests have been done on amd64 and ppc64el, including memory commands, bt. [Regression Potential] New crash versions may have bugs and some commands not work with older kernels. The smoke test helps a little, but more testing may be desirable. In case of regressions, analysis of kernel dumps will require that the crash file be moved to a system where crash doesn't have the regression. But live analysis won't be possible in that case. --------------------------------- == Comment: #0 - PAVITHRA R. PRAKASH <> - 2018-03-29 01:14:47 == ---Problem Description--- Ubuntu 18.04: "ipcs" command fails with error "invalid structure member offset" in crash prompt. ---Environment-- System Name : ltc-briggs2 Model/Type : P8 Platform : BML ---Uname output--- root@ltc-briggs2:~# uname -a Linux ltc-briggs2 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux ---Steps to reproduce-- 1. Configure kdump. 2. Trigger crash 3. run crash on captured dump ---Logs---- root@ltc-briggs2:~# dpkg -l|grep makedumpfile ii makedumpfile 1:1.6.3-1 ppc64el VMcore extraction tool root@ltc-briggs2:~# dpkg -l|grep crash ii apport 2.20.9-0ubuntu1 all automatically generate crash reports for debugging ii crash 7.2.1-1 ppc64el kernel debugging utility, allowing gdb like syntax ii kdump-tools 1:1.6.3-1 ppc64el scripts and tools for automating kdump (Linux crash dumps) ii python3-apport 2.20.9-0ubuntu1 all Python 3 library for Apport crash report handling root@ltc-briggs2:~# dpkg -l|grep kexec ii kexec-tools 1:2.0.16-1ubuntu1 ppc64el tools to support fast kexec reboots root@ltc-briggs2:~# .0-13-generic dump.201803272257 03272257# crash /usr/lib/debug/boot/vmlinux-4.15. crash 7.2.1 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu"...       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-13-generic     DUMPFILE: dump.201803272257 [PARTIAL DUMP]         CPUS: 160         DATE: Tue Mar 27 22:56:58 2018       UPTIME: 00:04:07 LOAD AVERAGE: 1.06, 0.53, 0.20        TASKS: 1734     NODENAME: ltc-briggs2      RELEASE: 4.15.0-13-generic      VERSION: #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 7420      COMMAND: "bash"         TASK: c000003e56c7c600 [THREAD_INFO: c000003e56cb0000]          CPU: 41        STATE: TASK_RUNNING (SYSRQ) crash> ? * files mach repeat timer alias foreach mod runq tree ascii fuser mount search union bt gdb net set vm btop help p sig vtop dev ipcs ps struct waitq dis irq pte swap whatis eval kmem ptob sym wr exit list ptov sys q extend log rd task crash version: 7.2.1 gdb version: 7.6 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 70374748930 => 703747482d8 => 703746e8e98 => 703745b1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> q == Comment: #3 - PAVITHRA R. PRAKASH <> - 2018-04-09 00:36:52 == (In reply to comment #1) > Hi Pavithra, > > Have you checked in older kernel versions, whether "ipcs" command > was working ...I think, it is because there were too many changes > to kernel's IPC code over several kernel versions..possibly crash tool > don't have the updated changes w.r.t new kernel IPC code > > also please check if following "options" are not working: > >>> > char *help_ipcs[] = { > "ipcs", > "System V IPC facilities", > "[-smMq] [-n pid|task] [id | addr]", > > " This command provides information on the System V IPC facilities. With > no", > " arguments, the command will display kernel usage of all three > factilities.", > " ", > " -s show semaphore arrays.", > " -m show shared memory segments.", > " -M show shared memory segments with additional details.", > " -q show message queues.", > " id show the data associated with this resource ID.", > " addr show the data associated with this virtual address of a", > " shmid_kernel, sem_array or msq_queue.", > "", > " For kernels supporting namespaces, the -n option may be used to", > " display the IPC facilities with respect to the namespace of a", > " specified task:\n", > " -n pid a process PID.", > " -n task a hexadecimal task_struct pointer.", > >>> > > Thanks!! Issue is observed even with old kernel.       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-12-generic     DUMPFILE: dump.201804090031 [PARTIAL DUMP]         CPUS: 160         DATE: Mon Apr 9 00:30:53 2018       UPTIME: 00:04:24 LOAD AVERAGE: 2.75, 1.56, 0.64        TASKS: 1738     NODENAME: ltc-briggs2      RELEASE: 4.15.0-12-generic      VERSION: #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 6995      COMMAND: "bash"         TASK: c000003f70dcfd00 [THREAD_INFO: c000003f70e78000]          CPU: 64        STATE: TASK_RUNNING (SYSRQ) crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 198c7e38930 => 198c7e382d8 => 198c7dd8e98 => 198c7ca1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> --------------------------------------------------------------------------------------------------------------------------------- crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -s SEM_ARRAY KEY SEMID UID PERMS NSEMS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c2ace0 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -m SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -M ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -q MSG_QUEUE KEY MSQID UID PERMS USED-BYTES MESSAGES (none allocated)
2018-11-12 19:44:52 Terry Rudd bug added subscriber Terry Rudd
2018-11-20 18:17:36 Dimitri John Ledkov crash (Ubuntu Bionic): status Incomplete Confirmed
2018-12-18 13:35:12 Thadeu Lima de Souza Cascardo bug added subscriber Ubuntu Stable Release Updates Team
2018-12-18 16:19:12 Brian Murray description [Impact] Some analysis won't be possible on new kernels. In this case, ipcs command could not be used with the bionic GA kernel on ppc64le. [Test Case] The ipcs command has been run after the fix was applied to crash, on a live amd64 system, with success. More tests have been done on amd64 and ppc64el, including memory commands, bt. [Regression Potential] New crash versions may have bugs and some commands not work with older kernels. The smoke test helps a little, but more testing may be desirable. In case of regressions, analysis of kernel dumps will require that the crash file be moved to a system where crash doesn't have the regression. But live analysis won't be possible in that case. --------------------------------- == Comment: #0 - PAVITHRA R. PRAKASH <> - 2018-03-29 01:14:47 == ---Problem Description--- Ubuntu 18.04: "ipcs" command fails with error "invalid structure member offset" in crash prompt. ---Environment-- System Name : ltc-briggs2 Model/Type : P8 Platform : BML ---Uname output--- root@ltc-briggs2:~# uname -a Linux ltc-briggs2 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux ---Steps to reproduce-- 1. Configure kdump. 2. Trigger crash 3. run crash on captured dump ---Logs---- root@ltc-briggs2:~# dpkg -l|grep makedumpfile ii makedumpfile 1:1.6.3-1 ppc64el VMcore extraction tool root@ltc-briggs2:~# dpkg -l|grep crash ii apport 2.20.9-0ubuntu1 all automatically generate crash reports for debugging ii crash 7.2.1-1 ppc64el kernel debugging utility, allowing gdb like syntax ii kdump-tools 1:1.6.3-1 ppc64el scripts and tools for automating kdump (Linux crash dumps) ii python3-apport 2.20.9-0ubuntu1 all Python 3 library for Apport crash report handling root@ltc-briggs2:~# dpkg -l|grep kexec ii kexec-tools 1:2.0.16-1ubuntu1 ppc64el tools to support fast kexec reboots root@ltc-briggs2:~# .0-13-generic dump.201803272257 03272257# crash /usr/lib/debug/boot/vmlinux-4.15. crash 7.2.1 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu"...       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-13-generic     DUMPFILE: dump.201803272257 [PARTIAL DUMP]         CPUS: 160         DATE: Tue Mar 27 22:56:58 2018       UPTIME: 00:04:07 LOAD AVERAGE: 1.06, 0.53, 0.20        TASKS: 1734     NODENAME: ltc-briggs2      RELEASE: 4.15.0-13-generic      VERSION: #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 7420      COMMAND: "bash"         TASK: c000003e56c7c600 [THREAD_INFO: c000003e56cb0000]          CPU: 41        STATE: TASK_RUNNING (SYSRQ) crash> ? * files mach repeat timer alias foreach mod runq tree ascii fuser mount search union bt gdb net set vm btop help p sig vtop dev ipcs ps struct waitq dis irq pte swap whatis eval kmem ptob sym wr exit list ptov sys q extend log rd task crash version: 7.2.1 gdb version: 7.6 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 70374748930 => 703747482d8 => 703746e8e98 => 703745b1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> q == Comment: #3 - PAVITHRA R. PRAKASH <> - 2018-04-09 00:36:52 == (In reply to comment #1) > Hi Pavithra, > > Have you checked in older kernel versions, whether "ipcs" command > was working ...I think, it is because there were too many changes > to kernel's IPC code over several kernel versions..possibly crash tool > don't have the updated changes w.r.t new kernel IPC code > > also please check if following "options" are not working: > >>> > char *help_ipcs[] = { > "ipcs", > "System V IPC facilities", > "[-smMq] [-n pid|task] [id | addr]", > > " This command provides information on the System V IPC facilities. With > no", > " arguments, the command will display kernel usage of all three > factilities.", > " ", > " -s show semaphore arrays.", > " -m show shared memory segments.", > " -M show shared memory segments with additional details.", > " -q show message queues.", > " id show the data associated with this resource ID.", > " addr show the data associated with this virtual address of a", > " shmid_kernel, sem_array or msq_queue.", > "", > " For kernels supporting namespaces, the -n option may be used to", > " display the IPC facilities with respect to the namespace of a", > " specified task:\n", > " -n pid a process PID.", > " -n task a hexadecimal task_struct pointer.", > >>> > > Thanks!! Issue is observed even with old kernel.       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-12-generic     DUMPFILE: dump.201804090031 [PARTIAL DUMP]         CPUS: 160         DATE: Mon Apr 9 00:30:53 2018       UPTIME: 00:04:24 LOAD AVERAGE: 2.75, 1.56, 0.64        TASKS: 1738     NODENAME: ltc-briggs2      RELEASE: 4.15.0-12-generic      VERSION: #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 6995      COMMAND: "bash"         TASK: c000003f70dcfd00 [THREAD_INFO: c000003f70e78000]          CPU: 64        STATE: TASK_RUNNING (SYSRQ) crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 198c7e38930 => 198c7e382d8 => 198c7dd8e98 => 198c7ca1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> --------------------------------------------------------------------------------------------------------------------------------- crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -s SEM_ARRAY KEY SEMID UID PERMS NSEMS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c2ace0 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -m SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -M ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -q MSG_QUEUE KEY MSQID UID PERMS USED-BYTES MESSAGES (none allocated) [Impact] Some analysis won't be possible on new kernels. In this case, ipcs command could not be used with the bionic GA kernel on ppc64le. [Test Case] The ipcs command has been run after the fix was applied to crash, on a live amd64 system, with success. More tests will be done with the version of the package from -proposed on amd64 and ppc64el, including memory commands, and bt. [Regression Potential] New crash versions may have bugs and some commands not work with older kernels. The smoke test helps a little, but more testing is desirable. In case of regressions, analysis of kernel dumps will require that the crash file be moved to a system where crash doesn't have the regression. But live analysis won't be possible in that case. --------------------------------- == Comment: #0 - PAVITHRA R. PRAKASH <> - 2018-03-29 01:14:47 == ---Problem Description--- Ubuntu 18.04: "ipcs" command fails with error "invalid structure member offset" in crash prompt. ---Environment-- System Name : ltc-briggs2 Model/Type : P8 Platform : BML ---Uname output--- root@ltc-briggs2:~# uname -a Linux ltc-briggs2 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux ---Steps to reproduce-- 1. Configure kdump. 2. Trigger crash 3. run crash on captured dump ---Logs---- root@ltc-briggs2:~# dpkg -l|grep makedumpfile ii makedumpfile 1:1.6.3-1 ppc64el VMcore extraction tool root@ltc-briggs2:~# dpkg -l|grep crash ii apport 2.20.9-0ubuntu1 all automatically generate crash reports for debugging ii crash 7.2.1-1 ppc64el kernel debugging utility, allowing gdb like syntax ii kdump-tools 1:1.6.3-1 ppc64el scripts and tools for automating kdump (Linux crash dumps) ii python3-apport 2.20.9-0ubuntu1 all Python 3 library for Apport crash report handling root@ltc-briggs2:~# dpkg -l|grep kexec ii kexec-tools 1:2.0.16-1ubuntu1 ppc64el tools to support fast kexec reboots root@ltc-briggs2:~# .0-13-generic dump.201803272257 03272257# crash /usr/lib/debug/boot/vmlinux-4.15. crash 7.2.1 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu"...       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-13-generic     DUMPFILE: dump.201803272257 [PARTIAL DUMP]         CPUS: 160         DATE: Tue Mar 27 22:56:58 2018       UPTIME: 00:04:07 LOAD AVERAGE: 1.06, 0.53, 0.20        TASKS: 1734     NODENAME: ltc-briggs2      RELEASE: 4.15.0-13-generic      VERSION: #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 7420      COMMAND: "bash"         TASK: c000003e56c7c600 [THREAD_INFO: c000003e56cb0000]          CPU: 41        STATE: TASK_RUNNING (SYSRQ) crash> ? * files mach repeat timer alias foreach mod runq tree ascii fuser mount search union bt gdb net set vm btop help p sig vtop dev ipcs ps struct waitq dis irq pte swap whatis eval kmem ptob sym wr exit list ptov sys q extend log rd task crash version: 7.2.1 gdb version: 7.6 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 70374748930 => 703747482d8 => 703746e8e98 => 703745b1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> q == Comment: #3 - PAVITHRA R. PRAKASH <> - 2018-04-09 00:36:52 == (In reply to comment #1) > Hi Pavithra, > > Have you checked in older kernel versions, whether "ipcs" command > was working ...I think, it is because there were too many changes > to kernel's IPC code over several kernel versions..possibly crash tool > don't have the updated changes w.r.t new kernel IPC code > > also please check if following "options" are not working: > >>> > char *help_ipcs[] = { > "ipcs", > "System V IPC facilities", > "[-smMq] [-n pid|task] [id | addr]", > > " This command provides information on the System V IPC facilities. With > no", > " arguments, the command will display kernel usage of all three > factilities.", > " ", > " -s show semaphore arrays.", > " -m show shared memory segments.", > " -M show shared memory segments with additional details.", > " -q show message queues.", > " id show the data associated with this resource ID.", > " addr show the data associated with this virtual address of a", > " shmid_kernel, sem_array or msq_queue.", > "", > " For kernels supporting namespaces, the -n option may be used to", > " display the IPC facilities with respect to the namespace of a", > " specified task:\n", > " -n pid a process PID.", > " -n task a hexadecimal task_struct pointer.", > >>> > > Thanks!! Issue is observed even with old kernel.       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-12-generic     DUMPFILE: dump.201804090031 [PARTIAL DUMP]         CPUS: 160         DATE: Mon Apr 9 00:30:53 2018       UPTIME: 00:04:24 LOAD AVERAGE: 2.75, 1.56, 0.64        TASKS: 1738     NODENAME: ltc-briggs2      RELEASE: 4.15.0-12-generic      VERSION: #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 6995      COMMAND: "bash"         TASK: c000003f70dcfd00 [THREAD_INFO: c000003f70e78000]          CPU: 64        STATE: TASK_RUNNING (SYSRQ) crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 198c7e38930 => 198c7e382d8 => 198c7dd8e98 => 198c7ca1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> --------------------------------------------------------------------------------------------------------------------------------- crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -s SEM_ARRAY KEY SEMID UID PERMS NSEMS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c2ace0 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -m SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -M ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -q MSG_QUEUE KEY MSQID UID PERMS USED-BYTES MESSAGES (none allocated)
2019-02-15 14:25:44 Thadeu Lima de Souza Cascardo description [Impact] Some analysis won't be possible on new kernels. In this case, ipcs command could not be used with the bionic GA kernel on ppc64le. [Test Case] The ipcs command has been run after the fix was applied to crash, on a live amd64 system, with success. More tests will be done with the version of the package from -proposed on amd64 and ppc64el, including memory commands, and bt. [Regression Potential] New crash versions may have bugs and some commands not work with older kernels. The smoke test helps a little, but more testing is desirable. In case of regressions, analysis of kernel dumps will require that the crash file be moved to a system where crash doesn't have the regression. But live analysis won't be possible in that case. --------------------------------- == Comment: #0 - PAVITHRA R. PRAKASH <> - 2018-03-29 01:14:47 == ---Problem Description--- Ubuntu 18.04: "ipcs" command fails with error "invalid structure member offset" in crash prompt. ---Environment-- System Name : ltc-briggs2 Model/Type : P8 Platform : BML ---Uname output--- root@ltc-briggs2:~# uname -a Linux ltc-briggs2 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux ---Steps to reproduce-- 1. Configure kdump. 2. Trigger crash 3. run crash on captured dump ---Logs---- root@ltc-briggs2:~# dpkg -l|grep makedumpfile ii makedumpfile 1:1.6.3-1 ppc64el VMcore extraction tool root@ltc-briggs2:~# dpkg -l|grep crash ii apport 2.20.9-0ubuntu1 all automatically generate crash reports for debugging ii crash 7.2.1-1 ppc64el kernel debugging utility, allowing gdb like syntax ii kdump-tools 1:1.6.3-1 ppc64el scripts and tools for automating kdump (Linux crash dumps) ii python3-apport 2.20.9-0ubuntu1 all Python 3 library for Apport crash report handling root@ltc-briggs2:~# dpkg -l|grep kexec ii kexec-tools 1:2.0.16-1ubuntu1 ppc64el tools to support fast kexec reboots root@ltc-briggs2:~# .0-13-generic dump.201803272257 03272257# crash /usr/lib/debug/boot/vmlinux-4.15. crash 7.2.1 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu"...       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-13-generic     DUMPFILE: dump.201803272257 [PARTIAL DUMP]         CPUS: 160         DATE: Tue Mar 27 22:56:58 2018       UPTIME: 00:04:07 LOAD AVERAGE: 1.06, 0.53, 0.20        TASKS: 1734     NODENAME: ltc-briggs2      RELEASE: 4.15.0-13-generic      VERSION: #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 7420      COMMAND: "bash"         TASK: c000003e56c7c600 [THREAD_INFO: c000003e56cb0000]          CPU: 41        STATE: TASK_RUNNING (SYSRQ) crash> ? * files mach repeat timer alias foreach mod runq tree ascii fuser mount search union bt gdb net set vm btop help p sig vtop dev ipcs ps struct waitq dis irq pte swap whatis eval kmem ptob sym wr exit list ptov sys q extend log rd task crash version: 7.2.1 gdb version: 7.6 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 70374748930 => 703747482d8 => 703746e8e98 => 703745b1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> q == Comment: #3 - PAVITHRA R. PRAKASH <> - 2018-04-09 00:36:52 == (In reply to comment #1) > Hi Pavithra, > > Have you checked in older kernel versions, whether "ipcs" command > was working ...I think, it is because there were too many changes > to kernel's IPC code over several kernel versions..possibly crash tool > don't have the updated changes w.r.t new kernel IPC code > > also please check if following "options" are not working: > >>> > char *help_ipcs[] = { > "ipcs", > "System V IPC facilities", > "[-smMq] [-n pid|task] [id | addr]", > > " This command provides information on the System V IPC facilities. With > no", > " arguments, the command will display kernel usage of all three > factilities.", > " ", > " -s show semaphore arrays.", > " -m show shared memory segments.", > " -M show shared memory segments with additional details.", > " -q show message queues.", > " id show the data associated with this resource ID.", > " addr show the data associated with this virtual address of a", > " shmid_kernel, sem_array or msq_queue.", > "", > " For kernels supporting namespaces, the -n option may be used to", > " display the IPC facilities with respect to the namespace of a", > " specified task:\n", > " -n pid a process PID.", > " -n task a hexadecimal task_struct pointer.", > >>> > > Thanks!! Issue is observed even with old kernel.       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-12-generic     DUMPFILE: dump.201804090031 [PARTIAL DUMP]         CPUS: 160         DATE: Mon Apr 9 00:30:53 2018       UPTIME: 00:04:24 LOAD AVERAGE: 2.75, 1.56, 0.64        TASKS: 1738     NODENAME: ltc-briggs2      RELEASE: 4.15.0-12-generic      VERSION: #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 6995      COMMAND: "bash"         TASK: c000003f70dcfd00 [THREAD_INFO: c000003f70e78000]          CPU: 64        STATE: TASK_RUNNING (SYSRQ) crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 198c7e38930 => 198c7e382d8 => 198c7dd8e98 => 198c7ca1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> --------------------------------------------------------------------------------------------------------------------------------- crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -s SEM_ARRAY KEY SEMID UID PERMS NSEMS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c2ace0 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -m SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -M ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -q MSG_QUEUE KEY MSQID UID PERMS USED-BYTES MESSAGES (none allocated) [Impact] ipcs command could not be used with the bionic GA kernel on ppc64le. [Test Case] The ipcs command has been run after the fix was applied to crash, on live amd64, ppc64el and s390x system. In fact, the new upload introduces a test to verify this bug does not happen again. [Regression Potential] We have only introduced the fix for this specific bug, also adding a test for it. It's less likely to be more broken than the current version on the release as an entire new upstream version could be. --------------------------------- == Comment: #0 - PAVITHRA R. PRAKASH <> - 2018-03-29 01:14:47 == ---Problem Description--- Ubuntu 18.04: "ipcs" command fails with error "invalid structure member offset" in crash prompt. ---Environment-- System Name : ltc-briggs2 Model/Type : P8 Platform : BML ---Uname output--- root@ltc-briggs2:~# uname -a Linux ltc-briggs2 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux ---Steps to reproduce-- 1. Configure kdump. 2. Trigger crash 3. run crash on captured dump ---Logs---- root@ltc-briggs2:~# dpkg -l|grep makedumpfile ii makedumpfile 1:1.6.3-1 ppc64el VMcore extraction tool root@ltc-briggs2:~# dpkg -l|grep crash ii apport 2.20.9-0ubuntu1 all automatically generate crash reports for debugging ii crash 7.2.1-1 ppc64el kernel debugging utility, allowing gdb like syntax ii kdump-tools 1:1.6.3-1 ppc64el scripts and tools for automating kdump (Linux crash dumps) ii python3-apport 2.20.9-0ubuntu1 all Python 3 library for Apport crash report handling root@ltc-briggs2:~# dpkg -l|grep kexec ii kexec-tools 1:2.0.16-1ubuntu1 ppc64el tools to support fast kexec reboots root@ltc-briggs2:~# .0-13-generic dump.201803272257 03272257# crash /usr/lib/debug/boot/vmlinux-4.15. crash 7.2.1 Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu"...       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-13-generic     DUMPFILE: dump.201803272257 [PARTIAL DUMP]         CPUS: 160         DATE: Tue Mar 27 22:56:58 2018       UPTIME: 00:04:07 LOAD AVERAGE: 1.06, 0.53, 0.20        TASKS: 1734     NODENAME: ltc-briggs2      RELEASE: 4.15.0-13-generic      VERSION: #14-Ubuntu SMP Sat Mar 17 13:43:15 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 7420      COMMAND: "bash"         TASK: c000003e56c7c600 [THREAD_INFO: c000003e56cb0000]          CPU: 41        STATE: TASK_RUNNING (SYSRQ) crash> ? * files mach repeat timer alias foreach mod runq tree ascii fuser mount search union bt gdb net set vm btop help p sig vtop dev ipcs ps struct waitq dis irq pte swap whatis eval kmem ptob sym wr exit list ptov sys q extend log rd task crash version: 7.2.1 gdb version: 7.6 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 70374748930 => 703747482d8 => 703746e8e98 => 703745b1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> q == Comment: #3 - PAVITHRA R. PRAKASH <> - 2018-04-09 00:36:52 == (In reply to comment #1) > Hi Pavithra, > > Have you checked in older kernel versions, whether "ipcs" command > was working ...I think, it is because there were too many changes > to kernel's IPC code over several kernel versions..possibly crash tool > don't have the updated changes w.r.t new kernel IPC code > > also please check if following "options" are not working: > >>> > char *help_ipcs[] = { > "ipcs", > "System V IPC facilities", > "[-smMq] [-n pid|task] [id | addr]", > > " This command provides information on the System V IPC facilities. With > no", > " arguments, the command will display kernel usage of all three > factilities.", > " ", > " -s show semaphore arrays.", > " -m show shared memory segments.", > " -M show shared memory segments with additional details.", > " -q show message queues.", > " id show the data associated with this resource ID.", > " addr show the data associated with this virtual address of a", > " shmid_kernel, sem_array or msq_queue.", > "", > " For kernels supporting namespaces, the -n option may be used to", > " display the IPC facilities with respect to the namespace of a", > " specified task:\n", > " -n pid a process PID.", > " -n task a hexadecimal task_struct pointer.", > >>> > > Thanks!! Issue is observed even with old kernel.       KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-12-generic     DUMPFILE: dump.201804090031 [PARTIAL DUMP]         CPUS: 160         DATE: Mon Apr 9 00:30:53 2018       UPTIME: 00:04:24 LOAD AVERAGE: 2.75, 1.56, 0.64        TASKS: 1738     NODENAME: ltc-briggs2      RELEASE: 4.15.0-12-generic      VERSION: #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018      MACHINE: ppc64le (2926 Mhz)       MEMORY: 512 GB        PANIC: "sysrq: SysRq : Trigger a crash"          PID: 6995      COMMAND: "bash"         TASK: c000003f70dcfd00 [THREAD_INFO: c000003f70e78000]          CPU: 64        STATE: TASK_RUNNING (SYSRQ) crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 198c7e38930 => 198c7e382d8 => 198c7dd8e98 => 198c7ca1488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> --------------------------------------------------------------------------------------------------------------------------------- crash> ipcs SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -s SEM_ARRAY KEY SEMID UID PERMS NSEMS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c2ace0 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -m SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -M ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() [/usr/bin/crash] error trace: 69b16c28930 => 69b16c282d8 => 69b16bc8e98 => 69b16a91488 ipcs: invalid structure member offset: idr_top       FILE: ipcs.c LINE: 628 FUNCTION: idr_find() crash> ipcs -q MSG_QUEUE KEY MSQID UID PERMS USED-BYTES MESSAGES (none allocated)
2019-02-18 07:23:53 Andy Whitcroft crash (Ubuntu Bionic): status Confirmed Fix Committed
2019-02-18 07:23:58 Andy Whitcroft bug added subscriber SRU Verification
2019-02-18 07:24:01 Andy Whitcroft tags architecture-ppc64le bugnameltc-166199 severity-high targetmilestone-inin1804 triage-g architecture-ppc64le bugnameltc-166199 severity-high targetmilestone-inin1804 triage-g verification-needed verification-needed-bionic
2019-02-18 10:16:18 Manoj Iyer ubuntu-power-systems: status In Progress Fix Committed
2019-02-18 10:39:41 bugproxy tags architecture-ppc64le bugnameltc-166199 severity-high targetmilestone-inin1804 triage-g verification-needed verification-needed-bionic architecture-ppc64le bugnameltc-166199 severity-high targetmilestone-inin1804 triage-g verification-done verification-done-bionic
2019-02-25 11:21:36 Launchpad Janitor crash (Ubuntu Bionic): status Fix Committed Fix Released
2019-02-25 11:21:40 Łukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team
2019-02-26 08:19:13 Andrew Cloke ubuntu-power-systems: status Fix Committed Fix Released