[PPA] 4.4 regression: kernel panic on reboot

Bug #1523586 reported by Martin Pitt on 2015-12-07
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
High
Unassigned
Xenial
High
Unassigned

Bug Description

While testing the unstable kernel PPA with linux-image-4.4.0-0-generic on ppc64el, autopkgtests often fail with this kernel crash on boot:

ubuntu@juju-prod-ues-proposed-migration-machine-12:~⟫ nova console-log adt-xenial-ppc64el-systemd-20151207-161711

SLOF[0m[?25l **********************************************************************
[1mQEMU Starting
[0m Build Date = Oct 9 2015 00:25:21
 FW Version = buildd@ release 20140630
 Press "s" to enter Open Firmware.

[0m[?25hC0000C0100C0120C0140C0200C0201C0220C0240C0260C0270C02E0C0300C0320C0340C0360C0370C0380C0371C0372C0373C0374C0390C03F0C0400C0480C04C0C04D0C0500Populating /vdevice methods
Populating /vdevice/vty@30000000
Populating /vdevice/vty@30001000
Populating /vdevice/nvram@71000000
C0580C05A0Populating /pci@800000020000000
 Adapters on 0800000020000000
                     00 0800 (D) : 1af4 1000 virtio [ net ]
                     00 1000 (D) : 106b 003f serial bus [ usb-ohci ]
                     00 1800 (D) : 1af4 1001 virtio [ block ]
                     00 2000 (D) : 1af4 1002 unknown-legacy-device*
C0600C0640C0690C06A0C06A8C06B0C06B8C06C0C06E0C0700C0800C0880No NVRAM common partition, re-initializing...
C0890C08A0C08A8C08B0Scanning USB
  OHCI: initializing
C08C0C08D0Using default console: /vdevice/vty@30000000
C08E0C08E8C08FF
  Welcome to Open Firmware

  Copyright (c) 2004, 2011 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php

Trying to load: from: /pci@800000020000000/scsi@3 ...
No DOS disk-label found.
  Successfully loaded
[?25l[37m[40m[37m[40merror: no suitable video mode found.
[?25l[37m[40m[2J[m[1;1H[2;22HGNU GRUB version 2.02~beta2-32ubuntu1

[37m[40m[4;2H+----------------------------------------------------------------------------+[5;2H|[5;79H|[6;2H|[6;79H|[7;2H|[7;79H|[8;2H|[8;79H|[9;2H|[9;79H|[10;2H|[10;79H|[11;2H|[11;79H|[12;2H|[12;79H|[13;2H|[13;79H|[14;2H|[14;79H|[15;2H|[15;79H|[16;2H|[16;79H|[17;2H+----------------------------------------------------------------------------+[37m[40m[18;2H[19;2H[37m[40m Use the ^ and v keys to select which entry is highlighted.
      Press enter to boot the selected OS, `e' to edit the commands
      before booting or `c' for a command-line. [5;80H [30m[47m[5;3H*Ubuntu [37m[40m[5;78H[37m[40m[37m[40m[6;3H Advanced options for Ubuntu [37m[40m[6;78H[37m[40m[37m[40m[7;3H [37m[40m[7;78H[37m[40m[37m[40m[8;3H [37m[40m[8;78H[37m[40m[37m[40m[9;3H [37m[40m[9;78H[37m[40m[37m[40m[10;3H [37m[40m[10;78H[37m[40m[37m[40m[11;3H [37m[40m[11;78H[37m[40m[37m[40m[12;3H [37m[40m[12;78H[37m[40m[37m[40m[13;3H [37m[40m[13;78H[37m[40m[37m[40m[14;3H [37m[40m[14;78H[37m[40m[37m[40m[15;3H [37m[40m[15;78H[37m[40m[37m[40m[16;3H [37m[40m[16;78H[37m[40m[16;80H [5;78H[22;1H The highlighted entry will be executed automatically in 10s. [5;78H[22;1H The highlighted entry will be executed automatically in 9s. [5;78H[22;1H The highlighted entry will be executed automatically in 8s. [5;78H[22;1H The highlighted entry will be executed automatically in 7s. [5;78H[22;1H The highlighted entry will be executed automatically in 6s. [5;78H[22;1H The highlighted entry will be executed automatically in 5s. [5;78H[22;1H The highlighted entry will be executed automatically in 4s. [5;78H[22;1H The highlighted entry will be executed automatically in 3s. [5;78H[22;1H The highlighted entry will be executed automatically in 2s. [5;78H[22;1H The highlighted entry will be executed automatically in 1s. [5;78H[22;1H The highlighted entry will be executed automatically in 0s. [5;78H[?25h[2J[m[1;1H[2J[m[1;1HLoading Linux 4.4.0-0-generic ...
Loading initial ramdisk ...
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 4.4.0-0-generic (buildd@bos01-ppc64el-028) (gcc version 5.2.1 20151129 (Ubuntu 5.2.1-27ubuntu1) ) #3-Ubuntu SMP Mon Dec 7 10:52:50 UTC 2015 (Ubuntu 4.4.0-0.3-generic 4.4.0-rc4)
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 2048 (NR_CPUS = 2048)
Calling ibm,client-architecture-support... done
command line: BOOT_IMAGE=/boot/vmlinux-4.4.0-0-generic root=UUID=9f6cb10b-904a-4b22-b9e1-80455f176cc2 ro console=hvc0 earlyprintk init=/lib/systemd/systemd-bootchart
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000005890000
  alloc_top : 0000000030000000
  alloc_top_hi : 0000000060000000
  rmo_top : 0000000030000000
  ram_top : 0000000060000000
instantiating rtas at 0x000000002fff0000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x00000000058a0000 -> 0x00000000058a0849
Device tree struct 0x00000000058b0000 -> 0x00000000058c0000
Quiescing Open Firmware ...
Booting Linux via __start() ...
[ 0.000000] Using pSeries machine description
[ 0.000000] Page sizes from device-tree:
[ 0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
[ 0.000000] base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1
[ 0.000000] Using 1TB segments
[ 0.000000] Found initrd at 0xc000000003a00000:0xc000000005884a58
[ 0.000000] bootconsole [udbg0] enabled
[ 0.000000] CPU maps initialized for 1 thread per core
 -> smp_release_cpus()
spinning_secondaries = 0
 <- smp_release_cpus()
[ 0.000000] Starting Linux ppc64le #3-Ubuntu SMP Mon Dec 7 10:52:50 UTC 2015
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x18
[ 0.000000] phys_mem_size = 0x60000000
[ 0.000000] cpu_features = 0x17fc7a6c18500249
[ 0.000000] possible = 0x1fffffef18500649
[ 0.000000] always = 0x0000000018100040
[ 0.000000] cpu_user_features = 0xdc0065c2 0xef000000
[ 0.000000] mmu_features = 0x58000001
[ 0.000000] firmware_features = 0x000000014052440b
[ 0.000000] htab_hash_mask = 0x1ffff
[ 0.000000] -----------------------------------------------------
 <- setup_system()
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[ 1.181166] NIP: c000000000183050 LR: c0000000001838c8 CTR: c000000000183880
[ 1.181216] REGS: c00000005e70b7d0 TRAP: 0300 Tainted: G W (4.4.0-0-generic)
[ 1.181274] MSR: 8000000100009033 <SF,EE,ME,IR,DR,RI,LE> CR: 24028404 XER: 20000000
[ 1.181394] CFAR: c000000000009958 DAR: 00000000000000c1 DSISR: 40000000 SOFTE: 1
GPR00: c0000000001838c8 c00000005e70ba50 c000000001573100 0000000000000001
GPR04: 000000000000000c 0000000000000000 0000000000000000 0000000000000000
GPR08: ffffffffffffffff 0000000000000001 ffffffffffffffff 000000000e57a24d
GPR12: c000000000183880 c00000000fb40000 c0000000000e4188 0000000004208040
GPR16: c000000058574a68 000000000000000a 0000000000000000 c00000005fe1f338
GPR20: 0000000000000000 c000000001481780 c000000000b03918 7fffffffffffffff
GPR24: 0000000000000001 c00000005e708000 c00000000147c180 c000000053079800
GPR28: c00000000152d090 c0000000015b0030 c000000053cf6950 00000000000000c1
[ 1.182066] NIP [c000000000183050] pids_cancel.constprop.4+0x30/0x90
[ 1.182109] LR [c0000000001838c8] pids_free+0x48/0x80
[ 1.182142] Call Trace:
[ 1.182160] [c00000005e70ba50] [c000000000b03918] cpu_online_mask+0x0/0x8 (unreliable)
[ 1.182220] [c00000005e70ba80] [c0000000001838c8] pids_free+0x48/0x80
[ 1.182271] [c00000005e70bab0] [c000000000181eb0] cgroup_free+0x90/0xe0
[ 1.182322] [c00000005e70bb00] [c0000000000b3998] __put_task_struct+0x68/0x170
[ 1.182381] [c00000005e70bb30] [c0000000000b8b6c] delayed_put_task_struct+0x6c/0xe0
[ 1.182440] [c00000005e70bb70] [c00000000013e3a0] rcu_process_callbacks+0x340/0x6e0
[ 1.182513] [c00000005e70bc10] [c0000000000bcff8] __do_softirq+0x188/0x3a0
[ 1.182563] [c00000005e70bd00] [c0000000000bd254] run_ksoftirqd+0x44/0xb0
[ 1.182614] [c00000005e70bd20] [c0000000000e9af0] smpboot_thread_fn+0x290/0x2a0
[ 1.182673] [c00000005e70bd80] [c0000000000e4290] kthread+0x110/0x130
[ 1.182724] [c00000005e70be30] [c000000000009538] ret_from_kernel_thread+0x5c/0xa4
[ 1.182782] Instruction dump:
[ 1.182808] 3c4c013f 384200e0 7c0802a6 fbe1fff8 f8010010 f821ffd1 7c7f1b78 60000000
[ 1.182895] 60000000 3bff00c0 3940ffff 7c2004ac <7d20f8a8> 7d2a4a14 7d20f9ad 40c2fff4
[ 1.182982] ---[ end trace 66241192affcee1c ]---
[ 1.184126]
[ 3.184191] Kernel panic - not syncing: Fatal exception in interrupt

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1523586

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Martin Pitt (pitti) on 2015-12-07
tags: added: bot-stop-nagging
Changed in linux (Ubuntu):
status: Incomplete → New
status: New → Confirmed
Joseph Salisbury (jsalisbury) wrote :

Is this regression between the 4.3 and 4.4 kernels? Do you have a way to install test kernels to test this? If so, I can perform a kernel bisect to identify the exact commit that caused this.

Changed in linux (Ubuntu):
importance: Undecided → High
tags: added: kernel-key xenial
Martin Pitt (pitti) wrote :

Yes, it is a regression. Andy found the offending commit and the fix is already queued in Tejun's tree, so this should get fixed with the next upstream update.

Joseph Salisbury (jsalisbury) wrote :

Thanks for the update, Martin.

tags: removed: kernel-key
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers