[power9] [Cosmic] kernel panic while testing kdump

[Manoj Iyer]

I installed cosmic on power9 (boston) a non-nvme system, installed initramfs-tools from -proposed and install kdump-tools. Triggered a dump, but I got a kernel panic. This might be a kernel bug/regression.

ubuntu@tiselius:~$ uname -a
Linux tiselius 4.18.0-21-generic #22-Ubuntu SMP Wed May 15 13:12:45 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux
ubuntu@tiselius:~$

ubuntu@tiselius:~$ apt policy initramfs-tools
initramfs-tools:
  Installed: 0.131ubuntu15.2
  Candidate: 0.131ubuntu15.2
  Version table:
 *** 0.131ubuntu15.2 500
        500 http://ports.ubuntu.com/ubuntu-ports cosmic-proposed/main ppc64el Packages
ubuntu@tiselius:~$

ubuntu@tiselius:~$ apt policy kdump-tools
kdump-tools:
  Installed: 1:1.6.5-1ubuntu1~18.10.1
  Candidate: 1:1.6.5-1ubuntu1~18.10.1
  Version table:
 *** 1:1.6.5-1ubuntu1~18.10.1 500
        500 http://ports.ubuntu.com/ubuntu-ports cosmic-updates/main ppc64el Packages
        100 /var/lib/dpkg/status
     1:1.6.4-2ubuntu1 500
        500 http://ports.ubuntu.com/ubuntu-ports cosmic/main ppc64el Packages
ubuntu@tiselius:~$

ubuntu@tiselius:~$ sudo kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.18.0-21-generic
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.18.0-21-generic
current state: ready to kdump

kexec command:
  /sbin/kexec -p --command-line="root=UUID=295f571b-b731-4ebb-b752-60aadc80fc1b ro console=hvc0 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll noirqdistrib nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
ubuntu@tiselius:~$

ubuntu@tiselius:~$ cat /proc/cmdline
root=UUID=295f571b-b731-4ebb-b752-60aadc80fc1b ro console=hvc0 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@128M
ubuntu@tiselius:~$

ubuntu@tiselius:~$ dmesg | grep Reser
[ 0.000000] Reserving 4096MB of memory at 128MB for crashkernel (System RAM: 131072MB)
[ 0.000000] cma: Reserved 6560 MiB at 0x0000200e62000000
ubuntu@tiselius:~$

root@tiselius:/home/ubuntu# echo 1 > /proc/sys/kernel/sysrq
root@tiselius:/home/ubuntu# echo c > /proc/sysrq-trigger

tiselius login: [ 150.167288] cloud-init[4529]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.10.1 running 'modules:final' at Mon, 10 Jun 2019 19:53:40 +0000. Up 149.80 seconds.
[ 150.167687] cloud-init[4529]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.10.1 finished at Mon, 10 Jun 2019 19:53:40 +0000. Datasource DataSourceMAAS [http://10-245-64-0--21.maas-internal:5248/MAAS/metadata/]. Up 150.11 seconds
[ 360.313029] kdump-tools[4915]: Stopping kdump-tools: * unloaded kdump kernel
[ 376.552452] kdump-tools[10449]: Starting kdump-tools: * Creating symlink /var/lib/kdump/vmlinuz
[ 376.553743] kdump-tools[10449]: * Creating symlink /var/lib/kdump/initrd.img
[ 376.585085] kdump-tools[10449]: Modified cmdline:root=UUID=295f571b-b731-4ebb-b752-60aadc80fc1b ro console=hvc0 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll noirqdistrib nousb elfcorehdr=158784K
[ 376.953223] kdump-tools[10449]: * loaded kdump kernel
[ 398.517900] sysrq: SysRq : Trigger a crash
[ 398.517952] Unable to handle kernel paging request for data at address 0x00000000
[ 398.518000] Faulting instruction address: 0xc00000000082a3a8
[ 398.518071] Oops: Kernel access of bad area, sig: 11 [#1]
[ 398.518115] LE SMP NR_CPUS=2048 NUMA PowerNV
[ 398.518172] Modules linked in: joydev input_leds mac_hid vmx_crypto ofpart crct10dif_vpmsum ipmi_powernv ipmi_devintf at24 uio_pdrv_genirq ipmi_msghandler uio cmdlinepart powernv_flash mtd opal_prd ibmpowernv sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure scsi_transport_sas hid_generic usbhid hid ast i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm crc32c_vpmsum i40e aacraid drm_panel_orientation_quirks
[ 398.518769] CPU: 0 PID: 10529 Comm: bash Kdump: loaded Not tainted 4.18.0-21-generic #22-Ubuntu
[ 398.518881] NIP: c00000000082a3a8 LR: c00000000082b234 CTR: c00000000082a380
[ 398.518976] REGS: c000000b9b88ba00 TRAP: 0300 Not tainted (4.18.0-21-generic)
[ 398.519068] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 48422222 XER: 20040000
[ 398.519161] CFAR: c00000000082b230 DAR: 0000000000000000 DSISR: 42000000 IRQMASK: 0
[ 398.519161] GPR00: c00000000082b234 c000000b9b88bc80 c00000000178ca00 0000000000000063
[ 398.519161] GPR04: 0000000000000001 000000000000036a 9000000000009033 0000000031c40058
[ 398.519161] GPR08: 0000000000000007 0000000000000001 0000000000000000 9000000000001003
[ 398.519161] GPR12: c00000000082a380 c000000001b00000 00000a452fe89760 0000000000000000
[ 398.519161] GPR16: 00000a4559cdaea0 00000a452fe24a48 00000a452fe89760 00000a452fda8b00
[ 398.519161] GPR20: 0000000000000000 0000000000000001 00000a452fe370b8 00007fffdf9d8604
[ 398.519161] GPR24: 00007fffdf9d8600 00000a452fe8afe4 0000739a5307bfa8 c0000000016836b0
[ 398.519161] GPR28: 0000000000000004 0000000000000002 0000000000000063 c000000001602e38
[ 398.519905] NIP [c00000000082a3a8] sysrq_handle_crash+0x28/0x30
[ 398.519979] LR [c00000000082b234] __handle_sysrq+0xf4/0x230
[ 398.520034] Call Trace:
[ 398.520069] [c000000b9b88bc80] [c00000000082b218] __handle_sysrq+0xd8/0x230 (unreliable)
[ 398.520168] [c000000b9b88bd10] [c00000000082b9a8] write_sysrq_trigger+0x68/0x90
[ 398.520249] [c000000b9b88bd40] [c0000000004ae664] proc_reg_write+0x84/0xc0
[ 398.520321] [c000000b9b88bd70] [c0000000003fe35c] __vfs_write+0x3c/0x70
[ 398.520393] [c000000b9b88bd90] [c0000000003fe5f8] vfs_write+0xd8/0x220
[ 398.520475] [c000000b9b88bde0] [c0000000003fe974] ksys_write+0x64/0x110
[ 398.520538] [c000000b9b88be30] [c00000000000b288] system_call+0x5c/0x70
[ 398.520580] Instruction dump:
[ 398.520601] 4bfff9e1 4bfffe44 3c4c00f6 38422680 7c0802a6 60000000 39200001 3d42001d
[ 398.520644] 394a90f8 912a0000 7c0004ac 39400000 <992a0000> 4e800020 3c4c00f6 38422650
[ 398.520689] ---[ end trace 720e78394d62f6eb ]---
[ 398.583396]
[ 398.583509] Sending IPI to other CPUs
[ 3[ 556.071752027,5] OPAL: Switch to big-endian OS
99.882106] IPI complete
[ 401.593332] kexec: Starting switchover sequence.
[ 559.475190556,5] OPAL: Switch to little-endian OS
[ 0.000000] dt-cpu-ftrs: setup for ISA 3000
[ 0.000000] dt-cpu-ftrs: not enabling: system-call-vectored (disabled or unsupported by kernel)
[ 0.000000] dt-cpu-ftrs: final cpu/mmu features = 0x0000f86f8f5fb1a7 0x3c006041
[ 0.000000] radix-mmu: Page sizes from device-tree:
[ 0.000000] radix-mmu: Page size shift = 12 AP=0x0
[ 0.000000] radix-mmu: Page size shift = 16 AP=0x5
[ 0.000000] radix-mmu: Page size shift = 21 AP=0x1
[ 0.000000] radix-mmu: Page size shift = 30 AP=0x2
[ 0.000000] radix-mmu: Initializing Radix MMU
[ 0.000000] radix-mmu: Partition table (____ptrval____)
[ 0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000100000000 with 1.00 GiB pages
[ 0.000000] radix-mmu: Mapped 0x0000000100000000-0x0000000108000000 with 2.00 MiB pages
[ 0.000000] radix-mmu: Process table (____ptrval____) and radix root for kernel: (____ptrval____)
[ 0.000000] Linux version 4.18.0-21-generic (buildd@bos02-ppc64el-001) (gcc version 8.2.0 (Ubuntu 8.2.0-7ubuntu1)) #22-Ubuntu SMP Wed May 15 13:12:45 UTC 2019 (Ubuntu 4.18.0-21.22-generic 4.18.20)
[ 0.000000] Found initrd at 0xc000000009b30000:0xc00000000b743360
[ 0.000000] OPAL: Found memory mapped LPC bus on chip 0
[ 0.000000] Using PowerNV machine description
[ 0.000000] bootconsole [udbg0] enabled
[ 0.000000] CPU maps initialized for 4 threads per core
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x0
[ 0.000000] phys_mem_size = 0x108000000
[ 0.000000] dcache_bsize = 0x80
[ 0.000000] icache_bsize = 0x80
[ 0.000000] cpu_features = 0x0000f86f8f5fb1a7
[ 0.000000] possible = 0x0000ffffcf5fb1a7
[ 0.000000] always = 0x00000003800081a1
[ 0.000000] cpu_user_features = 0xdc0065c2 0xaee00000
[ 0.000000] mmu_features = 0x3c006041
[ 0.000000] firmware_features = 0x0000000110000000
[ 0.000000] physical_start = 0x8000000
[ 0.000000] -----------------------------------------------------
[ 0.000000] cma: Reserved 224 MiB at 0x00000000f8000000
[ 0.000000] numa: NODE_DATA [mem 0x107ec7c80-0x107ed3fff]
[ 0.000000] numa: NODE_DATA [mem 0x107eab900-0x107eb7c7f]
[ 0.000000] numa: NODE_DATA(8) on node 0
[ 0.000000] rfi-flush: mttrig type flush available
[ 0.000000] barrier-nospec: using ORI speculation barrier
[ 0.000000] stf-barrier: eieio barrier available
[ 0.000000] Initializing IODA2 PHB (/pciex@600c3c0000000)
[ 0.000000] PCI host bridge /pciex@600c3c0000000 (primary) ranges:
[ 0.000000] MEM 0x000600c000000000..0x000600c07ffeffff -> 0x0000000080000000
[ 0.000000] MEM 0x0006000000000000..0x0006003fffffffff -> 0x0006000000000000 (M64 #1..31)
[ 0.000000] Using M64 #31 as default window
[ 0.000000] 512 (511) PE's M32: 0x80000000 [segment=0x400000]
[ 0.000000] M64: 0x4000000000 [segment=0x20000000]
[ 0.000000] Allocated bitmap for 4088 MSIs (base IRQ 0xfe000)
[ 0.000000] Issue PHB reset ...
[ 561.289221670,3] PHB#0000[0:0]: CRESET: Unexpected slot state 00000102, resetting...
[ 0.000000] Initializing IODA2 PHB (/pciex@600c3c0100000)
[ 0.000000] PCI host bridge /pciex@600c3c0100000 ranges:
[ 0.000000] MEM 0x000600c080000000..0x000600c0fffeffff -> 0x0000000080000000
[ 0.000000] MEM 0x0006004000000000..0x0006007fffffffff -> 0x0006004000000000 (M64 #1..15)
[ 0.000000] Using M64 #15 as default window
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x4000000000 [segment=0x40000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0xfd800)
[ 0.000000] Issue PHB reset ...
[ 563.160826313,3] PHB#0001[0:1]: CRESET: Unexpected slot state 00000102, resetting...
[ 0.000000] Initializing IODA2 PHB (/pciex@600c3c0200000)
[ 0.000000] PCI host bridge /pciex@600c3c0200000 ranges:
[ 0.000000] MEM 0x000600c100000000..0x000600c17ffeffff -> 0x0000000080000000
[ 0.000000] MEM 0x0006008000000000..0x000600bfffffffff -> 0x0006008000000000 (M64 #1..15)
[ 0.000000] Using M64 #15 as default window
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x4000000000 [segment=0x40000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0xfd000)
[ 0.000000] Issue PHB reset ...
[ 0.000000] Initializing IODA2 PHB (/pciex@600c3c0300000)
[ 0.000000] PCI host bridge /pciex@600c3c0300000 ranges:
[ 0.000000] MEM 0x000600c180000000..0x000600c1fffeffff -> 0x0000000080000000
[ 0.000000] MEM 0x0006020000000000..0x0006023fffffffff -> 0x0006020000000000 (M64 #1..31)
[ 0.000000] Using M64 #31 as default window
[ 0.000000] 512 (511) PE's M32: 0x80000000 [segment=0x400000]
[ 0.000000] M64: 0x4000000000 [segment=0x20000000]
[ 0.000000] Allocated bitmap for 4088 MSIs (base IRQ 0xfc000)
[ 0.000000] Issue PHB reset ...
[ 0.000000] Initializing IODA2 PHB (/pciex@600c3c0400000)
[ 0.000000] PCI host bridge /pciex@600c3c0400000 ranges:
[ 0.000000] MEM 0x000600c200000000..0x000600c27ffeffff -> 0x0000000080000000
[ 0.000000] MEM 0x0006024000000000..0x0006027fffffffff -> 0x0006024000000000 (M64 #1..15)
[ 0.000000] Using M64 #15 as default window
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x4000000000 [segment=0x40000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0xfb800)
[ 0.000000] Issue PHB reset ...
[ 0.000000] Initializing IODA2 PHB (/pciex@600c3c0500000)
[ 0.000000] PCI host bridge /pciex@600c3c0500000 ranges:
[ 0.000000] MEM 0x000600c280000000..0x000600c2fffeffff -> 0x0000000080000000
[ 0.000000] MEM 0x0006028000000000..0x000602bfffffffff -> 0x0006028000000000 (M64 #1..15)
[ 0.000000] Using M64 #15 as default window
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x4000000000 [segment=0x40000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0xfb000)
[ 0.000000] Issue PHB reset ...
[ 0.000000] Initializing IODA2 PHB (/pciex@620c3c0000000)
[ 0.000000] PCI host bridge /pciex@620c3c0000000 ranges:
[ 0.000000] MEM 0x000620c000000000..0x000620c07ffeffff -> 0x0000000080000000
[ 0.000000] MEM 0x0006200000000000..0x0006203fffffffff -> 0x0006200000000000 (M64 #1..31)
[ 0.000000] Using M64 #31 as default window
[ 0.000000] 512 (511) PE's M32: 0x80000000 [segment=0x400000]
[ 0.000000] M64: 0x4000000000 [segment=0x20000000]
[ 0.000000] Allocated bitmap for 4088 MSIs (base IRQ 0x1fe000)
[ 0.000000] Issue PHB reset ...
[ 573.138043988,3] PHB#0030[8:0]: CRESET: Unexpected slot state 00000102, resetting...
[ 0.000000] Initializing IODA2 PHB (/pciex@620c3c0100000)
[ 0.000000] PCI host bridge /pciex@620c3c0100000 ranges:
[ 0.000000] MEM 0x000620c080000000..0x000620c0fffeffff -> 0x0000000080000000
[ 0.000000] MEM 0x0006204000000000..0x0006207fffffffff -> 0x0006204000000000 (M64 #1..15)
[ 0.000000] Using M64 #15 as default window
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x4000000000 [segment=0x40000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0x1fd800)
[ 0.000000] Issue PHB reset ...
[ 575.009882231,3] PHB#0031[8:1]: CRESET: Unexpected slot state 00000102, resetting...
[ 0.000000] Initializing IODA2 PHB (/pciex@620c3c0200000)
[ 0.000000] PCI host bridge /pciex@620c3c0200000 ranges:
[ 0.000000] MEM 0x000620c100000000..0x000620c17ffeffff -> 0x0000000080000000
[ 0.000000] MEM 0x0006208000000000..0x000620bfffffffff -> 0x0006208000000000 (M64 #1..15)
[ 0.000000] Using M64 #15 as default window
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x4000000000 [segment=0x40000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0x1fd000)
[ 0.000000] Issue PHB reset ...
[ 576.393558385,3] PHB#0032[8:2]: CRESET: Unexpected slot state 00000102, resetting...
[ 0.000000] Initializing IODA2 PHB (/pciex@620c3c0300000)
[ 0.000000] PCI host bridge /pciex@620c3c0300000 ranges:
[ 0.000000] MEM 0x000620c180000000..0x000620c1fffeffff -> 0x0000000080000000
[ 0.000000] MEM 0x0006220000000000..0x0006223fffffffff -> 0x0006220000000000 (M64 #1..31)
[ 0.000000] Using M64 #31 as default window
[ 0.000000] 512 (511) PE's M32: 0x80000000 [segment=0x400000]
[ 0.000000] M64: 0x4000000000 [segment=0x20000000]
[ 0.000000] Allocated bitmap for 4088 MSIs (base IRQ 0x1fc000)
[ 0.000000] Issue PHB reset ...
[ 577.127956119,3] PHB#0033[8:3]: CRESET: Unexpected slot state 00000102, resetting...
[ 0.000000] OPAL nvram setup, 589824 bytes
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x0000000107ffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal empty
[ 0.000000] Device empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x0000000107ffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000107ffffff]
[ 0.000000] Could not find start_pfn for node 8
[ 0.000000] Initmem setup node 8 [mem 0x0000000000000000-0x0000000000000000]
[ 0.000000] percpu: Embedded 4 pages/cpu @(____ptrval____) s172440 r0 d89704 u262144
[ 0.000000] Built 2 zonelists, mobility grouping on. Total pages: 67518
[ 0.000000] Policy zone: DMA
[ 0.000000] Kernel command line: root=UUID=295f571b-b731-4ebb-b752-60aadc80fc1b ro console=hvc0 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll noirqdistrib nousb elfcorehdr=158784K
[ 0.000000] Misrouted IRQ fixup and polling support enabled
[ 0.000000] This may significantly impact system performance
[ 0.000000] Memory: 3731200K/4325376K available (13760K kernel code, 2112K rwdata, 3712K rodata, 4928K init, 3013K bss, 364800K reserved, 229376K cma-reserved)
[ 0.000000] random: get_random_u64 called from kmem_cache_open+0x50/0x630 with crng_init=0
[ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=1, Nodes=9
[ 0.000000] ftrace: allocating 35887 entries in 14 pages
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=1.
[ 0.000000] Tasks RCU enabled.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[ 0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[ 0.000000] xive: Interrupt handling initialized with native backend
[ 0.000000] xive: Using priority 7 for all interrupts
[ 0.000000] xive: Using 64kB queues
[ 0.000003] time_init: 56 bit decrementer (max: 7fffffffffffff)
[ 0.000468] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[ 0.001281] clocksource: timebase mult[1f40000] shift[24] registered
[ 0.001831] Console: colour dummy device 80x25
[ 0.002356] console [hvc0] enabled
[ 0.002356] console [hvc0] enabled
[ 0.002678] bootconsole [udbg0] disabled
[ 0.002678] bootconsole [udbg0] disabled
[ 0.003101] mempolicy: Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl
[ 0.003258] pid_max: default: 32768 minimum: 301
[ 0.003499] Security Framework initialized
[ 0.003541] Yama: becoming mindful.
[ 0.003622] AppArmor: AppArmor initialized
[ 0.004087] Dentry cache hash table entries: 524288 (order: 6, 4194304 bytes)
[ 0.004346] Inode-cache hash table entries: 262144 (order: 5, 2097152 bytes)
[ 0.004470] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes)
[ 0.004569] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes)
[ 0.005461] EEH: PowerNV platform initialized
[ 0.005530] POWER9 performance monitor hardware support registered
[ 0.005649] Hierarchical SRCU implementation.
[ 0.006756] smp: Bringing up secondary CPUs ...
[ 0.006825] smp: Brought up 2 nodes, 1 CPU
[ 0.006882] numa: Node 0 CPUs: 0
[ 0.006931] numa: Node 8 CPUs:
[ 0.006988] Using standard scheduler topology
[ 0.009697] devtmpfs: initialized
[ 0.026259] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.026356] futex hash table entries: 256 (order: -1, 32768 bytes)
[ 0.026687] pinctrl core: initialized pinctrl subsystem
[ 0.027177] NET: Registered protocol family 16
[ 0.027305] EEH: No capable adapters found
[ 0.027642] audit: initializing netlink subsys (disabled)
[ 0.027996] cpuidle: using governor ladder
[ 0.028045] cpuidle: using governor menu
[ 0.028218] pstore: Registered nvram as persistent store backend
[ 0.029675] PCI: Probing PCI hardware
[ 0.029799] PCI host bridge to bus 0000:00
[ 0.029872] pci_bus 0000:00: root bus resource [mem 0x600c000000000-0x600c07ffeffff] (bus address [0x80000000-0xfffeffff])
[ 0.030020] pci_bus 0000:00: root bus resource [mem 0x6000000000000-0x6003fbfffffff 64bit pref]
[ 0.030145] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.033999] audit: type=2000 audit(1560196693.024:1): state=initialized audit_enabled=0 res=1
[ 0.041199] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[ 0.041391] PCI host bridge to bus 0001:00
[ 0.041452] pci_bus 0001:00: root bus resource [mem 0x600c080000000-0x600c0fffeffff] (bus address [0x80000000-0xfffeffff])
[ 0.041608] pci_bus 0001:00: root bus resource [mem 0x6004000000000-0x6007f7fffffff 64bit pref]
[ 0.041736] pci_bus 0001:00: root bus resource [bus 00-ff]
[ 0.052585] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[ 0.052788] PCI host bridge to bus 0002:00
[ 0.052845] pci_bus 0002:00: root bus resource [mem 0x600c100000000-0x600c17ffeffff] (bus address [0x80000000-0xfffeffff])
[ 0.052987] pci_bus 0002:00: root bus resource [mem 0x6008000000000-0x600bf7fffffff 64bit pref]
[ 0.053122] pci_bus 0002:00: root bus resource [bus 00-ff]
[ 0.059032] pci 0002:01:00.0: BAR3 [mem size 0x00008000 64bit pref]: requesting alignment to 0x10000
[ 0.059430] pci 0002:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x001fffff 64bit pref] (contains BAR0 for 32 VFs)
[ 0.059628] pci 0002:01:00.0: VF(n) BAR3 space: [mem 0x00000000-0x001fffff 64bit pref] (contains BAR3 for 32 VFs)
[ 0.060081] pci 0002:01:00.0: VF BAR0: [mem 0x00000000-0x00ffffff 64bit pref] (expanded to 256 VFs for PE alignment)
[ 0.060221] pci 0002:01:00.0: VF BAR3: [mem 0x00000000-0x00ffffff 64bit pref] (expanded to 256 VFs for PE alignment)
[ 0.060673] pci 0002:01:00.1: BAR3 [mem size 0x00008000 64bit pref]: requesting alignment to 0x10000
[ 0.061039] pci 0002:01:00.1: VF(n) BAR0 space: [mem 0x00000000-0x001fffff 64bit pref] (contains BAR0 for 32 VFs)
[ 0.061235] pci 0002:01:00.1: VF(n) BAR3 space: [mem 0x00000000-0x001fffff 64bit pref] (contains BAR3 for 32 VFs)
[ 0.061668] pci 0002:01:00.1: VF BAR0: [mem 0x00000000-0x00ffffff 64bit pref] (expanded to 256 VFs for PE alignment)
[ 0.061795] pci 0002:01:00.1: VF BAR3: [mem 0x00000000-0x00ffffff 64bit pref] (expanded to 256 VFs for PE alignment)
[ 0.062284] pci 0002:01:00.2: BAR3 [mem size 0x00008000 64bit pref]: requesting alignment to 0x10000
[ 0.062663] pci 0002:01:00.2: VF(n) BAR0 space: [mem 0x00000000-0x001fffff 64bit pref] (contains BAR0 for 32 VFs)
[ 0.062861] pci 0002:01:00.2: VF(n) BAR3 space: [mem 0x00000000-0x001fffff 64bit pref] (contains BAR3 for 32 VFs)
[ 0.063292] pci 0002:01:00.2: VF BAR0: [mem 0x00000000-0x00ffffff 64bit pref] (expanded to 256 VFs for PE alignment)
[ 0.063436] pci 0002:01:00.2: VF BAR3: [mem 0x00000000-0x00ffffff 64bit pref] (expanded to 256 VFs for PE alignment)
[ 0.063897] pci 0002:01:00.3: BAR3 [mem size 0x00008000 64bit pref]: requesting alignment to 0x10000
[ 0.064265] pci 0002:01:00.3: VF(n) BAR0 space: [mem 0x00000000-0x001fffff 64bit pref] (contains BAR0 for 32 VFs)
[ 0.064454] pci 0002:01:00.3: VF(n) BAR3 space: [mem 0x00000000-0x001fffff 64bit pref] (contains BAR3 for 32 VFs)
[ 0.064848] pci 0002:01:00.3: VF BAR0: [mem 0x00000000-0x00ffffff 64bit pref] (expanded to 256 VFs for PE alignment)
[ 0.064920] pci 0002:01:00.3: VF BAR3: [mem 0x00000000-0x00ffffff 64bit pref] (expanded to 256 VFs for PE alignment)
[ 0.070549] pci 0002:00:00.0: PCI bridge to [bus 01]
[ 0.070788] PCI host bridge to bus 0003:00
[ 0.070856] pci_bus 0003:00: root bus resource [mem 0x600c180000000-0x600c1fffeffff] (bus address [0x80000000-0xfffeffff])
[ 0.070979] pci_bus 0003:00: root bus resource [mem 0x6020000000000-0x6023fbfffffff 64bit pref]
[ 0.071042] pci_bus 0003:00: root bus resource [bus 00-ff]
[ 0.077072] pci 0003:01:00.0: enabling Extended Tags
[ 0.077165] pci 0003:01:00.0: BAR2 [mem size 0x00000400 64bit]: requesting alignment to 0x10000
[ 0.082684] pci 0003:00:00.0: PCI bridge to [bus 01]
[ 0.082933] PCI host bridge to bus 0004:00
[ 0.082983] pci_bus 0004:00: root bus resource [mem 0x600c200000000-0x600c27ffeffff] (bus address [0x80000000-0xfffeffff])
[ 0.083148] pci_bus 0004:00: root bus resource [mem 0x6024000000000-0x6027f7fffffff 64bit pref]
[ 0.083276] pci_bus 0004:00: root bus resource [bus 00-ff]
[ 0.089102] pci 0004:01:00.0: enabling Extended Tags
[ 0.094632] pci 0004:00:00.0: PCI bridge to [bus 01-02]
[ 0.094851] pci_bus 0004:02: extended config space not accessible
[ 0.100569] pci 0004:01:00.0: PCI bridge to [bus 02]
[ 0.100953] PCI host bridge to bus 0005:00
[ 0.101001] pci_bus 0005:00: root bus resource [mem 0x600c280000000-0x600c2fffeffff] (bus address [0x80000000-0xfffeffff])
[ 0.101143] pci_bus 0005:00: root bus resource [mem 0x6028000000000-0x602bf7fffffff 64bit pref]
[ 0.101287] pci_bus 0005:00: root bus resource [bus 00-ff]
[ 0.107168] pci 0005:01:00.0: BAR2 [mem size 0x00002000 64bit]: requesting alignment to 0x10000
[ 0.112742] pci 0005:00:00.0: PCI bridge to [bus 01]
[ 0.112953] PCI host bridge to bus 0030:00
[ 0.113010] pci_bus 0030:00: root bus resource [mem 0x620c000000000-0x620c07ffeffff] (bus address [0x80000000-0xfffeffff])
[ 0.113171] pci_bus 0030:00: root bus resource [mem 0x6200000000000-0x6203fbfffffff 64bit pref]
[ 0.113292] pci_bus 0030:00: root bus resource [bus 00-ff]
[ 0.124211] pci 0030:00:00.0: PCI bridge to [bus 01-ff]
[ 0.124406] PCI host bridge to bus 0031:00
[ 0.124465] pci_bus 0031:00: root bus resource [mem 0x620c080000000-0x620c0fffeffff] (bus address [0x80000000-0xfffeffff])
[ 0.124623] pci_bus 0031:00: root bus resource [mem 0x6204000000000-0x6207f7fffffff 64bit pref]
[ 0.124756] pci_bus 0031:00: root bus resource [bus 00-ff]
[ 0.135655] pci 0031:00:00.0: PCI bridge to [bus 01-ff]
[ 0.135858] PCI host bridge to bus 0032:00
[ 0.135918] pci_bus 0032:00: root bus resource [mem 0x620c100000000-0x620c17ffeffff] (bus address [0x80000000-0xfffeffff])
[ 0.136074] pci_bus 0032:00: root bus resource [mem 0x6208000000000-0x620bf7fffffff 64bit pref]
[ 0.136208] pci_bus 0032:00: root bus resource [bus 00-ff]
[ 0.147102] pci 0032:00:00.0: PCI bridge to [bus 01-ff]
[ 0.147301] PCI host bridge to bus 0033:00
[ 0.147354] pci_bus 0033:00: root bus resource [mem 0x620c180000000-0x620c1fffeffff] (bus address [0x80000000-0xfffeffff])
[ 0.147511] pci_bus 0033:00: root bus resource [mem 0x6220000000000-0x6223fbfffffff 64bit pref]
[ 0.147657] pci_bus 0033:00: root bus resource [bus 00-ff]
[ 0.158559] pci 0033:00:00.0: PCI bridge to [bus 01-ff]
[ 0.158730] pci 0000:00 : [PE# 1fe] Secondary bus 0 associated with PE#1fe
[ 0.159136] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[ 0.159258] pci 0001:00 : [PE# fe] Secondary bus 0 associated with PE#fe
[ 0.159656] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[ 0.159792] pci 0002:00:00.0: BAR 15: assigned [mem 0x6008000000000-0x600803fffffff 64bit pref]
[ 0.159910] pci 0002:00:00.0: BAR 14: assigned [mem 0x600c100000000-0x600c1007fffff]
[ 0.160050] pci 0002:01:00.0: BAR 7: assigned [mem 0x6008000000000-0x6008000ffffff 64bit pref]
[ 0.160182] pci 0002:01:00.0: BAR 10: assigned [mem 0x6008001000000-0x6008001ffffff 64bit pref]
[ 0.160320] pci 0002:01:00.1: BAR 7: assigned [mem 0x6008002000000-0x6008002ffffff 64bit pref]
[ 0.160461] pci 0002:01:00.1: BAR 10: assigned [mem 0x6008003000000-0x6008003ffffff 64bit pref]
[ 0.160594] pci 0002:01:00.2: BAR 7: assigned [mem 0x6008004000000-0x6008004ffffff 64bit pref]
[ 0.160724] pci 0002:01:00.2: BAR 10: assigned [mem 0x6008005000000-0x6008005ffffff 64bit pref]
[ 0.160858] pci 0002:01:00.3: BAR 7: assigned [mem 0x6008006000000-0x6008006ffffff 64bit pref]
[ 0.160987] pci 0002:01:00.3: BAR 10: assigned [mem 0x6008007000000-0x6008007ffffff 64bit pref]
[ 0.161126] pci 0002:01:00.0: BAR 0: assigned [mem 0x6008008000000-0x60080087fffff 64bit pref]
[ 0.161271] pci 0002:01:00.1: BAR 0: assigned [mem 0x6008008800000-0x6008008ffffff 64bit pref]
[ 0.161392] random: fast init done
[ 0.161469] pci 0002:01:00.2: BAR 0: assigned [mem 0x6008009000000-0x60080097fffff 64bit pref]
[ 0.161632] pci 0002:01:00.3: BAR 0: assigned [mem 0x6008009800000-0x6008009ffffff 64bit pref]
[ 0.161774] pci 0002:01:00.0: BAR 6: assigned [mem 0x600c100000000-0x600c10007ffff pref]
[ 0.161886] pci 0002:01:00.1: BAR 6: assigned [mem 0x600c100080000-0x600c1000fffff pref]
[ 0.162019] pci 0002:01:00.2: BAR 6: assigned [mem 0x600c100100000-0x600c10017ffff pref]
[ 0.162132] pci 0002:01:00.3: BAR 6: assigned [mem 0x600c100180000-0x600c1001fffff pref]
[ 0.162220] pci 0002:01:00.0: BAR 3: assigned [mem 0x600800a000000-0x600800a007fff 64bit pref]
[ 0.162301] pci 0002:01:00.1: BAR 3: assigned [mem 0x600800a010000-0x600800a017fff 64bit pref]
[ 0.162382] pci 0002:01:00.2: BAR 3: assigned [mem 0x600800a020000-0x600800a027fff 64bit pref]
[ 0.162463] pci 0002:01:00.3: BAR 3: assigned [mem 0x600800a030000-0x600800a037fff 64bit pref]
[ 0.162546] pci 0002:00 : [PE# fe] Secondary bus 0 associated with PE#fe
[ 0.162902] pci 0002:01 : [PE# 00] Secondary bus 1 associated with PE#0
[ 0.163292] pci 0002:01 : [PE# 00] Setting up 32-bit TCE table at 0..80000000
[ 0.184595] IOMMU table initialized, virtual merging enabled
[ 0.184677] pci 0002:01 : [PE# 00] Setting up window#0 0..7fffffff pg=1000
[ 0.184791] pci 0002:01 : [PE# 00] Enabling 64-bit DMA bypass
[ 0.184906] iommu: Adding device 0002:01:00.0 to group 0
[ 0.184995] iommu: Adding device 0002:01:00.1 to group 0
[ 0.185079] iommu: Adding device 0002:01:00.2 to group 0
[ 0.185159] iommu: Adding device 0002:01:00.3 to group 0
[ 0.185237] pci 0002:00:00.0: PCI bridge to [bus 01]
[ 0.185319] pci 0002:00:00.0: bridge window [mem 0x600c100000000-0x600c17fefffff]
[ 0.185431] pci 0002:00:00.0: bridge window [mem 0x6008000000000-0x600bf7ff0ffff 64bit pref]
[ 0.185600] pci 0003:00:00.0: BAR 14: assigned [mem 0x600c180000000-0x600c1803fffff]
[ 0.185717] pci 0003:01:00.0: BAR 0: assigned [mem 0x600c180000000-0x600c1800fffff 64bit]
[ 0.185856] pci 0003:01:00.0: BAR 6: assigned [mem 0x600c180100000-0x600c18017ffff pref]
[ 0.185968] pci 0003:01:00.0: BAR 2: assigned [mem 0x600c180180000-0x600c1801803ff 64bit]
[ 0.186117] pci 0003:01:00.0: BAR 4: no space for [io size 0x0100]
[ 0.186208] pci 0003:01:00.0: BAR 4: failed to assign [io size 0x0100]
[ 0.186311] pci 0003:00 : [PE# 1fe] Secondary bus 0 associated with PE#1fe
[ 0.186710] pci 0003:01 : [PE# 1fd] Secondary bus 1 associated with PE#1fd
[ 0.187117] pci 0003:01 : [PE# 1fd] Setting up 32-bit TCE table at 0..80000000
[ 0.207940] pci 0003:01 : [PE# 1fd] Setting up window#0 0..7fffffff pg=1000
[ 0.208053] pci 0003:01 : [PE# 1fd] Enabling 64-bit DMA bypass
[ 0.208155] iommu: Adding device 0003:01:00.0 to group 1
[ 0.208226] pci 0003:00:00.0: PCI bridge to [bus 01]
[ 0.208323] pci 0003:00:00.0: bridge window [mem 0x600c180000000-0x600c1ffefffff]
[ 0.208449] pci_bus 0003:00: Some PCI device resources are unassigned, try booting with pci=realloc
[ 0.208638] pci 0004:00:00.0: BAR 14: assigned [mem 0x600c200000000-0x600c2017fffff]
[ 0.208742] pci 0004:01:00.0: BAR 14: assigned [mem 0x600c200000000-0x600c2017fffff]
[ 0.208869] pci 0004:01:00.0: BAR 13: no space for [io size 0x1000]
[ 0.208960] pci 0004:01:00.0: BAR 13: failed to assign [io size 0x1000]
[ 0.209055] pci 0004:02:00.0: BAR 0: assigned [mem 0x600c200000000-0x600c200ffffff]
[ 0.209178] pci 0004:02:00.0: BAR 1: assigned [mem 0x600c201000000-0x600c20101ffff]
[ 0.209292] pci 0004:02:00.0: BAR 2: no space for [io size 0x0080]
[ 0.209383] pci 0004:02:00.0: BAR 2: failed to assign [io size 0x0080]
[ 0.209450] pci 0004:00 : [PE# fe] Secondary bus 0 associated with PE#fe
[ 0.209828] pci 0004:02 : [PE# fd] Secondary bus 2..2 associated with PE#fd
[ 0.210259] pci 0004:02 : [PE# fd] Setting up 32-bit TCE table at 0..80000000
[ 0.231122] pci 0004:02 : [PE# fd] Setting up window#0 0..7fffffff pg=1000
[ 0.231211] pci 0004:02 : [PE# fd] Enabling 64-bit DMA bypass
[ 0.231327] iommu: Adding device 0004:02:00.0 to group 2
[ 0.231422] pci 0004:01:00.0: PCI bridge to [bus 02]
[ 0.231504] pci 0004:01:00.0: bridge window [mem 0x600c200000000-0x600c27fefffff]
[ 0.231642] pci 0004:01 : [PE# fc] Secondary bus 1 associated with PE#fc
[ 0.232040] pci 0004:00:00.0: PCI bridge to [bus 01-02]
[ 0.232093] pci 0004:00:00.0: bridge window [mem 0x600c200000000-0x600c27fefffff]
[ 0.232225] pci_bus 0004:00: Some PCI device resources are unassigned, try booting with pci=realloc
[ 0.232438] pci 0005:00:00.0: BAR 14: assigned [mem 0x600c280000000-0x600c2807fffff]
[ 0.232559] pci 0005:01:00.0: BAR 0: assigned [mem 0x600c280000000-0x600c28000ffff 64bit]
[ 0.232695] pci 0005:01:00.0: BAR 2: assigned [mem 0x600c280010000-0x600c280011fff 64bit]
[ 0.232833] pci 0005:00 : [PE# fe] Secondary bus 0 associated with PE#fe
[ 0.233224] pci 0005:01 : [PE# fd] Secondary bus 1 associated with PE#fd
[ 0.233630] pci 0005:01 : [PE# fd] Setting up 32-bit TCE table at 0..80000000
[ 0.254454] pci 0005:01 : [PE# fd] Setting up window#0 0..7fffffff pg=1000
[ 0.254570] pci 0005:01 : [PE# fd] Enabling 64-bit DMA bypass
[ 0.254666] iommu: Adding device 0005:01:00.0 to group 3
[ 0.254756] pci 0005:00:00.0: PCI bridge to [bus 01]
[ 0.254837] pci 0005:00:00.0: bridge window [mem 0x600c280000000-0x600c2ffefffff]
[ 0.255002] pci 0030:00 : [PE# 1fe] Secondary bus 0 associated with PE#1fe
[ 0.255401] pci 0030:00:00.0: PCI bridge to [bus 01-ff]
[ 0.255541] pci 0031:00 : [PE# fe] Secondary bus 0 associated with PE#fe
[ 0.255939] pci 0031:00:00.0: PCI bridge to [bus 01-ff]
[ 0.256071] pci 0032:00 : [PE# fe] Secondary bus 0 associated with PE#fe
[ 0.256464] pci 0032:00:00.0: PCI bridge to [bus 01-ff]
[ 0.256605] pci 0033:00 : [PE# 1fe] Secondary bus 0 associated with PE#1fe
[ 0.257000] pci 0033:00:00.0: PCI bridge to [bus 01-ff]
[ 0.257174] pci 0002:00:00.0: enabling device (0104 -> 0106)
[ 0.257279] pci 0003:00:00.0: enabling device (0104 -> 0106)
[ 0.257375] pci 0004:00:00.0: enabling device (0104 -> 0106)
[ 0.257475] pci 0004:01:00.0: enabling device (0140 -> 0142)
[ 0.257580] pci 0005:00:00.0: enabling device (0104 -> 0106)
[ 0.261242] cpuidle-powernv: Default stop: psscr = 0x0000000000000330,mask=0x00000000003003ff
[ 0.261349] cpuidle-powernv: Deepest stop: psscr = 0x0000000000300375,mask=0x00000000003003ff
[ 0.261461] cpuidle-powernv: Requested Level (RL) value of first deep stop = 0x4
[ 0.264789] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.264874] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.266100] SCSI subsystem initialized
[ 0.266287] pci 0004:02:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 0.266380] pci 0004:02:00.0: vgaarb: bridge control possible
[ 0.266481] pci 0004:02:00.0: vgaarb: setting as boot device (VGA legacy resources not available)
[ 0.266616] vgaarb: loaded
[ 0.266749] usbcore: registered new interface driver usbfs
[ 0.266836] usbcore: registered new interface driver hub
[ 0.266956] usbcore: registered new device driver usb
[ 0.267079] pps_core: LinuxPPS API ver. 1 registered
[ 0.267141] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <email address hidden>
[ 0.267289] PTP clock support registered
[ 0.267410] EDAC MC: Ver: 3.0.0
[ 0.267898] NetLabel: Initializing
[ 0.267952] NetLabel: domain hash size = 128
[ 0.268023] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
[ 0.268136] NetLabel: unlabeled traffic allowed by default
[ 0.268416] clocksource: Switched to clocksource timebase
[ 0.296743] VFS: Disk quotas dquot_6.6.0
[ 0.296869] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[ 0.297880] AppArmor: AppArmor Filesystem Enabled
[ 0.300950] NET: Registered protocol family 2
[ 0.301296] tcp_listen_portaddr_hash hash table entries: 4096 (order: 0, 65536 bytes)
[ 0.301449] TCP established hash table entries: 32768 (order: 2, 262144 bytes)
[ 0.301685] TCP bind hash table entries: 32768 (order: 3, 524288 bytes)
[ 0.301958] TCP: Hash tables configured (established 32768 bind 32768)
[ 0.302088] UDP hash table entries: 2048 (order: 0, 65536 bytes)
[ 0.302207] UDP-Lite hash table entries: 2048 (order: 0, 65536 bytes)
[ 0.302383] NET: Registered protocol family 1
[ 0.302465] NET: Registered protocol family 44
[ 0.302607] pci 0005:01:00.0: enabling device (0140 -> 0142)
[ 0.302794] Unpacking initramfs...
04:02:00.0: enabling device (0140 -> 0142)
[ 1.168174] Using unsupported 1024x768 vga at 600c200010000, depth=32, pitch=4096
[ 1.359211] Console: switching to colour frame buffer device 128x48
[ 1.550200] fb0: Open Firmware frame buffer device on /pciex@600c3c0400000/pci@0/pci@0/vga@0
[ 1.550515] hvc0: raw protocol on /ibm,opal/consoles/serial@0 (boot console)
[ 1.550617] hvc0: No interrupts property, using OPAL event
[ 1.551123] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[ 1.555209] Linux agpgart interface v0.103
[ 1.557546] loop: module loaded
[ 1.557985] libphy: Fixed MDIO Bus: probed
[ 1.558043] tun: Universal TUN/TAP device driver, 1.6
[ 1.558166] PPP generic driver version 2.4.2
[ 1.558327] VFIO - User Level meta-driver version: 0.3
[ 1.558555] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 1.558685] ehci-pci: EHCI PCI platform driver
[ 1.558771] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 1.558868] ohci-pci: OHCI PCI platform driver
[ 1.558954] uhci_hcd: USB Universal Host Controller Interface driver
[ 1.559150] xhci_hcd 0005:01:00.0: xHCI Host Controller
[ 1.559228] xhci_hcd 0005:01:00.0: new USB bus registered, assigned bus number 1
[ 1.559426] xhci_hcd 0005:01:00.0: Using 64-bit DMA iommu bypass
[ 1.559565] xhci_hcd 0005:01:00.0: hcc params 0x0270f06d hci version 0x96 quirks 0x0000000004000000
[ 1.560030] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.18
[ 1.560138] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.560261] usb usb1: Product: xHCI Host Controller
[ 1.560331] usb usb1: Manufacturer: Linux 4.18.0-21-generic xhci-hcd
[ 1.560441] usb usb1: SerialNumber: 0005:01:00.0
[ 1.560724] hub 1-0:1.0: USB hub found
[ 1.560796] hub 1-0:1.0: 4 ports detected
[ 1.561133] xhci_hcd 0005:01:00.0: xHCI Host Controller
[ 1.561206] xhci_hcd 0005:01:00.0: new USB bus registered, assigned bus number 2
[ 1.561335] xhci_hcd 0005:01:00.0: Host supports USB 3.0 SuperSpeed
[ 1.561487] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[ 1.561648] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 4.18
[ 1.561757] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.561869] usb usb2: Product: xHCI Host Controller
[ 1.561943] usb usb2: Manufacturer: Linux 4.18.0-21-generic xhci-hcd
[ 1.562031] usb usb2: SerialNumber: 0005:01:00.0
[ 1.562272] hub 2-0:1.0: USB hub found
[ 1.562346] hub 2-0:1.0: 4 ports detected
[ 1.563183] mousedev: PS/2 mouse device common for all mice
[ 1.580480] rtc-opal opal-rtc: rtc core: registered rtc-opal as rtc0
[ 1.580573] i2c /dev entries driver
[ 1.581820] tpm_i2c_nuvoton 2-0057: VID: 1050 DID: FE RID: 04
[ 1.600613] random: crng init done
[ 24.723500] usb 1-3: new high-speed USB device number 2 using xhci_hcd
[ 34.535347] usb 1-3: New USB device found, idVendor=0557, idProduct=7000, bcdDevice= 0.00
[ 34.587494] usb 1-3: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 37.519907] usb 2-4: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[ 44.571763] usb 2-4: New USB device found, idVendor=0451, idProduct=8140, bcdDevice= 1.00
[ 44.583774] usb 2-4: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 64.940603] hub 1-3:1.0: USB hub found
[ 64.971852] hub 1-3:1.0: 4 ports detected
[ 71.396632] hub 2-4:1.0: USB hub found
[ 71.396899] hub 2-4:1.0: 4 ports detected
[ 71.398889] device-mapper: uevent: version 1.0.3
[ 71.399090] device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: <email address hidden>
[ 71.399225] powernv-cpufreq: cpufreq pstate min 0x62 nominal 0x60 max 0x0
[ 71.399327] powernv-cpufreq: Workload Optimized Frequency is enabled in the platform
[ 71.399671] ledtrig-cpu: registered to indicate activity on CPUs
[ 71.400109] NET: Registered protocol family 10
[ 71.404448] Segment Routing with IPv6
[ 71.404524] NET: Registered protocol family 17
[ 71.404632] Key type dns_resolver registered
[ 71.404779] drmem: No dynamic reconfiguration memory found
[ 71.404987] registered taskstats version 1
[ 71.405056] Loading compiled-in X.509 certificates
[ 71.408094] Loaded X.509 cert 'Build time autogenerated kernel key: bceaac41ed085e5e92aa0c0760c8502ce62fd026'
[ 71.408220] zswap: loaded using pool lzo/zbud
[ 71.412074] Key type big_key registered
[ 71.412120] Key type trusted registered
[ 71.414136] Key type encrypted registered
[ 71.414178] AppArmor: AppArmor sha1 policy hashing enabled
[ 71.426800] ima: Allocated hash algorithm: sha256
[ 71.428432] Unable to handle kernel paging request for data at address 0x00000000
[ 71.428891] Faulting instruction address: 0xc0000000080340f4
[ 71.428953] Oops: Kernel access of bad area, sig: 7 [#1]
[ 71.428999] LE SMP NR_CPUS=2048 NUMA PowerNV
[ 71.429056] Modules linked in:
[ 71.429108] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.18.0-21-generic #22-Ubuntu
[ 71.429195] NIP: c0000000080340f4 LR: c0000000080a76fc CTR: c0000000080340b0
[ 71.429272] REGS: c00000000978b7b0 TRAP: 0300 Not tainted (4.18.0-21-generic)
[ 71.429358] MSR: 9000000000001033 <SF,HV,ME,IR,DR,RI,LE> CR: 48422824 XER: 00000000
[ 71.429451] CFAR: c0000000080340e4 DAR: 0000000000000000 DSISR: 00080000 IRQMASK: 1
[ 71.429451] GPR00: 0000000000000000 c00000000978ba30 c00000000978ca00 0000000000000005
[ 71.429451] GPR04: 0000000000000004 c0000000095f5860 0000000000000008 0000000000000000
[ 71.429451] GPR08: 0000000000000000 0000000000000000 c000000009788000 c0000001076a3600
[ 71.429451] GPR12: c000000008b0f240 c000000009b12000 0000000000000000 0000000009b20600
[ 71.429451] GPR16: 000000002ffa0000 0000000000000008 00000a452fe89760 00000a452fda8b00
[ 71.429451] GPR20: 0000000000000000 0000000000000001 00000a452fe370b8 00007fffdf9d8604
[ 71.429451] GPR24: 0000000000000006 0000000000000006 0000000000000000 0000000000000006
[ 71.429451] GPR28: c0000000096ba948 c0000001076b7748 00000000003003ff 0000000000300375
[ 71.430142] NIP [c0000000080340f4] lwarx_loop_stop+0x0/0x24
[ 71.430199] LR [c0000000080a76fc] __power9_idle_type+0x7c/0xb0
[ 71.430269] Call Trace:
[ 71.430296] [c00000000978ba30] [c00000000978baa0] init_stack+0x3aa0/0x4000 (unreliable)
[ 71.430375] [c00000000978bd20] [c0000000080a76fc] __power9_idle_type+0x7c/0xb0
[ 71.430463] [c00000000978bd60] [c0000000080a7cc0] power9_idle_type+0x20/0x40
[ 71.430543] [c00000000978bd80] [c000000008b0f280] stop_loop+0x40/0x5c
[ 71.430616] [c00000000978bdb0] [c000000008b0b614] cpuidle_enter_state+0xa4/0x420
[ 71.430704] [c00000000978be10] [c000000008158c0c] call_cpuidle+0x4c/0x80
[ 71.430776] [c00000000978be30] [c00000000815923c] do_idle+0x34c/0x3d0
[ 71.430848] [c00000000978bea0] [c0000000081594f8] cpu_startup_entry+0x38/0x40
[ 71.430936] [c00000000978bed0] [c00000000800de78] rest_init+0xe0/0xf8
[ 71.431009] [c00000000978bf00] [c000000009124284] start_kernel+0x614/0x634
[ 71.431082] [c00000000978bf90] [c00000000800ac7c] start_here_common+0x1c/0x520
[ 71.431168] Instruction dump:
[ 71.431209] f86d09b8 39800000 480003d8 60000000 60000000 e8a28080 e8850000 7c232000
[ 71.431264] 40800008 4c0002e4 88ed09a9 e9cd09a0 <7de07028> 75e91000 40c2fe2d 7def3878
[ 71.431312] ---[ end trace 1db489a184daa674 ]---
[ 71.513768]
[ 72.513797] Kernel panic - not syncing: Attempted to kill the idle task!
[ 72.597250] Rebooting in 10[ 661.416892857,5] OPAL: Reboot request...
 seconds..
[ 661.417107122,5] RESET: Initiating fast reboot 1...
[ 662.297731917,5] PCI: Clearing all devices...
[ 662.308067833,5] PCI: Resetting PHBs and training links...
[ 669.291748702,5] STB: BOOTKERNEL verified
[ 669.291785415,3] STB: BOOTKERNEL NOT MEASURED. Already exited from boot services
[ 670.060932306,5] PCI: Probing slots...
[ 670.117398441,5] PHB#0000:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..ff SLOT=UIO Slot1
[ 670.117478867,5] PHB#0001:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..ff SLOT=UIO Slot2
[ 670.117550040,5] PHB#0002:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..01 SLOT=Onboard LAN
[ 670.117618347,5] PHB#0002:01:00.0 [EP ] 8086 1589 R:02 C:020000 ( ethernet) LOC_CODE=Onboard LAN
[ 670.117690628,5] PHB#0002:01:00.1 [EP ] 8086 1589 R:02 C:020000 ( ethernet) LOC_CODE=Onboard LAN
[ 670.117760830,5] PHB#0002:01:00.2 [EP ] 8086 1589 R:02 C:020000 ( ethernet) LOC_CODE=Onboard LAN
[ 670.117826072,5] PHB#0002:01:00.3 [EP ] 8086 1589 R:02 C:020000 ( ethernet) LOC_CODE=Onboard LAN
[ 670.117895739,5] PHB#0003:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..01 SLOT=Onboard SAS
[ 670.117958942,5] PHB#0003:01:00.0 [EP ] 9005 028d R:01 C:010700 ( sas) LOC_CODE=Onboard SAS
[ 670.118030489,5] PHB#0004:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..02 SLOT=Onboard BMC
[ 670.118093839,5] PHB#0004:01:00.0 [ETOX] 1a03 1150 R:04 C:060400 B:02..02 LOC_CODE=Onboard BMC
[ 670.118165254,5] PHB#0004:02:00.0 [PCID] 1a03 2000 R:41 C:030000 ( vga) LOC_CODE=Onboard BMC
[ 670.118241207,5] PHB#0005:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..01 SLOT=Onboard USB
[ 670.118304730,5] PHB#0005:01:00.0 [EP ] 104c 8241 R:02 C:0c0330 ( usb-xhci) LOC_CODE=Onboard USB
[ 670.118371853,5] PHB#0030:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..ff SLOT=WIO Slot1
[ 670.118434817,5] PHB#0031:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..ff SLOT=WIO-R Slot
[ 670.118497984,5] PHB#0032:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..ff SLOT=WIO Slot3
[ 670.118563073,5] PHB#0033:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..ff SLOT=WIO Slot2
[ 670.118629958,5] Clearing unused memory:
[ 670.118669313,5] Clearing region 39342000-1000000000
[ 674.387127949,5] Clearing region 200003a44000-200ffce70000
[ 679.177053131,5] Clearing region 2000-20000000
[ 679.195456253,5] Clearing region 200ffe000000-200fff800000
[ 679.196376971,5] IPMI: Resetting boot count on successful boot
[ 679.196421090,5] INIT: Waiting for kernel...
[ 679.196452396,5] INIT: 64-bit LE kernel discovered
[ 679.196485414,3] STB: EV_SEPARATOR (pcr0) NOT MEASURED. No TPM registered/ena Petitboot (v1.7.5-p8f5fc86) 9006-12P 1302NXA

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1832388

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Thadeu Lima de Souza Cascardo (cascardo)]

Manoj,

How reproducible is this? Did you try multiple times? Did you only see this on cosmic?

I haven't been able to see this on an LPAR. This is certainly specific to Power9, by the stack trace, as it involves the power9 idle driver. Still, I haven't found anything on new kernels that seem to fix anything like this, so it's possible newer releases are still affected.

On the other hand, there doesn't seem to be anything here that looks specific to kdump. Maybe the use of nr_cpus=1 or a kernel that is kexec'ed. So, it would be interesting to try these as well.

Finally, if you can reproduce it, we should coordinate, so we turn xmon on and I can grab the system and take a look.

Thanks.
Cascardo.

[Manoj Iyer (manjo)]

On Fri, Jun 14, 2019 at 11:20 AM Thadeu Lima de Souza Cascardo <
<email address hidden>> wrote:

> Manoj,
>
> How reproducible is this? Did you try multiple times? Did you only see
> this on cosmic?
>

It reproduces in this form only on Cosmic, and I have been able to repro
this a few times on this particular power9 (tesilius)

>
> I haven't been able to see this on an LPAR. This is certainly specific
> to Power9, by the stack trace, as it involves the power9 idle driver.
>

However this issue seems to be isolated to this particular power9 system
(tesilius) which is development grade system, and I was not able to repro
this on lewis which is another development grade power9 system. I have done
a few experiments and looks like the issue might be hardware related. I was
sometimes running into CPU hard lockups on this system during my
experiments.

The production level power9s are currently being used by others, and IBM is
shipping us our witherspoon power9 back next week after proc/fw upgrades.
Also these two development systems are scheduled to be shipped back to IBM
soon for upgrade to production hardware.

I will retest this on the witherpoon that we will receive next week and
report back if this can be reproduced there. If not I will close this as
invalid.

> Still, I haven't found anything on new kernels that seem to fix anything
> like this, so it's possible newer releases are still affected.
>
> On the other hand, there doesn't seem to be anything here that looks
> specific to kdump. Maybe the use of nr_cpus=1 or a kernel that is
> kexec'ed. So, it would be interesting to try these as well.
>
> Finally, if you can reproduce it, we should coordinate, so we turn xmon
> on and I can grab the system and take a look.
>
> Thanks.
> Cascardo.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1832388
>
> Title:
> [power9] [Cosmic] kernel panic while testing kdump
>
> Status in The Ubuntu-power-systems project:
> New
> Status in linux package in Ubuntu:
> Incomplete
> Status in linux source package in Cosmic:
> Incomplete
>
> Bug description:
> I installed cosmic on power9 (boston) a non-nvme system, installed
> initramfs-tools from -proposed and install kdump-tools. Triggered a
> dump, but I got a kernel panic. This might be a kernel bug/regression.
>
> ubuntu@tiselius:~$ uname -a
> Linux tiselius 4.18.0-21-generic #22-Ubuntu SMP Wed May 15 13:12:45 UTC
> 2019 ppc64le ppc64le ppc64le GNU/Linux
> ubuntu@tiselius:~$
>
> ubuntu@tiselius:~$ apt policy initramfs-tools
> initramfs-tools:
> Installed: 0.131ubuntu15.2
> Candidate: 0.131ubuntu15.2
> Version table:
> *** 0.131ubuntu15.2 500
> 500 http://ports.ubuntu.com/ubuntu-ports cosmic-proposed/main
> ppc64el Packages
> ubuntu@tiselius:~$
>
> ubuntu@tiselius:~$ apt policy kdump-tools
> kdump-tools:
> Installed: 1:1.6.5-1ubuntu1~18.10.1
> Candidate: 1:1.6.5-1ubuntu1~18.10.1
> Version table:
> *** 1:1.6.5-1ubuntu1~18.10.1 500
> 500 http://ports.ubuntu.com/ubuntu-ports cosmic-updates/main
> ppc64el Packages
> 100 /var/lib/dpkg/status
>...

[Terry Rudd (terrykrudd)]

Manoj, what is next with this bug? Were we able to isolate it to a particular hardware platform?

[Frank Heimes (fheimes)]

@terrykrudd let me cover this for Manoj

The witherpoon hardware that was supposed to be used for another test is currently blocked on LP 1811439 which is again blocked on RT 120488.

But since this bug was opened against cosmic / 18.10 which is no longer in service (and 18.04.3 is planned to be released today with HWE 5.0 and no longer cosmic's 4.18), I retried again on the same hardware where manjo was able to reproduce the situation, but using disco instead of cosmic and I was not able to see the kernel panic anymore.

Hence I'm going to close this ticket now, since it seems to be fixed with the updates and delta that came in between cosmic and disco.
If one faces this issue again, please open a new LP bug for that particular Ubuntu release (in service) you run.