Guest kernel crashes with kvm_pr on POWER8

[Timothy Pearson]

When attempting to use the kvm_pr module with QEMU 2.10 on a POWER8 host, Debian and Ubuntu guests hang and show crashes.

Host kernel is 4.14. Issue is observed with host kernels 4.9 and 4.13 as well; no other host kernels were tested.

Is this the correct place to report a kvm_pr bug?

Output from Ubuntu 17.10 guest:

Quiescing Open Firmware ...
Booting Linux via __start() @ 0x0000000002000000 ...
[ 0.000000] Page sizes from device-tree:
[ 0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
[ 0.000000] base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1
[ 0.000000] base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0
[ 0.000000] Using 1TB segments
[ 0.000000] Initializing hash mmu with SLB
[ 0.000000] Linux version 4.13.0-16-generic (buildd@bos01-ppc64el-029) (gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu2)) #19-Ubuntu SMP Wed Oct 11 18:37:02 UTC 2017 (Ubuntu 4.13.0-16.19-generic 4.13.4)
[ 0.000000] Found initrd at 0xc000000003b00000:0xc0000000048cf68b
[ 0.000000] Using pSeries machine description
[ 0.000000] bootconsole [udbg0] enabled
[ 0.000000] Partition configured for 2 cpus.
[ 0.000000] CPU maps initialized for 1 thread per core
 -> smp_release_cpus()
spinning_secondaries = 1
 <- smp_release_cpus()
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x19
[ 0.000000] phys_mem_size = 0x100000000
[ 0.000000] dcache_bsize = 0x80
[ 0.000000] icache_bsize = 0x80
[ 0.000000] cpu_features = 0x077c7a6c18500249
[ 0.000000] possible = 0x5fffffff18500649
[ 0.000000] always = 0x0000000018100040
[ 0.000000] cpu_user_features = 0xdc0065c2 0xae000000
[ 0.000000] mmu_features = 0x7c006001
[ 0.000000] firmware_features = 0x00000000415a445f
[ 0.000000] htab_hash_mask = 0x3ffff
[ 0.000000] -----------------------------------------------------
[ 0.000000] numa: NODE_DATA [mem 0xfffd7c80-0xfffe3fff]
[ 0.000000] PCI host bridge /pci@800000020000000 ranges:
[ 0.000000] IO 0x0000200000000000..0x000020000000ffff -> 0x0000000000000000
[ 0.000000] MEM 0x0000200080000000..0x00002000ffffffff -> 0x0000000080000000
[ 0.000000] MEM 0x0000210000000000..0x000021ffffffffff -> 0x0000210000000000
[ 0.000000] PPC64 nvram contains 65536 bytes
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x00000000ffffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal empty
[ 0.000000] Device empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x00000000ffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000000ffffffff]
[ 0.000000] percpu: Embedded 4 pages/cpu @c0000000ffe00000 s162840 r0 d99304 u524288
[ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 65472
[ 0.000000] Policy zone: DMA
[ 0.000000] Kernel command line: BOOT_IMAGE=/install/vmlinux file=/cdrom/preseed/ubuntu-server.seed no_timer_check printk.time=1 ---
[ 0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes)
[ 0.000000] Memory: 4070016K/4194304K available (12800K kernel code, 2048K rwdata, 3456K rodata, 4608K init, 3021K bss, 124288K reserved, 0K cma-reserved)
[ 0.000000] random: get_random_u64 called from cache_random_seq_create+0x80/0x180 with crng_init=0
[ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] ftrace: allocating 33631 entries in 13 pages
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=2.
[ 0.000000] Tasks RCU enabled.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[ 0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[ 0.000006] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[ 0.000696] clocksource: timebase mult[1f40000] shift[24] registered
[ 0.001189] Console: colour dummy device 80x25
[ 0.001500] console [hvc0] enabled
[ 0.001500] console [hvc0] enabled
[ 0.001751] bootconsole [udbg0] disabled
[ 0.001751] bootconsole [udbg0] disabled
[ 0.002142] pid_max: default: 32768 minimum: 301
[ 0.002358] Security Framework initialized
[ 0.002377] Yama: becoming mindful.
[ 0.002466] AppArmor: AppArmor initialized
[ 0.007008] Dentry cache hash table entries: 524288 (order: 6, 4194304 bytes)
[ 0.009037] Inode-cache hash table entries: 262144 (order: 5, 2097152 bytes)
[ 0.009144] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes)
[ 0.009282] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes)
[ 0.011066] EEH: pSeries platform initialized
[ 0.011137] POWER8 performance monitor hardware support registered
[ 0.011231] Hierarchical SRCU implementation.
[ 0.012560] smp: Bringing up secondary CPUs ...
[ 0.014620] smp: Brought up 1 node, 2 CPUs
[ 0.014669] numa: Node 0 CPUs: 0-1
[ 0.017357] devtmpfs: initialized
[ 0.020796] evm: security.selinux
[ 0.020816] evm: security.SMACK64
[ 0.020832] evm: security.SMACK64EXEC
[ 0.020849] evm: security.SMACK64TRANSMUTE
[ 0.020865] evm: security.SMACK64MMAP
[ 0.020882] evm: security.ima
[ 0.020898] evm: security.capability
[ 0.021384] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.021428] futex hash table entries: 512 (order: 0, 65536 bytes)
[ 0.022217] NET: Registered protocol family 16
[ 0.023456] EEH: No capable adapters found
[ 0.068790] KVM: Live patching for a fast VM worked
[ 0.069504] cpuidle: using governor ladder
[ 0.069606] cpuidle: using governor menu
[ 0.070109] pstore: using zlib compression
[ 0.070162] pstore: Registered nvram as persistent store backend
Linux ppc64le
#19-Ubuntu SMP W[ 0.073385] PCI: Probing PCI hardware
[ 0.073595] PCI host bridge to bus 0000:00
[ 0.073650] pci_bus 0000:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff])
[ 0.073722] pci_bus 0000:00: root bus resource [mem 0x200080000000-0x2000ffffffff] (bus address [0x80000000-0xffffffff])
[ 0.073827] pci_bus 0000:00: root bus resource [mem 0x210000000000-0x21ffffffffff]
[ 0.073913] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.081145] IOMMU table initialized, virtual merging enabled
[ 0.081231] iommu: Adding device 0000:00:00.0 to group 0
[ 0.083493] HugeTLB registered 16.0 MiB page size, pre-allocated 0 pages
[ 0.085216] SCSI subsystem initialized
[ 0.085722] vgaarb: loaded
[ 0.085885] usbcore: registered new interface driver usbfs
[ 0.085961] usbcore: registered new interface driver hub
[ 0.086096] usbcore: registered new device driver usb
[ 0.086175] pps_core: LinuxPPS API ver. 1 registered
[ 0.086217] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <email address hidden>
[ 0.086316] PTP clock support registered
[ 0.086629] EDAC MC: Ver: 3.0.0
[ 0.087455] NetLabel: Initializing
[ 0.087509] NetLabel: domain hash size = 128
[ 0.087550] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
[ 0.087676] NetLabel: unlabeled traffic allowed by default
[ 0.088226] clocksource: Switched to clocksource timebase
[ 0.109127] VFS: Disk quotas dquot_6.6.0
[ 0.109244] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[ 0.109543] AppArmor: AppArmor Filesystem Enabled
[ 0.121635] NET: Registered protocol family 2
[ 0.122074] TCP established hash table entries: 32768 (order: 2, 262144 bytes)
[ 0.122584] TCP bind hash table entries: 32768 (order: 3, 524288 bytes)
[ 0.123346] TCP: Hash tables configured (established 32768 bind 32768)
[ 0.123472] UDP hash table entries: 2048 (order: 0, 65536 bytes)
[ 0.123692] UDP-Lite hash table entries: 2048 (order: 0, 65536 bytes)
[ 0.123937] NET: Registered protocol family 1
[ 0.124257] Unpacking initramfs...
[ 0.467838] Freeing initrd memory: 14080K
[ 0.472109] audit: initializing netlink subsys (disabled)
[ 0.472949] audit: type=2000 audit(1513569522.428:1): state=initialized audit_enabled=0 res=1
[ 0.473972] Initialise system trusted keyrings
[ 0.474068] Key type blacklist registered
[ 0.474308] workingset: timestamp_bits=38 max_order=16 bucket_order=0
[ 0.476124] zbud: loaded
[ 0.477006] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 0.477456] fuse init (API version 7.26)
[ 0.478394] random: fast init done
[ 0.483013] Key type asymmetric registered
[ 0.483040] Asymmetric key parser 'x509' registered
[ 0.483150] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[ 0.483363] io scheduler noop registered
[ 0.483383] io scheduler deadline registered
[ 0.483450] io scheduler cfq registered (default)
[ 0.484056] virtio-pci 0000:00:00.0: enabling device (0100 -> 0103)
[ 0.485519] virtio-pci 0000:00:00.0: ibm,query-pe-dma-windows(2026) 0 8000000 20000000 returned 0
[ 0.485916] virtio-pci 0000:00:00.0: ibm,create-pe-dma-window(2027) 0 8000000 20000000 10 20 returned 0 (liobn = 0x80000001 starting addr = 8000000 0)
[ 0.501557] virtio-pci 0000:00:00.0: Using 64-bit direct DMA at offset 800000000000000
[ 0.503803] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[ 0.507398] Linux agpgart interface v0.103
[ 0.511296] loop: module loaded
[ 0.511671] libphy: Fixed MDIO Bus: probed
[ 0.511698] tun: Universal TUN/TAP device driver, 1.6
[ 0.511860] PPP generic driver version 2.4.2
[ 0.512086] VFIO - User Level meta-driver version: 0.3
[ 0.512309] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 0.512367] ehci-pci: EHCI PCI platform driver
[ 0.512420] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 0.512457] ohci-pci: OHCI PCI platform driver
[ 0.512501] uhci_hcd: USB Universal Host Controller Interface driver
[ 0.512814] mousedev: PS/2 mouse device common for all mice
[ 0.513152] rtc-generic rtc-generic: rtc core: registered rtc-generic as rtc0
[ 0.513200] i2c /dev entries driver
[ 0.513320] device-mapper: uevent: version 1.0.3
[ 0.513482] device-mapper: ioctl: 4.36.0-ioctl (2017-06-09) initialised: <email address hidden>
[ 0.513710] ledtrig-cpu: registered to indicate activity on CPUs
[ 0.514095] NET: Registered protocol family 10
[ 0.526547] modprobe[89]: unhandled signal 11 at 0000000000000008 nip 000073724fd9645c lr 000073724fd855c0 code 30001
[ 0.528919] modprobe[90]: unhandled signal 11 at 00000000001e4250 nip 000076c0ae90e0f8 lr 000076c0ae90e6a4 code 30001
[ 0.529819] Segment Routing with IPv6
[ 0.529874] NET: Registered protocol family 17
[ 0.529922] Key type dns_resolver registered
[ 0.530832] registered taskstats version 1
[ 0.530902] Loading compiled-in X.509 certificates
[ 0.531719] modprobe[93]: unhandled signal 11 at 0000000000000008 nip 0000741ba74e645c lr 0000741ba74d55c0 code 30001
[ 0.532899] modprobe[94]: unhandled signal 11 at 0000000000000008 nip 0000764dd97f645c lr 0000764dd97e55c0 code 30001
[ 0.534414] Loaded X.509 cert 'Build time autogenerated kernel key: bc297e5938e0456833a4c0c157e5483b77785cf1'
[ 0.534505] zswap: loaded using pool lzo/zbud
[ 0.535375] modprobe[97]: unhandled signal 11 at 0000000000000008 nip 00007e85a34b645c lr 00007e85a34a55c0 code 30001
[ 0.536618] modprobe[98]: unhandled signal 11 at 0000000000000008 nip 0000713d7724645c lr 0000713d772355c0 code 30001
[ 0.537392] Key type big_key registered
[ 0.537418] Key type trusted registered
[ 0.545589] Key type encrypted registered
[ 0.545642] AppArmor: AppArmor sha1 policy hashing enabled
[ 0.545689] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
[ 0.545799] evm: HMAC attrs: 0x1
[ 0.551224] rtc-generic rtc-generic: setting system clock to 2017-12-18 03:58:43 UTC (1513569523)
[ 0.552107] Unable to open file: /etc/keys/x509_ima.der (-2)
[ 0.552109] Unable to open file: /etc/keys/x509_evm.der (-2)
[ 0.591193] Freeing unused kernel memory: 4608K
[ 0.591643] This architecture does not have kernel memory protection.
<hang>

[Timothy Pearson (kb9vqf)]

Is this the correct place to file kvm-pr bug reports?

[Thomas Huth (th-huth)]

No, this bug tracker is for QEMU bugs only. Please report KVM-PR bugs to the <email address hidden> mailing list (see also https://www.linux-kvm.org/page/Bugs for how to report KVM kernel bugs in general)

[Murilo Opsfelder Araújo (mopsfelder)]

Hi, Timothy.

I tried to reproduce this issue on a POWER8 box and couldn't reproduce it.

Whatever the issue was, it seems to be fixed on kernel v4.16-rc4 with qemu 2.11.50.

I downloaded vmlinux/initrd.gz from Ubuntu 18.04 to boot guest. It booted fine up to the installer initial screen.

Please find my environment information listed below.

I'm closing this bug but feel free to reopen it or file a new one.

Cheers
Murilo

Machine type/model: 8247-22L

[muriloo@baratheon ~]$ uname -a
Linux localhost.localdomain 4.16.0-rc4+ #1 SMP Thu Mar 8 22:54:31 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

[muriloo@baratheon ~]$ lsmod | grep kvm
kvm_pr 100276 0
kvm 217753 1 kvm_pr

[muriloo@baratheon ~]$ ~/qemu/build/ppc64-softmmu/qemu-system-ppc64 --version
QEMU emulator version 2.11.50 (v2.11.0-2108-g83d2e94)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

[muriloo@baratheon ~]$ ~/qemu/build/ppc64-softmmu/qemu-system-ppc64 -kernel ~/ubuntu/18.04/vmlinux -initrd ~/ubuntu/18.04/initrd.gz -append "console=hvc0 verbose" -nodefaults -nographic -serial mon:stdio -accel kvm

vmlinux: http://ports.ubuntu.com/ubuntu-ports/dists/bionic/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/vmlinux
initrd.gz: http://ports.ubuntu.com/ubuntu-ports/dists/bionic/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/initrd.gz