qemu-system-ppc64 freezes on starting image on ppc64le

[Mike Rushton]

qemu-system-ppc64 running on Ubuntu 16.04 beta-2 fails to start an image as part of the certification process. This on an IBM ppc64le in PowerVM mode running Ubuntu 16.04 beta-2 deployed by MAAS 1.9.1. There is no error output.

ubuntu@alpine01:~/kvm$ qemu-system-ppc64 -m 256 -display none -nographic -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 -machine pseries -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio -drive file=seed.iso,if=virtio
WARNING: Image format was not specified for 'seed.iso' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.

SLOF **********************************************************************
QEMU Starting
 Build Date = Jan 29 2016 18:58:37
 FW Version = buildd@ release 20151103
 Press "s" to enter Open Firmware.

Populating /vdevice methods
Populating /vdevice/vty@71000000
Populating /vdevice/nvram@71000001
Populating /vdevice/l-lan@71000002
Populating /vdevice/v-scsi@71000003
       SCSI: Looking for devices
          8200000000000000 CD-ROM : "QEMU QEMU CD-ROM 2.5+"
Populating /pci@800000020000000
                     00 1800 (D) : 1af4 1001 virtio [ block ]
                     00 1000 (D) : 1af4 1001 virtio [ block ]
                     00 0800 (D) : 106b 003f serial bus [ usb-ohci ]
                     00 0000 (D) : 1234 1111 qemu vga
No NVRAM common partition, re-initializing...
Installing QEMU fb

Scanning USB
  OHCI: initializing
    USB Keyboard
    USB mouse
No console specified using screen & keyboard

  Welcome to Open Firmware

  Copyright (c) 2004, 2011 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php

Trying to load: from: /pci@800000020000000/scsi@3 ...
E3404: Not a bootable device!
Trying to load: from: /pci@800000020000000/scsi@2 ... Successfully loaded
Linux ppc64le
#31-Ubuntu SMP F

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: qemu-system-ppc 1:2.5+dfsg-5ubuntu6
ProcVersionSignature: Ubuntu 4.4.0-16.32-generic 4.4.6
Uname: Linux 4.4.0-16-generic ppc64le
ApportVersion: 2.20-0ubuntu3
Architecture: ppc64el
Date: Wed Mar 30 14:10:01 2016
KvmCmdLine:
 COMMAND STAT EUID RUID PID PPID %CPU COMMAND
 kvm-irqfd-clean S< 0 0 1172 2 0.0 [kvm-irqfd-clean]
 qemu-nbd Ssl 0 0 13467 1 0.0 qemu-nbd -c /dev/nbd0 xenial-server-cloudimg-ppc64el-disk1.img
 qemu-system-ppc Sl+ 1000 1000 18973 18896 101 qemu-system-ppc64 -m 256 -display none -nographic -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 -machine pseries -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio -drive file=seed.iso,if=virtio
Lsusb: Error: command ['lsusb'] failed with exit code 1:
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinux-4.4.0-16-generic root=UUID=92d820c8-ab25-497b-9b1e-f1435992bbf3 ro
ProcLoadAvg: 1.08 0.94 0.58 2/616 19571
ProcLocks:
 1: POSIX ADVISORY WRITE 886 00:13:381 0 EOF
 2: POSIX ADVISORY WRITE 1339 00:13:528 0 EOF
 3: FLOCK ADVISORY WRITE 1284 00:13:522 0 EOF
 4: POSIX ADVISORY WRITE 2281 00:13:563 0 EOF
 5: POSIX ADVISORY WRITE 1331 00:13:536 0 EOF
ProcSwaps:
 Filename Type Size Used Priority
 /swap.img file 8388544 0 -1
ProcVersion: Linux version 4.4.0-16-generic (buildd@bos01-ppc64el-001) (gcc version 5.3.1 20160320 (Ubuntu/Linaro/IBM 5.3.1-12ubuntu4) ) #32-Ubuntu SMP Thu Mar 24 22:31:14 UTC 2016
SourcePackage: qemu
UpgradeStatus: No upgrade log present (probably fresh install)
bootlist:
 /pci@800000020000011/pci1014,034A@0/sas/disk@4068402c40
 /pci@800000020000018/ethernet@0:speed=auto,duplex=auto,csarch,000.000.000.000,,000.000.000.000,000.000.000.000,5,5,000.000.000.000,512
 /pci@800000020000018/ethernet@0,1:speed=auto,duplex=auto,csarch,000.000.000.000,,000.000.000.000,000.000.000.000,5,5,000.000.000.000,512
 /pci@800000020000018/ethernet@0,2:speed=auto,duplex=auto,csarch,000.000.000.000,,000.000.000.000,000.000.000.000,5,5,000.000.000.000,512
 /pci@800000020000018/ethernet@0,3:speed=auto,duplex=auto,csarch,000.000.000.000,,000.000.000.000,000.000.000.000,5,5,000.000.000.000,512
cpu_cores: Number of cores present = 8
cpu_coreson: Number of cores online = 8
cpu_smt: SMT=8
lscfg_vp: Error: [Errno 2] No such file or directory: 'lscfg'
lsmcode: Error: [Errno 2] No such file or directory: 'lsmcode'

[Mike Rushton (leftyfb)]

In Ubuntu 14.04.4 running on the same hardware, the fix was to update qemu-system-ppc from cloud-archive:kilo. This repository is only supported on trusty and contains an earlier version of qemu-system-ppc than what is available on xenial.

[Serge Hallyn (serge-hallyn)]

Hi,

looking at the information in the Description, it looks like you still have /dev/nbd0 attached to the file you're using in qemu. Just to be sure, does this still happen when you disconnect that first?

[Mike Rushton (leftyfb)]

Yes. I only mounted (a different file) to troubleshoot what was going on. The issue was happening before and after mounting. I have since unmounted it.

[Serge Hallyn (serge-hallyn)]

Could you try this using upstream qemu?

git clone http://git.qemu.org/git/qemu.git
cd qemu
sudo apt-get -y build-dep qemu
./configure --target-list=ppc64-softmmu
make
cd ppc64-softmmu
./qemu-system-ppc64 [...]

[Mike Rushton (leftyfb)]

Same issue. Stopped on the exact same spot.

[Serge Hallyn (serge-hallyn)]

Just to be sure before I mark this as affecting upstream, could you
try adding more memory, maybe -m 1024?

[Mike Rushton (leftyfb)]

Same issue with -m 2048

[Serge Hallyn (serge-hallyn)]

Failing any brighter ideas, this should be pretty bisectable to figure out what happened. Hardware availability is the main problem. Would you be able to use your system to bisect to the commit introducing the bug?

Actually before we get to that, could you try installing the 14.04 slof package on the 16.04 system, and see whether that fixes it?

[Mike Rushton (leftyfb)]

I have installed qemu-slof from trusty and ran again:

qemu-slof:
  Installed: 20151103+dfsg-1ubuntu1
  Candidate: 20151103+dfsg-1ubuntu1
  Version table:
 *** 20151103+dfsg-1ubuntu1 500
        500 http://ports.ubuntu.com/ubuntu-ports xenial/main ppc64el Packages
        100 /var/lib/dpkg/status
     20131015+dfsg-1ubuntu1 500
        500 http://ports.ubuntu.com/ubuntu-ports trusty/universe ppc64el Packages
ubuntu@alpine01:~$ sudo apt-get install qemu-slof=20131015+dfsg-1ubuntu1

It froze on the exact same spot.

[Serge Hallyn (serge-hallyn)]

Thanks very much for testing.

I think I have a system I can use to try and bisect tonight/tomorrow.

[Serge Hallyn (serge-hallyn)]

Hm, building 2.2.0 (close to what is in the kilo cloud archive) doesn't help.

[Serge Hallyn (serge-hallyn)]

I'll try building the package source from kilo on the xenial host and see if that succeeds. I'm having doubts.

[Serge Hallyn (serge-hallyn)]

Indeed building the kilo package from source gives me the same hang. So something else (seabios maybe)

[Serge Hallyn (serge-hallyn)]

Hi,

I've redeployed my test box with 14.04 with kilo-staging archive, but i get a core dump when i try to run kvm the same way you did.

Can you show your /etc/apt/sources.list and /etc/apt/sources.list.d, as well as output for

uname -a
dpkg -l | egrep -e '(qemu|linux|bios)'

[Michael Roth (mdroth)]

Quoting Serge Hallyn (2016-04-01 11:56:29)
> Hi,
>
> I've redeployed my test box with 14.04 with kilo-staging archive, but i
> get a core dump when i try to run kvm the same way you did.

What does `ppc64_cpu --info` report? The original bug had some output
that suggested SMT was enabled:

> cpu_cores: Number of cores present = 8
> cpu_coreson: Number of cores online = 8
> cpu_smt: SMT=8

But AFAIK, SMT should be disabled in the host (ppc64_cpu --smt off)
and left to individual guests to enable/disable in guest mode.

For KVM on powernv host kernel at least... original bug suggested
PowerVM mode:

> Bug description:
> qemu-system-ppc64 running on Ubuntu 16.04 beta-2 fails to start an
> image as part of the certification process. This on an IBM ppc64le in
> PowerVM mode running Ubuntu 16.04 beta-2 deployed by MAAS 1.9.1. There
> is no error output.

which I'm assuming didn't actually mean PowerVM instead of
powernv+KVM? Or was this actually a PowerVM partition trying to run
a KVM guest?

[Serge Hallyn (serge-hallyn)]

Hm - I can boot a wily cloud image, just not a xenial one.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Serge Hallyn (serge-hallyn)]

Result of doing qemu-system-ppc64 -m 1024 -vnc :1 -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 -machine pseries -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio -drive file=my-seed.img,if=virtio

[Mike Rushton (leftyfb)]

ubuntu@alpine01:~$ ppc64_cpu --info
Core 0: 0* 1* 2* 3* 4* 5* 6* 7*
Core 1: 8* 9* 10* 11* 12* 13* 14* 15*
Core 2: 16* 17* 18* 19* 20* 21* 22* 23*
Core 3: 24* 25* 26* 27* 28* 29* 30* 31*
Core 4: 32* 33* 34* 35* 36* 37* 38* 39*
Core 5: 40* 41* 42* 43* 44* 45* 46* 47*
Core 6: 48* 49* 50* 51* 52* 53* 54* 55*
Core 7: 56* 57* 58* 59* 60* 61* 62* 63*
ubuntu@alpine01:~$ ppc64_cpu --smt off
SMT=8
ubuntu@alpine01:~$ ppc64_cpu --info
Core 0: 0* 1* 2* 3* 4* 5* 6* 7*
Core 1: 8* 9* 10* 11* 12* 13* 14* 15*
Core 2: 16* 17* 18* 19* 20* 21* 22* 23*
Core 3: 24* 25* 26* 27* 28* 29* 30* 31*
Core 4: 32* 33* 34* 35* 36* 37* 38* 39*
Core 5: 40* 41* 42* 43* 44* 45* 46* 47*
Core 6: 48* 49* 50* 51* 52* 53* 54* 55*
Core 7: 56* 57* 58* 59* 60* 61* 62* 63*

Still freezing at the same point:

Trying to load: from: disk ... Successfully loaded
* finddevice /memory grub workaround *
* finddevice /memory grub workaround *
* finddevice /memory grub workaround *
Linux ppc64le
#31-Ubuntu SMP F

This is running on an LPAR in PowerNV mode.

[Serge Hallyn (serge-hallyn)]

Actually the clou dimages have a 4.2 kernel. When I use a xenial beta2 iso which has 4.4.0-15-generic #31, it boots fine. I can install, and I can boot the installed image (with same kernel) just fine.

[Serge Hallyn (serge-hallyn)]

4.4.0-16 also works.

[Michael Roth (mdroth)]

Quoting Mike Rushton (2016-04-04 09:49:29)
> ubuntu@alpine01:~$ ppc64_cpu --info
> Core 0: 0* 1* 2* 3* 4* 5* 6* 7*
> Core 1: 8* 9* 10* 11* 12* 13* 14* 15*
> Core 2: 16* 17* 18* 19* 20* 21* 22* 23*
> Core 3: 24* 25* 26* 27* 28* 29* 30* 31*
> Core 4: 32* 33* 34* 35* 36* 37* 38* 39*
> Core 5: 40* 41* 42* 43* 44* 45* 46* 47*
> Core 6: 48* 49* 50* 51* 52* 53* 54* 55*
> Core 7: 56* 57* 58* 59* 60* 61* 62* 63*
> ubuntu@alpine01:~$ ppc64_cpu --smt off
> SMT=8
> ubuntu@alpine01:~$ ppc64_cpu --info
> Core 0: 0* 1* 2* 3* 4* 5* 6* 7*
> Core 1: 8* 9* 10* 11* 12* 13* 14* 15*
> Core 2: 16* 17* 18* 19* 20* 21* 22* 23*
> Core 3: 24* 25* 26* 27* 28* 29* 30* 31*
> Core 4: 32* 33* 34* 35* 36* 37* 38* 39*
> Core 5: 40* 41* 42* 43* 44* 45* 46* 47*
> Core 6: 48* 49* 50* 51* 52* 53* 54* 55*
> Core 7: 56* 57* 58* 59* 60* 61* 62* 63*

Yah, this is not right.

I think the command you need is actually:

  ppc64_cpu --smt=1

I don't know if this is the root cause, but I would not
expect things to work in general without SMT=1

>
> Still freezing at the same point:
>
> Trying to load: from: disk ... Successfully loaded
> * finddevice /memory grub workaround *
> * finddevice /memory grub workaround *
> * finddevice /memory grub workaround *
> Linux ppc64le
> #31-Ubuntu SMP F
>
>
> This is running on an LPAR in PowerNV mode.
>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1563887
>
> Title:
> qemu-system-ppc64 freezes on starting image on ppc64le
>
> Status in QEMU:
> New
> Status in linux package in Ubuntu:
> Confirmed
> Status in qemu package in Ubuntu:
> Confirmed
>
> Bug description:
> qemu-system-ppc64 running on Ubuntu 16.04 beta-2 fails to start an
> image as part of the certification process. This on an IBM ppc64le in
> PowerVM mode running Ubuntu 16.04 beta-2 deployed by MAAS 1.9.1. There
> is no error output.
>
> ubuntu@alpine01:~/kvm$ qemu-system-ppc64 -m 256 -display none -nographic -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 -machine pseries -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio -drive file=seed.iso,if=virtio
> WARNING: Image format was not specified for 'seed.iso' and probing guessed raw.
>          Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
>          Specify the 'raw' format explicitly to remove the restrictions.
>
> SLOF **********************************************************************
> QEMU Starting
>  Build Date = Jan 29 2016 18:58:37
>  FW Version = buildd@ release 20151103
>  Press "s" to enter Open Firmware.
>
> Populating /vdevice methods
> Populating /vdevice/vty@71000000
> Populating /vdevice/nvram@71000001
> Populating /vdevice/l-lan@71000002
> P...

[Mike Rushton (leftyfb)]

ubuntu@alpine01:~$ sudo ppc64_cpu --smt=1
ubuntu@alpine01:~$ ppc64_cpu --info
Core 0: 0* 1 2 3 4 5 6 7
Core 1: 8* 9 10 11 12 13 14 15
Core 2: 16* 17 18 19 20 21 22 23
Core 3: 24* 25 26 27 28 29 30 31
Core 4: 32* 33 34 35 36 37 38 39
Core 5: 40* 41 42 43 44 45 46 47
Core 6: 48* 49 50 51 52 53 54 55
Core 7: 56* 57 58 59 60 61 62 63

ubuntu@alpine01:~$ sudo ppc64_cpu --smt
SMT is off

Still freezing at the same point.

[Serge Hallyn (serge-hallyn)]

If you can reproduce this with the ppc64 xenial iso or a rootfs
installed from that, using 4.4 kernel, please let us know. Otherwise,
I think the fix will be for cloud images to be updated with a 4.4 kernel.

[Michael Roth (mdroth)]

Quoting Mike Rushton (2016-04-05 13:10:17)
> ubuntu@alpine01:~$ sudo ppc64_cpu --smt=1
> ubuntu@alpine01:~$ ppc64_cpu --info
> Core 0: 0* 1 2 3 4 5 6 7
> Core 1: 8* 9 10 11 12 13 14 15
> Core 2: 16* 17 18 19 20 21 22 23
> Core 3: 24* 25 26 27 28 29 30 31
> Core 4: 32* 33 34 35 36 37 38 39
> Core 5: 40* 41 42 43 44 45 46 47
> Core 6: 48* 49 50 51 52 53 54 55
> Core 7: 56* 57 58 59 60 61 62 63
>
> ubuntu@alpine01:~$ sudo ppc64_cpu --smt
> SMT is off
>
>
> Still freezing at the same point.
>
> Bug description:
> qemu-system-ppc64 running on Ubuntu 16.04 beta-2 fails to start an
> image as part of the certification process. This on an IBM ppc64le in
> PowerVM mode running Ubuntu 16.04 beta-2 deployed by MAAS 1.9.1. There
> is no error output.
>
> ubuntu@alpine01:~/kvm$ qemu-system-ppc64 -m 256 -display none -nographic -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 -machine pseries -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio -drive file=seed.iso,if=virtio
> WARNING: Image format was not specified for 'seed.iso' and probing guessed raw.
>          Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
>          Specify the 'raw' format explicitly to remove the restrictions.

Unless things are different with Ubuntu, I don't think KVM is enabled by default. Try your command with -enable-kvm to be sure.

[Mike Rushton (leftyfb)]

@serge-hallyn

ubuntu@alpine01:~/kvm$ cat /etc/issue; uname -a ; ppc64_cpu --smt
Ubuntu Xenial Xerus (development branch) \n \l

Linux alpine01 4.4.0-17-generic #33-Ubuntu SMP Tue Mar 29 17:15:31 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
SMT is off
ubuntu@alpine01:~/kvm$ qemu-system-ppc64 -m 256 -display none -nographic -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 -machine pseries -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio -drive file=seed.iso,if=virtio
WARNING: Image format was not specified for 'seed.iso' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.

SLOF **********************************************************************
QEMU Starting
 Build Date = Jan 29 2016 18:58:37
 FW Version = buildd@ release 20151103
 Press "s" to enter Open Firmware.

Populating /vdevice methods
Populating /vdevice/vty@71000000
Populating /vdevice/nvram@71000001
Populating /vdevice/l-lan@71000002
Populating /vdevice/v-scsi@71000003
       SCSI: Looking for devices
          8200000000000000 CD-ROM : "QEMU QEMU CD-ROM 2.5+"
Populating /pci@800000020000000
                     00 1800 (D) : 1af4 1001 virtio [ block ]
                     00 1000 (D) : 1af4 1001 virtio [ block ]
                     00 0800 (D) : 106b 003f serial bus [ usb-ohci ]
                     00 0000 (D) : 1234 1111 qemu vga
No NVRAM common partition, re-initializing...
Installing QEMU fb

Scanning USB
  OHCI: initializing
    USB Keyboard
    USB mouse
No console specified using screen & keyboard

  Welcome to Open Firmware

  Copyright (c) 2004, 2011 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php

Trying to load: from: /pci@800000020000000/scsi@3 ...
E3404: Not a bootable device!
Trying to load: from: /pci@800000020000000/scsi@2 ... Successfully loaded
Linux ppc64le
#31-Ubuntu SMP F

[Mike Rushton (leftyfb)]

@mdroth

I don't think PPC supports kvm the same way as x86:

ubuntu@alpine01:~/kvm$ qemu-system-ppc64 -enable-kvm -m 256 -display none -nographic -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 -machine pseries -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio -drive file=seed.iso,if=virtio
Could not access KVM kernel module: Permission denied
failed to initialize KVM: Permission denied

[Michael Roth (mdroth)]

Quoting Mike Rushton (2016-04-05 16:47:29)
> @mdroth
>
> I don't think PPC supports kvm the same way as x86:
>
> ubuntu@alpine01:~/kvm$ qemu-system-ppc64 -enable-kvm -m 256 -display none -nographic -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 -machine pseries -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio -drive file=seed.iso,if=virtio
> Could not access KVM kernel module: Permission denied
> failed to initialize KVM: Permission denied

-enable-kvm should work as expected on Power machines that support
KVM.

You seem to be running the command as a normal user. KVM generally
requires a priviledged user.

[Mike Rushton (leftyfb)]

Sorry, I copied the wrong thing:

sudo qemu-system-ppc64 -enable-kvm -m 256 -display none -nographic -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 -machine pseries -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio -drive file=seed.iso,if=virtio
ioctl(KVM_CREATE_VM) failed: 22 Invalid argument
failed to initialize KVM: Invalid argument

[Michael Roth (mdroth)]

Quoting Mike Rushton (2016-04-05 17:22:24)
> Sorry, I copied the wrong thing:
>
> sudo qemu-system-ppc64 -enable-kvm -m 256 -display none -nographic -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 -machine pseries -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio -drive file=seed.iso,if=virtio
> ioctl(KVM_CREATE_VM) failed: 22 Invalid argument
> failed to initialize KVM: Invalid argument

Can you post the output of this from the host?

  cat /proc/cpuinfo | grep platform

If it reports "PowerNV" then things should be working, so the error
needs investigation.

If it reports "pSeries", then that's not a normally supported scenario
for running KVM (since your host is already running as virtualized
pSeries partition running on top of the pHyp hypervisor rather than
KVM). You *might* have KVM support via 'PR' mode as opposed to the normal
'HV' mode, but that's generally used to do nested virtualization on top
of a KVM HV guest. To do it on top of a pHyp guest partition is probably
not something that's been actively tested.

I'd assume you're using PowerNV host, but you mentioned "PowerVM"
earlier which is synonymous with pHyp, so it warrants confirming.

>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1563887
>
> Title:
> qemu-system-ppc64 freezes on starting image on ppc64le
>
> Status in QEMU:
> New
> Status in linux package in Ubuntu:
> Confirmed
> Status in livecd-rootfs package in Ubuntu:
> New
> Status in qemu package in Ubuntu:
> Confirmed
>
> Bug description:
> qemu-system-ppc64 running on Ubuntu 16.04 beta-2 fails to start an
> image as part of the certification process. This on an IBM ppc64le in
> PowerVM mode running Ubuntu 16.04 beta-2 deployed by MAAS 1.9.1. There
> is no error output.
>
> ubuntu@alpine01:~/kvm$ qemu-system-ppc64 -m 256 -display none -nographic -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 -machine pseries -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio -drive file=seed.iso,if=virtio
> WARNING: Image format was not specified for 'seed.iso' and probing guessed raw.
>          Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
>          Specify the 'raw' format explicitly to remove the restrictions.
>
> SLOF **********************************************************************
> QEMU Starting
>  Build Date = Jan 29 2016 18:58:37
>  FW Version = buildd@ release 20151103
>  Press "s" to enter Open Firmware.
>
> Populating /vdevice methods
> Populating /vdevice/vty@71000000
> Populating /vdevice/nvram@71000001
> Populating /vdevice/l-lan@71000002
> Populating /vdevice/v-scsi@71000003
>        SCSI: Looking for devices
>           8200000000000000 CD-ROM : "QEMU QEMU CD-ROM 2.5+"
> Populating /pci@800000020000000
>                      00 1800 (D) : 1af4 1001 virtio [ block ]
>                      00 1000 (D) : 1af4 1001 virtio [ block ]
>                      00 0800 (D) : 106b 003f serial bus [ us...

[Mike Rushton (leftyfb)]

The PowerVM machine I've been using for these tests has the following output:

ubuntu@alpine01:~/kvm$ cat /proc/cpuinfo | grep platform
platform : pSeries

The PowerNV server has the following output:

ubuntu@binacle:~$ cat /etc/issue; uname -a ; ppc64_cpu --smt ; cat /proc/cpuinfo |grep platform
Ubuntu Xenial Xerus (development branch) \n \l

Linux binacle 4.4.0-17-generic #33-Ubuntu SMP Tue Mar 29 17:15:31 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
SMT is off
platform : PowerNV

ubuntu@binacle:~$ sudo qemu-system-ppc64 -enable-kvm -m 256 -display none -nographic -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 -machine pseries -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio -drive file=seed.iso,if=virtio
WARNING: Image format was not specified for 'seed.iso' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.

SLOF **********************************************************************
QEMU Starting
 Build Date = Jan 29 2016 18:58:37
 FW Version = buildd@ release 20151103
 Press "s" to enter Open Firmware.

Populating /vdevice methods
Populating /vdevice/vty@71000000
Populating /vdevice/nvram@71000001
Populating /vdevice/l-lan@71000002
Populating /vdevice/v-scsi@71000003
       SCSI: Looking for devices
          8200000000000000 CD-ROM : "QEMU QEMU CD-ROM 2.5+"
Populating /pci@800000020000000
                     00 1800 (D) : 1af4 1001 virtio [ block ]
                     00 1000 (D) : 1af4 1001 virtio [ block ]
                     00 0800 (D) : 106b 003f serial bus [ usb-ohci ]
                     00 0000 (D) : 1234 1111 qemu vga
No NVRAM common partition, re-initializing...
Installing QEMU fb

Scanning USB
  OHCI: initializing
    USB Keyboard
    USB mouse
No console specified using screen & keyboard

  Welcome to Open Firmware

  Copyright (c) 2004, 2011 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php

Trying to load: from: /pci@800000020000000/scsi@3 ...
E3404: Not a bootable device!
Trying to load: from: /pci@800000020000000/scsi@2 ... Successfully loaded
Linux ppc64le
#32-Ubuntu SMP T

[Serge Hallyn (serge-hallyn)]

Ok so if I'm following this right there are two issues:

1. the bug reporter is using a powervm partition. KVM cannot be used there. This is not a KVM bug.

2. the xenial cloud images have an outdated 4.2 kernel which doesn't boot in kvm on powernv. A workaround is to use the isos which do boot. This is a cloud-images bug.

AFAICS there is no qemu bug here, so marking invalid for that package.

[Mike Rushton (leftyfb)]

1. the bug reporter is using a powervm partition. KVM cannot be used there. This is not a KVM bug.

PowerVM mode is an LPAR. It is not a kvm instance trying to run an KVM within. This was also working in 14.04 without issue and is being asked of us from IBM to certify

2. the xenial cloud images have an outdated 4.2 kernel which doesn't boot in kvm on powernv. A workaround is to use the isos which do boot. This is a cloud-images bug.

All the Xenial daily images i've been using from MAAS deployments have been the 4.4 kernel which is still not working.

[Mike Rushton (leftyfb)]

I have also shown that this issue has the exact same results on PowerNV above.

[Scott Moser (smoser)]

ok. so after playing some, this all comes down to needing 'usb=off' on the qemu command line.

works:
qemu-system-ppc64 -enable-kvm -machine usb=off -device virtio-net-pci,netdev=net00 -netdev type=user,id=net00 -drive if=virtio,file=xenial-server-cloudimg-ppc64el-disk1.img,format=qcow2 -drive if=virtio,file=my-seed.img,format=raw -snapshot -m 256 -nographic

doesnt work:
qemu-system-ppc64 -enable-kvm virtio-net-pci,netdev=net00 -netdev type=user,id=net00 -drive if=virtio,file=xenial-server-cloudimg-ppc64el-disk1.img,format=qcow2 -drive if=virtio,file=my-seed.img,format=raw -snapshot -m 256 -nographic

the interesting bit of 'doesnt work' is that it *does* work, the user just thinks it doesnt because output is eaten up and doesnt go to console, but it *does* work fine.
http://paste.ubuntu.com/15660816/

[Scott Moser (smoser)]

Note, that Michael Roth's comments are probably valid. I suspect that if leftyfb ever saw this working on powerVM host, then it was through the kvm_pr module. In my experience, that works generally pretty well (buit my testing has only ever been done powerNV host).

[Serge Hallyn (serge-hallyn)]

@leftyfb - what exactly is IBM asking to verify? Whether kvm works under powervm? Did smoser's info help?

[Ryan Harper (raharper)]

Here's an update.

The Xenial kernel doesn;t like the emulated POWER7 cpu that the command line being used generates by default.

processor : 0
cpu : POWER7 (raw), altivec supported
clock : 1000.000000MHz
revision : 2.3 (pvr 003f 0203)

timebase : 512000000
platform : pSeries
model : IBM pSeries (emulated by qemu)
machine : CHRP IBM pSeries (emulated by qemu)

We can boot a Wily image (kernel 4.2.0-35) just fine with the POWER7 cpu.

When booting Xenial's kernel with POWER7 cpu, it produces a stacktrace during module load:

[ 9.885165] Loaded X.509 cert 'Build time autogenerated kernel key: 6687eed33bf99302166296c3e5cafe31ef38ad41'
[ 9.886507] zswap: loaded using pool lzo/zbud
[ 9.916000] modprobe[74]: unhandled signal 4 at 00003fffb5a4d03c nip 00003fffb5a4d03c lr 00003fffb5a25e24 code 30001
[ 9.925819] modprobe[76]: unhandled signal 4 at 00003fff85b9d03c nip 00003fff85b9d03c lr 00003fff85b75e24 code 30001
[ 9.928401] Key type trusted registered
[ 9.930762] modprobe[79]: unhandled signal 4 at 00003fff7d05d03c nip 00003fff7d05d03c lr 00003fff7d035e24 code 30001
[ 9.933360] modprobe[80]: unhandled signal 4 at 00003fff8820d03c nip 00003fff8820d03c lr 00003fff881e5e24 code 30001
[ 9.936240] modprobe[83]: unhandled signal 4 at 00003fffb4fbd03c nip 00003fffb4fbd03c lr 00003fffb4f95e24 code 30001
[ 9.938873] modprobe[84]: unhandled signal 4 at 00003fff92d4d03c nip 00003fff92d4d03c lr 00003fff92d25e24 code 30001
[ 9.940335] Key type encrypted registered
[ 9.940461] AppArmor: AppArmor sha1 policy hashing enabled
[ 9.941005] ima: No TPM chip found, activating TPM-bypass!
[ 9.942985] evm: HMAC attrs: 0x1
[ 9.947081] hctosys: unable to open rtc device (rtc0)
[ 9.987867] Freeing unused kernel memory: 6144K (c000000000ea0000 - c0000000014a0000)
[ 9.991123] init[1]: unhandled signal 4 at 00003fff8edfd03c nip 00003fff8edfd03c lr 00003fff8edd5e24 code 30001
[ 9.994581] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004
[ 9.994581]
[ 9.994889] CPU: 0 PID: 1 Comm: init Not tainted 4.4.0-18-generic #34-Ubuntu
[ 9.995054] Call Trace:
[ 9.995216] [c00000001e4c3a50] [c000000000aed6fc] dump_stack+0xb0/0xf0 (unreliable)
[ 9.995336] [c00000001e4c3a90] [c000000000ae9930] panic+0x100/0x2c0
[ 9.995398] [c00000001e4c3b20] [c0000000000bd554] do_exit+0xc24/0xc30
[ 9.995443] [c00000001e4c3be0] [c0000000000bd644] do_group_exit+0x64/0x100
[ 9.995490] [c00000001e4c3c20] [c0000000000ceaac] get_signal+0x55c/0x7b0
[ 9.995534] [c00000001e4c3d10] [c000000000017424] do_signal+0x54/0x2b0
[ 9.995578] [c00000001e4c3e00] [c00000000001787c] do_notify_resume+0xbc/0xd0
[ 9.995677] [c00000001e4c3e30] [c000000000009838] ret_from_except_lite+0x64/0x68
[ 10.011069] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004
[ 10.011069]

When we use -enable-kvm, this bypasses the tcg POWER7 cpu, and uses the host cpu type (POWER8) which is why we can boot the Xenial kernel with KVM.

We need to open a linux task to help track down that issue; also if someone is testing Xenial on POWER7 hardware, t...

[Ryan Harper (raharper)]

I should also mention that the original command used will work if we specify which cpu type to use:

qemu-system-ppc64 -m 256 \
  -cpu POWER8 \
  -display none -nographic \
  -net nic -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp::2222-:22 \
  -machine pseries \
  -drive file=xenial-server-cloudimg-ppc64el-disk1.img,if=virtio \
  -drive file=seed.iso,if=virtio

[Michael Hohnbaum (hohnbaum)]

Build target was changed to P8 starting with Xenial. Previously it had been a P7 target. I would expect Xenial kernel to have problems booting on a P7 model. IMO this kernel bug is invalid.

[Ryan Harper (raharper)]

As noted, Xenial kernel is not supporting POWER7 cpu.

[Robin H. Johnson (robbat2)]

TL;DR: pass '-vga none' with -nographic, or redirect the screen somewhere!

I ended up digging into this after it was mentioned by smoser. The bug is invalid because of a bad assumption in the QEMU inputs. smoser's workaround of usb=off removes USB as a workaround.

The kernel, OpenFirmware, and QEMU are behaving correctly, but the original command-line hides the screen output.

If you have both a supported* keyboard AND screen, then OpenFirmware/SLOF sets OF stdout to be the screen.
This causes the OF output of 'No console specified using screen & keyboard'

If you are missing either a keyboard OR screen, then OF sets stdout to be hvterm (serial).
This causes the OF output of 'No console specified using hvterm'

https://git.qemu.org/?p=SLOF.git;a=blob;f=board-qemu/slof/OF.fs;h=4e04b840d5e6ff6c39191939e1a4f70f3d169d5d;hb=HEAD#l215

Which devices are available depends on the QEMU commandline...
Base QEMU defaults to '-vga std'.
QEMU '-machine pseries' defaults to enabling USB.

Re supported keyboards: The OpenFirmware/SLOF only seems to support USB keyboards at this time (virtio keyboard not supported).

[Christian Ehrhardt  (paelzer)]

Thanks for the Detail Robin!