virsh start, after virsh managed save hangs and vm goes to paused state with qemu version v2.12.0-813-g5a5c383b13-dirty on powerpc

[Satheesh Rajendran]

Host Env:
IBM Power8 with Fedora28 base with compiled upstream kernel, qemu, libvirt.

Host Kernel: 4.17.0-rc5-00069-g3acf4e395260

qemu-kvm(5a5c383b1373aeb6c87a0d6060f6c3dc7c53082b): v2.12.0-813-g5a5c383b13-dirty

libvirt(4804a4db33a37f828d033733bc47f6eff5d262c3):

Guest Kernel: 4.17.0-rc7

Steps to recreate:
Define a guest attached with above setup and start.
# virsh start avocado-vt-vm1

guest console;...
# uname -r
4.17.0-rc7
[root@atest-guest ~]# lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 3
On-line CPU(s) list: 0-2
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 3
NUMA node(s): 1
Model: 2.1 (pvr 004b 0201)
Model name: POWER8 (architected), altivec supported
Hypervisor vendor: KVM
Virtualization type: para
L1d cache: 64K
L1i cache: 32K
NUMA node0 CPU(s): 0-2

# virsh managedsave avocado-vt-vm1

Domain avocado-vt-vm1 state saved by libvirt

# virsh list
 Id Name State
----------------------------------------------------

# virsh start avocado-vt-vm1 ----Hangs forever and vm state goes to paused.

# virsh list
 Id Name State
----------------------------------------------------
 87 avocado-vt-vm1 paused

P:S:- with same above setup, just changing the qemu-kvm comes bydefault with F28 works fine.

/usr/bin/qemu-kvm --version
QEMU emulator version 2.11.1(qemu-2.11.1-2.fc28)

Summary: with above other setup.
machine type pseries-2.12 and qemu-2.11.1-2.fc28 -Works fine.

machine type pseries-2.12/pseries-2.13 and qemu 5a5c383b1373aeb6c87a0d6060f6c3dc7c53082b - Does not work.

[Satheesh Rajendran (sathnaga)]

To recover from the failed state it requires below steps to be run.

# virsh destroy avocado-vt-vm1
Domain avocado-vt-vm1 destroyed

# virsh undefine --managed-save avocado-vt-vm1
Domain avocado-vt-vm1 has been undefined

[Peter Xu (peterx)]

On Mon, May 28, 2018 at 09:12:21AM -0000, Satheesh Rajendran wrote:
> Public bug reported:
>
> Host Env:
> IBM Power8 with Fedora28 base with compiled upstream kernel, qemu, libvirt.
>
> Host Kernel: 4.17.0-rc5-00069-g3acf4e395260
>
> qemu-kvm(5a5c383b1373aeb6c87a0d6060f6c3dc7c53082b):
> v2.12.0-813-g5a5c383b13-dirty
>
> libvirt(4804a4db33a37f828d033733bc47f6eff5d262c3):
>
> Guest Kernel: 4.17.0-rc7
>
> Steps to recreate:
> Define a guest attached with above setup and start.
> # virsh start avocado-vt-vm1
>
> guest console;...
> # uname -r
> 4.17.0-rc7
> [root@atest-guest ~]# lscpu
> Architecture: ppc64le
> Byte Order: Little Endian
> CPU(s): 3
> On-line CPU(s) list: 0-2
> Thread(s) per core: 1
> Core(s) per socket: 1
> Socket(s): 3
> NUMA node(s): 1
> Model: 2.1 (pvr 004b 0201)
> Model name: POWER8 (architected), altivec supported
> Hypervisor vendor: KVM
> Virtualization type: para
> L1d cache: 64K
> L1i cache: 32K
> NUMA node0 CPU(s): 0-2
>
>
> # virsh managedsave avocado-vt-vm1
>
> Domain avocado-vt-vm1 state saved by libvirt
>
> # virsh list
> Id Name State
> ----------------------------------------------------
>
> # virsh start avocado-vt-vm1 ----Hangs forever and vm state goes to
> paused.

Libvirt is using fd migration, right? If so, I suspect this is the
same problem with the iotest failure, and the fix should be in a pull
request:

  Message-Id: <email address hidden>
  Subject: [Qemu-devel] [PULL 1/2] migration: fix exec/fd migrations

Regards,

--
Peter Xu

[Satheesh Rajendran (sathnaga)]

with above patch compiled on top of latest upstream fails with below error:

# virsh managedsave avocado-vt-vm1
error: Failed to save domain avocado-vt-vm1 state
error: internal error: guest unexpectedly quit

rest of the behaviour same..
# virsh start avocado-vt-vm1 ----gets hung
---crtl+c --> to comeback to prompt
#

# virsh destroy avocado-vt-vm1
Domain avocado-vt-vm1 destroyed

# virsh undefine --managed-save avocado-vt-vm1
Domain avocado-vt-vm1 has been undefined

[Satheesh Rajendran (sathnaga)]

followed by further attempts saves the domains as reported but issue still same.

#virsh managedsave avocado-vt-vm1

Domain avocado-vt-vm1 state saved by libvirt
# virsh start avocado-vt-vm1 ----hung

# virsh list --all
 Id Name State
----------------------------------------------------
 98 avocado-vt-vm1 paused

I tried restarting libvirt, after which guest goes to shutoff state, with reason as crash in the qemu log

# service libvirtd restart
Redirecting to /bin/systemctl restart libvirtd.service

# virsh list --all
 Id Name State
----------------------------------------------------
 - avocado-vt-vm1 shut off

2018-05-28 12:59:46.748+0000: starting up libvirt version: 4.4.0, package: 1.fc28 (Unknown, 2018-05-28-03:15:39, 9.40.192.86), qemu version: 2.12.50v2.12.0-813-g5a5c383b13-dirty, kernel: 4.17.0-rc5-00069-g3acf4e395260, hostname: 9.40.192.86
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/share/avocado-plugins-vt/bin/qemu -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-97-avocado-vt-vm1/master-key.aes -machine pseries-2.13,accel=kvm,usb=off,dump-guest-core=off -m 1024 -realtime mlock=off -smp 2,maxcpus=4,sockets=4,cores=1,threads=1 -uuid ba3012d5-3244-47d9-bedc-0b60821f7cd1 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-97-avocado-vt-vm1/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -kernel /home/kvmci/linux/vmlinux -append 'root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug' -device qemu-xhci,id=usb,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/avocado/data/avocado-vt/images/jeos-27-ppc64le.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:3d:3e:3f,bus=pci.0,addr=0x1 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/avocado-vt-vm1-guest.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -sandbox off -msg timestamp=on
2018-05-28T12:59:46.826738Z qemu: -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label charserial0)
2018-05-28 13:00:52.948+0000: shutting down, reason=saved

2018-05-28T13:00:52.950802Z qemu: terminating on signal 15 from pid 41456 (/usr/sbin/libvirtd)
2018-05-28 13:01:00.467+0000: starting up libvirt version: 4.4.0, package: 1.fc28 (Unknown, 2018-05-28-03:15:39, 9.40.192.86), qemu version: 2.12.50v2.12.0-813-g5a5c383b13-dirty, kernel: 4.17.0-rc5-00069-g3acf4e395260, hostname: 9....

[Fabiano Rosas (farosas)]

I was able to reproduce this with:

qemu - v2.12.0-813-g5a5c383
host/guest kernel - 4.11
libvirt - 3.9.0

It bisects to:

  36c2f8b migration: Delay start of migration main routines

However, the issue did *not* reproduce with:

qemu - v2.12.0-865-ge609fa7
host/guest kernel - 4.11
libvirt - 3.9.0

As Peter suggested, it is fixed by:

  0efc914 migration: fix exec/fd migrations

So perhaps there is still something on libvirt side? I'll try again with a more
recent one.

[Fabiano Rosas (farosas)]

Could not reproduce with:

qemu - v2.12.0-865-ge609fa7
host/guest kernel - 4.11
libvirt - 4.4.0

and

qemu - v2.12.0-865-ge609fa7
host kernel - v4.17-rc7-22-g3d661e2
guest kernel - 4.11
libvirt - 4.4.0

So I'd say that this is fixed by:

https://git.qemu.org/?p=qemu.git;a=commitdiff;h=0efc914

[Satheesh Rajendran (sathnaga)]

Yes, tested again with below levels and not issue is not reproducible.

Issue is fixed!

qemu: 2.12.50 (v2.12.0-949-g392fba9f58-dirty)
host/guest kernel: 4.17.0-rc7-00045-g0512e0134582

libvirt:
Compiled against library: libvirt 4.4.0
Using library: libvirt 4.4.0
Using API: QEMU 4.4.0
Running hypervisor: QEMU 2.12.50

#virsh managedsave avocado-vt-vm1

Domain avocado-vt-vm1 state saved by libvirt

# virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started

Guest console.
# uname -r
4.17.0-rc7-00045-g0512e0134582

This bug can be closed.

[Satheesh Rajendran (sathnaga)]

libvirt compiled against 105bcdde76bc8c64f2d9aca9db684186a5e96e63