[19.10][qemu] virsh dompmwakeup fails to wake VM from dompmsuspend state (kvm)

[bugproxy]

== Comment: #0 - Balamuruhan S <email address hidden> - 2018-03-26 08:32:44 ==
---Problem Description---

virsh dompmwakeup fails to wake VM from dompmsuspend state, VM state remains running instead of pmsuspended even in suspended state. Guest kernel enters to suspend state and qemu/libvirt are not aware of it.

# tail -f /var/log/syslog
Mar 26 08:22:44 ubuntu systemd[1]: Started User Manager for UID 0.
Mar 26 08:23:45 ubuntu systemd-timesyncd[1213]: Network configuration changed, trying to establish connection.
Mar 26 08:23:45 ubuntu kernel: [ 1802.926114] PM: suspend entry (s2idle)

Machine Type = Boston

---Steps to Reproduce---
 1. Start a VM and ensure it reaches login prompt
2. Ensure qemu-guest-agent is available and running in the VM
3. Suspend the VM to mem using virsh dompmsuspend <domain> mem --duration 0
4. VM enters suspended state, try to wake up the VM using virsh dompmwakeup <domain>
5. Libvirt returns that VM got wokeup but ideally it doesn't

# virsh list --all
 Id Name State
----------------------------------------------------
 2 virt-tests-vm1 running

# virsh dompmsuspend virt-tests-vm1 mem
Domain virt-tests-vm1 successfully suspended

# echo $?
0

After dompmsuspend,
# virsh list --all
 Id Name State
----------------------------------------------------
 14 virt-tests-vm1 running
 - nrs shut off

# virsh domstate virt-tests-vm1 --reason
running (booted)

---uname output---

Host Kernel
# uname -a
Linux ltc-boston17 4.15.0-12-generic #13 SMP Thu Mar 22 14:16:58 CDT 2018 ppc64le ppc64le ppc64le GNU/Linux

Guest Kernel:
# uname -a
Linux ubuntu 4.15.0-12-generic #13-Ubuntu SMP Wed Mar 7 21:37:03 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

---Debugger---
A debugger is not configured

Contact Information = Balamuruhan S / <email address hidden>

Userspace tool common name: qemu

Userspace rpm:

Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu5 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu5 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.11+dfsg-1ubuntu5 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu5 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.11+dfsg-1ubuntu5 ppc64el QEMU utilities

Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 4.0.0-1ubuntu7 ppc64el programs for the libvirt library
ii libvirt-clients 4.0.0-1ubuntu7 ppc64el Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu7 ppc64el Virtualization daemon
ii libvirt-daemon-driver-storage-rbd 4.0.0-1ubuntu7 ppc64el Virtualization daemon RBD storage driver
ii libvirt-daemon-system 4.0.0-1ubuntu7 ppc64el Libvirt daemon configuration files
ii libvirt-dev:ppc64el 4.0.0-1ubuntu7 ppc64el development files for the libvirt library
ii libvirt-glib-1.0-0:ppc64el 1.0.0-1 ppc64el libvirt GLib and GObject mapping library
ii libvirt0:ppc64el 4.0.0-1ubuntu7 ppc64el library for interfacing with different virtualization systems
ii python-libvirt 4.0.0-1 ppc64el libvirt Python bindings

The userspace tool has the following bit modes: ppc64le

Userspace tool obtained from project website: na

*Additional Instructions for Balamuruhan S / <email address hidden>:
-Post a private note with access information to the machine that the bug is occuring on.
-Attach ltrace and strace of userspace application.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-03-28 06:24 EDT-------
Note from Daniel, who is looking into this bug.

-----

Quick recap: this problem happens because we don't have the wake-up from software suspend implemented in the pseries machine. In fact, at least last time I checked, only the x86 machines implement it.

I've send patches to the QEMU ML about 3 months ago to implement a new API (well, technically a change in an existing API) to detect whether the guest has this capability or not. The idea is to use it in Libvirt to prevent pm-suspend and similar commands to execute at all if the guest can't proper wake up. The patches were left behind along the way, and now people are working towards the RC releases of 2.12 (i.e just critical bug fixes).

I'll re-re-send that patches aiming for 2.13 and see if we can finally get them upstream.

[Frank Heimes (fheimes)]

Please let us know as soon as the needed patches got upstream accepted.
Changing this ticket incomplete for now.

[Christian Ehrhardt  (paelzer)]

Since 18.10 will likely be 2.12 based it would be nice to get the post 2.12 upstream commit IDs as soon as they are existing as we won't pick it up automagically :-)

[Christian Ehrhardt  (paelzer)]

Ping for my question of post 2.12 commit IDs - see comment #3

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-06-22 05:54 EDT-------
Hi,

> Ping for my question of post 2.12 commit IDs - see comment #3

We don't have it yet, the review process is still ongoing. I'll post it here as soon as it is pushed upstream.

[Manoj Iyer (manjo)]

IBM, 18.10 feature freeze is on the 23rd of Aug, any news on the upstream status of this patch?

[Manoj Iyer (manjo)]

Have these patches made its way upstream? if so could you please post links here?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-12-04 18:02 EDT-------
The current posting is here:
https://lists.gnu.org/archive/html/qemu-devel/2018-11/msg01774.html

Daniel said there should be a v11 soon, but right now it's marked for qemu 4.0.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-12-20 18:32 EDT-------
The patch series was pushed to QEMU upstream. The commits id are:

46ea94ca9cfcd56c27efafd2ff32281360e0267f
f8a577773815cc8ff7154bdb114a6f8ee4165edc
fb0641121072d9d612773370688a6887bb78db08

This series alone will *not* fix the problem reported here. I've made improvements in the system_wakeup QEMU call (which is invoked by dompmwakeup), which now should report a clearer error in the case of a pseries guest:

(qemu) system_wakeup
wake-up from suspend is not supported by this guest
(qemu)
To fully fix the problem in Libvirt, changes must be made to dompmsuspend so the guest will not go into suspend state if the underlying arch does not have support for it. This is now possible with these QEMU changes I've made. I can do the Libvirt side changes myself after I'm done with another Libvirt/QEMU bug I'm looking into.

[Christian Ehrhardt  (paelzer)]

I've read through the changes and think they are fine to be added to hour qemu 3.1 currently being prepared for Ubuntu 19.04

If you later on have the libvirt changes to fully evaluate the newly added result on wakeup and they are backportable in a non broken way to our libvirt in the same release as well we can add that as well.

So let me know once you have upstreamed the libvirt changes, I'll add an incomplete task for that and add the qemu portion to my current items.

[Andrew Cloke (andrew-cloke)]

Marking as Incomplete while waiting for changes to be upstreamed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:3.1+dfsg-2ubuntu1

---------------
qemu (1:3.1+dfsg-2ubuntu1) disco; urgency=medium

  * Merge with Debian testing, Among many other things this fixes LP Bugs:
    LP: #1806104 - fix misleading page size error on ppc64el
    LP: #1782205 - SnowRidge enabled new ISAs
    LP: #1786956 - upgrade to qemu >= 3.0
    LP: #1809083 - Backward migration to Xenial on ppc64el
    LP: #1803315 - s390x Huge page enablement
    LP: #1657409 - enable virglrenderer
    Remaining Changes:
    - qemu-kvm to systemd unit
      - d/qemu-kvm-init: script for QEMU KVM preparation modules, ksm,
        hugepages and architecture specifics
      - d/qemu-kvm.service: systemd unit to call qemu-kvm-init
      - d/qemu-system-common.install: install systemd unit and helper script
      - d/qemu-system-common.maintscript: clean old sysv and upstart scripts
      - d/qemu-system-common.qemu-kvm.default: defaults for
        /etc/default/qemu-kvm
      - d/rules: install /etc/default/qemu-kvm
    - Enable nesting by default
      - d/qemu-system-x86.modprobe: set nested=1 module option on intel.
        (is default on amd)
      - d/qemu-system-x86.postinst: re-load kvm_intel.ko if it was loaded
        without nested=1
      - d/p/ubuntu/expose-vmx_qemu64cpu.patch: expose nested kvm by default
        in qemu64 cpu type.
      - d/p/ubuntu/enable-svm-by-default.patch: Enable nested svm by default
        in qemu64 on amd
      - d/qemu-system-x86.README.Debian: document intention of nested being
        default is comfort, not full support
    - Distribution specific machine type (LP: 1304107 1621042 1776189 1761372)
      - d/p/ubuntu/define-ubuntu-machine-types.patch: define distro machine
        types
      - d/qemu-system-x86.NEWS Info on fixed machine type defintions
        for host-phys-bits=true (LP: 1776189)
      - add an info about -hpb machine type in debian/qemu-system-x86.NEWS
      - d/p/ubuntu/lp-1761372-*: provide pseries-bionic-2.11-sxxm type as
        convenience with all meltdown/spectre workarounds enabled by default.
        (LP: 1761372).
    - improved dependencies
      - Make qemu-system-common depend on qemu-block-extra
      - Make qemu-utils depend on qemu-block-extra
      - let qemu-utils recommend sharutils
    - s390x support
      - Create qemu-system-s390x package
      - Enable numa support for s390x
    - arch aware kvm wrappers
    - d/control: update VCS links (updated to match latest Ubuntu)
    - qemu-guest-agent: freeze-hook fixes (LP: 1484990)
      - d/qemu-guest-agent.install: provide /etc/qemu/fsfreeze-hook
      - d/qemu-guest-agent.dirs: provide /etc/qemu/fsfreeze-hook.d
    - d/control-in: enable RDMA support in qemu (LP: 1692476)
        - enable RDMA config option
        - add libibumad-dev build-dep
    - tolerate ipxe size change on migrations to >=18.04 (LP: 1713490)
      - d/p/ubuntu/pre-bionic-256k-ipxe-efi-roms.patch: old machine types
        reference 256k path
      - d/control-in: depend on ipxe-qemu-256k-compat-efi-roms to be able to
        handle incoming migrations from former releases.
    - d/control-in: Disable capstone disassembler library support (universe...

[Christian Ehrhardt  (paelzer)]

We have the qemu bits in Ubuntu 19.04 now, but I have not heard of any libvirt bits yet.
Furthermore an SRU makes no sense yet as the qemu changes alone won't fix the overall case.
Therefore I'm unassigning myself until there are clear next steps to take.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-01-31 16:19 EDT-------
Yeah, we're still missing the Libvirt part of this feature/fix that will use the new QEMU APIs to enhance dompmsuspend behavior.

I'll do a best effort to see if I can get patches in the Libvirt mailing list before Feb 21st, which I believe it's the feature freeze for 19.04. Not sure if we can get them accepted upstream by that date though.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-04-01 17:38 EDT-------
Hi,

I just posted the Libvirt side of this feature to the mailing list:

https://www.redhat.com/archives/libvir-list/2019-April/msg00104.html

which reads "[libvirt] [PATCH 0/3] introducing QEMU_CAPS_QUERY_CURRENT_MACHINE and QEMU_CAPS_WAKEUP_SUSPEND_SUPPORT"

I'll follow up with the reviews and keep this bug posted with any updates.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-04-25 09:16 EDT-------
Hi,

The Libvirt patches were pushed upstream. This bug is now fixed in both QEMU (4.0+) and Libvirt (probably will be available in the next version, 5.3.0).

The Libvirt patches are:

https://github.com/libvirt/libvirt/commit/dca1b1d007c7efda675fce30ae63d701a09acd43
https://github.com/libvirt/libvirt/commit/70a4e3ee075ca12723fadaece13b62b767e40547
https://github.com/libvirt/libvirt/commit/cc1d1dbbd5fa18876a5ca8ac99a991b32ad49409

Thanks,

Daniel

[Andrew Cloke (andrew-cloke)]

Status update: qemu patches now released. Libvirt patches are targeted for release in 19.10.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 5.4.0-0ubuntu1

---------------
libvirt (5.4.0-0ubuntu1) eoan; urgency=medium

  * Merged with Debian git 5.3.0-1~1.gbp7b1637 and upstreams 5.4 release
    Among many other new features and fixes this includes fixes for:
    LP: #1759509 - virsh dompmwakeup fails to wake VM from dompmsuspend state
    Remaining changes:
    - Disable libssh2 support (universe dependency)
    - Disable firewalld support (universe dependency)
    - Set qemu-group to kvm (for compat with older ubuntu)
    - Additional apport package-hook
    - Autostart default bridged network (As upstream does, but not Debian).
      In addition to just enabling it our solution provides:
      + do not autostart if subnet is already taken (e.g. in guests).
      + iterate some alternative subnets before giving up
    - d/p/ubuntu/Allow-libvirt-group-to-access-the-socket.patch: This is
      the group based access to libvirt functions as it was used in Ubuntu
      for quite long.
      + d/p/ubuntu/daemon-augeas-fix-expected.patch fix some related tests
        due to the group access change.
      + d/libvirt-daemon-system.postinst: add users in sudo to the libvirt
        group.
    - ubuntu/parallel-shutdown.patch: set parallel shutdown by default.
    - Update Vcs-Git and Vcs-Browser fields to point to launchpad
    - Xen related
      - d/p/ubuntu/ubuntu-libxl-qemu-path.patch: this change was split. The
        section that adapts the path of the emulator to the Debian/Ubuntu
        packaging is kept.
      - d/p/ubuntu/ubuntu-libxl-Fix-up-VRAM-to-minimum-requirements.patch: auto
        set VRAM to minimum requirements
      - d/p/ubuntu/xen-default-uri.patch: set default URI on xen hosts
      - Add libxl log directory
      - libvirt-uri.sh: Automatically switch default libvirt URI for users on
        Xen dom0 via user profile (was missing on changelogs before)
    - d/p/ubuntu/apibuild-skip-libvirt-common.h: drop libvirt-common.h from
      included_files to avoid build failures due to duplicate definitions.
    - Update README.Debian with Ubuntu changes
    - Enable some additional features on ppc64el and s390x (for arch parity)
      + systemtap, zfs, numa and numad on s390x.
      + systemtap on ppc64el.
    - d/t/control, d/t/smoke-qemu-session: fixup smoke-qemu-session by making
      vmlinuz available and accessible (Debian bug 848314)
    - d/t/control, d/t/smoke-lxc: fix up lxc smoke test isolation
    - d/p/ubuntu/ubuntu_machine_type.patch: accept ubuntu types as pci440fx
    - Further upstreamed apparmor Delta, especially any new one
      Our former delta is split into logical pieces and is either Ubuntu only
      or is part of a continuous upstreaming effort.
      Listing related remaining changes in debian/patches/ubuntu-aa/:
      + 0001-apparmor-Allow-pygrub-to-run-on-Debian-Ubuntu.patch: apparmor:
        Allow pygrub to run on Debian/Ubuntu
      + 0003-apparmor-libvirt-qemu-Allow-read-access-to-overcommi.patch:
        apparmor, libvirt-qemu: Allow read access to overcommit_memory
      + 0007-apparmor-libvirt-qemu-Allow-owner-read-access-to-PRO.patch:
        apparmor, libvirt-qemu: Allow owner read acc...