[UBUNTU 20.04] KVM: s390: pv: don't allow userspace to set the clock under PV - qemu part

[bugproxy]

[ Impact ]

 * suspend and resume (and other situations that cause vm
   state change) lead to situations where the hipervisor
   wants to change the clock.
   Now the s390x clock can be considered staged with
   offsets potentially added at each level of virtualization.
   In the secure execution mode on s390x the hipervisor
   is not allowed to touch this, leading to wrong timing.

 * The fix skips changing the tod (time of day -> clock)
   on a kvm_s390_tod_vm_state_change

[ Test Plan ]

 * 1. Start SE guest using libvirt.
   2. Pause the guest using "virsh suspend", wait for a few
      seconds, resume using "virsh resume".
   3. Run "time sleep 1" in the guest.
   4. The sleep will sleep much longer than one second.

[ Where problems could occur ]

 * This is gladly very isolated to s390x clock handling, so
   that (and only that) would be the place to look out for
   problems or regressions.

[ Other info ]

 * s390x secure execution needs HW feature my machine lacks,
   this will need IBM to verify the case once accepted to proposed.

--- original report ---

Description: KVM: s390: pv: don't allow userspace to set the clock under PV

Symptom: Timer issues and RCU stalls after suspending and resuming an IBM
               Secure Execution guest

Problem: KVM and QEMU try to set the guest's TOD clock after resume
               under PV, even though that is not permitted under SE. Hence,
               their view of the guest clock may deviate from the ultravisor's,
               possibly causing KVM to re-dispatch the guest too late on clock
               comparator interrupts.

Solution: Don't set the clock after resume under PV. Note that kernel and
               QEMU patches are required in lockstep, to avoid a warning message
               in QEMU.

Reproduction: 1. Start SE guest using libvirt.
               2. Pause the guest using "virsh suspend", wait for a few
                  seconds, resume using "virsh resume".
               3. Run "time sleep 1" in the guest.
               4. The sleep will sleep much longer than one second.

Component: qemu

Upstream-ID: 38621181ae3cbec62e3490fbc14f6ac01642d07a

Preventive: yes

Author: Nico Boehr <email address hidden>

Please note that fixing the described problem requires patches for the kernel as well as for QEMU.
This bug covers the qemu part, whereas the required kernel part is described and handled in the following related bug:

   Bug 200889 - [UBUNTU 20.04] KVM: s390: pv: don't allow userspace to set the clock under PV - kernel part

Both parts, the kernel and the qemu patches should be applied / released at the same time to avoid problems resulting in the following warning message for customers:
   'warning: Unable to set KVM guest TOD clock: Operation not supported'

[Frank Heimes (fheimes)]

Meanwhile 7.2 landed in lunar-proposed:
qemu | 1:7.2+dfsg-3ubuntu1 | lunar-proposed
hence updating ticket status to Fix Committed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:7.2+dfsg-4ubuntu1

---------------
qemu (1:7.2+dfsg-4ubuntu1) lunar; urgency=medium

  * Merge with Debian unstable (LP: #1993438), among many other fixes
    this resolvs these bugs:
    (LP: #1957924) - support for querying stats,
    (LP: #1853307) - Enhanced Interpretation for PCI Functions (s390x)
    (LP: #1959966) - guest dump encryption with customer keys (s390x)
    (LP: #1999885) - pv: don't allow userspace to set the clock under PV
    (LP: #1957924) - add filtering of statistics by target vCPU
    remaining changes:
    - qemu-kvm to systemd unit
      - d/qemu-kvm-init: script for QEMU KVM preparation modules, ksm,
        hugepages and architecture specifics
      - d/qemu-system-common.qemu-kvm.service: systemd unit to call
        qemu-kvm-init
      - d/qemu-system-common.install: install helper script
      - d/qemu-system-common.qemu-kvm.default: defaults for
        /etc/default/qemu-kvm
      - d/rules: call dh_installinit and dh_installsystemd for qemu-kvm
    - Distribution specific machine type
      (LP: 1304107 1621042 1776189 1761372 1761372 1776189)
      - d/p/ubuntu/define-ubuntu-machine-types.patch: define distro machine
        types containing release versioned machine attributes
      - d/qemu-system-x86.NEWS Info on fixed machine type defintions
        for host-phys-bits=true
      - Add an info about -hpb machine type in debian/qemu-system-x86.NEWS
      - ubuntu-q35 alias added to auto-select the most recent q35 ubuntu type
    - Enable nesting by default
      - d/p/ubuntu/enable-svm-by-default.patch: Enable nested svm by default
        in qemu64 on amd
        [ No more strictly needed, but required for backward compatibility ]
    - tolerate ipxe size change on migrations to >=18.04 (LP: 1713490)
      - d/p/ubuntu/pre-bionic-256k-ipxe-efi-roms.patch: old machine types
        reference 256k path
      - d/control-in: depend on ipxe-qemu-256k-compat-efi-roms to be able to
        handle incoming migrations from former releases.
    - d/qemu-system-x86.README.Debian: add info about updated nesting changes
    - Ease the use of module retention on upgrades (LP 1913421)
      - debian/qemu-block-extra.postinst: enable mount unit on install/upgrade
    - d/control-in: switch qemu-system-x86-xen to qemu-system-xen as this
      landed in Debian but under a different name.
    - Remaining GCC-12 FTBFS (LP 1988710 + LP 1921664)
      + d/p/u/qboot-Disable-LTO-for-ELF-binary-build-step.patch:
        fix qboot FTBFS with LTO
  * Dropped Changes [now part of upstream v7.2.0]
    - d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
      error 'migration was active, but no RAM info was set' (LP 1994002)
    - d/p/u/ebpf-replace-deprecated-bpf_program__set_socket_filt.patch:
      Fix FTBFS with libbpf 1.0.1-2.
      + Header updates that were added as part of the libbpf fixes
        but not mentioned in changelog
    - d/p/u/lp-1981339-*: fix s390x system emulation (LP 1981339)
    - Fix I/O stalls when using NVMe storage (LP 1970737).
      + d/p/lp1970737-linux-aio-*.patch: Fix unbalanced plugged counter
        in laio_io_unplug.
    - SECURITY UPDATE...

[Frank Heimes (fheimes)]

Hi Nico, do you know if backports exist to get 38621181ae3cbec62e3490fbc14f6ac01642d07a "KVM: s390: pv: don't allow userspace to set the clock under PV" applied to kinetic (qemu 7.0), Jammy (qemu 6.2) and focal (qemu 4.2) - or does the patch/commit apply cleanly?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-03-07 05:56 EDT-------
Hi Frank,

after having a quick glance, for kinetic and jammy the patch should apply cleanly.

For focal, the attached patch should apply cleanly.

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2023-03-07 05:56 EDT-------

[Christian Ehrhardt  (paelzer)]

Thanks Nico,
we just entered an SRU process with three other fixes, that needs to complete before we can prep this one.

[Christian Ehrhardt  (paelzer)]

FYI - I have not forgotten, but the SRU cases I mentioned (a set of security fixes and bug 1921664, bug 2009048 and bug 1994002) are still ongoing for focal and jammy.

Only in Kinetic the queue is cleared already.

[Christian Ehrhardt  (paelzer)]

As it is unlocked, this was started for Kinetic:
- PPA: https://launchpad.net/~paelzer/+archive/ubuntu/lp1999885-tod-in-pv-kinetic
- MR: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/439453

@IBM, you'll need to help us with testing on this due to HW constraints.
Eventually you'll have to test it once accepted to -proposed for sure.
If that is easy, I'd appreciate checking the PPA as well, if not at least later on once accepted by the SRU team please.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-03-24 05:03 EDT-------
(In reply to comment #13)
> As it is unlocked, this was started for Kinetic:
> - PPA:
> https://launchpad.net/~paelzer/+archive/ubuntu/lp1999885-tod-in-pv-kinetic
> - MR:
> https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/
> 439453
>
> @IBM, you'll need to help us with testing on this due to HW constraints.
> Eventually you'll have to test it once accepted to -proposed for sure.
> If that is easy, I'd appreciate checking the PPA as well, if not at least
> later on once accepted by the SRU team please.

I tried the PPA and was able to verify it fixes the issue. Thanks!

[Christian Ehrhardt  (paelzer)]

FYI: I abandoned the old PPAs and MRs in favor of new ones combining the planned uploads.

PPA: https://launchpad.net/~paelzer/+archive/ubuntu/lp-2011832-emulation-issues-lp-1999885-tod-in-pv-sruprep
MRs:
- F: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/439681
- J: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/439682
- K: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/439683

[Christian Ehrhardt  (paelzer)]

The former SRUs cleared, now uploaded to -unapproved

[Steve Langasek (vorlon)]

Hello bugproxy, or anyone else affected,

Accepted qemu into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:7.0+dfsg-7ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello bugproxy, or anyone else affected,

Accepted qemu into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:6.2+dfsg-2ubuntu6.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello bugproxy, or anyone else affected,

Accepted qemu into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:4.2-3ubuntu6.26 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:7.0+dfsg-7ubuntu2.3) for kinetic have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-image/2.2+22.04ubuntu3 (arm64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/kinetic/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:6.2+dfsg-2ubuntu6.8) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.140ubuntu13.1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-04-19 04:10 EDT-------
Verification Result Ubuntu 22.04:
PASS

Verification Environment:
ii qemu-system-s390x 1:6.2+dfsg-2ubuntu6.8 s390x QEMU full system emulation binaries (s390x)

Linux a83lp41.lnxne.boe 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:22:11 UTC 2023 s390x s390x s390x GNU/Linux

Verification Steps:
0. Start guest with SE enabled and a monitor console -> PASS
1. Verify guest timer works as expected -> PASS
~ # time sleep 1
real 0m 1.00s
user 0m 0.00s
sys 0m 0.00s
2. Stop/continue guest via QEMU monitor -> PASS, no warning message in monitor
3. Verify guest timer works as expected after stop/continue -> PASS
~ # time sleep 1
real 0m 1.00s
user 0m 0.00s
sys 0m 0.00s

Verification Result Ubuntu 20.04:
PASS

Verification Environment:
ii qemu-system-s390x 1:4.2-3ubuntu6.26 s390x QEMU full system emulation binaries (s390x)

Linux a83lp41.lnxne.boe 5.4.0-147-generic #164-Ubuntu SMP Tue Mar 21 14:22:27 UTC 2023 s390x s390x s390x GNU/Linux

Verification Steps:
0. Start guest with SE enabled and a monitor console -> PASS
1. Verify guest timer works as expected -> PASS
~ # time sleep 1
real 0m 1.00s
user 0m 0.00s
sys 0m 0.00s
2. Stop/continue guest via QEMU monitor -> PASS, no warning message in monitor
3. Verify guest timer works as expected after stop/continue -> PASS
~ # time sleep 1
real 0m 1.00s
user 0m 0.00s
sys 0m 0.00s

22.10 is still pending due to some internal issues. I hope to finish by the end of this week.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-04-19 05:40 EDT-------
OK, good news, internal issues were resolved more quickly than expected. So here's the

Verification Result Ubuntu 22.10:
PASS

Verification Environment:
ii qemu-system-s390x 1:7.0+dfsg-7ubuntu2.3 s390x QEMU full system emulation binaries (s390x)

Linux a83lp41.lnxne.boe 5.19.0-40-generic #41-Ubuntu SMP Thu Mar 23 21:39:25 UTC 2023 s390x s390x s390x GNU/Linux

Verification Steps:
0. Start guest with SE enabled and a monitor console -> PASS
1. Verify guest timer works as expected -> PASS
~ # time sleep 1
real 0m 1.00s
user 0m 0.00s
sys 0m 0.00s
2. Stop/continue guest via QEMU monitor -> PASS, no warning message in monitor
3. Verify guest timer works as expected after stop/continue -> PASS
~ # time sleep 1
real 0m 1.00s
user 0m 0.00s
sys 0m 0.00s

[Frank Heimes (fheimes)]

Many thx for the validations, Nico!

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:4.2-3ubuntu6.26

---------------
qemu (1:4.2-3ubuntu6.26) focal; urgency=medium

  * d/p/u/lp-1999885-s390x-tod-kvm-don-t-save-restore-the-TOD-in-PV-guest.patch:
    avoid timer issues in s390x secure execution guests (LP: #1999885)

 -- Christian Ehrhardt <email address hidden> Thu, 23 Mar 2023 08:18:28 +0100

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:6.2+dfsg-2ubuntu6.8

---------------
qemu (1:6.2+dfsg-2ubuntu6.8) jammy; urgency=medium

  * d/p/u/lp-1999885-s390x-tod-kvm-don-t-save-restore-the-TOD-in-PV-guest.patch:
    avoid timer issues in s390x secure execution guests (LP: #1999885)
  * d/p/u/lp-2011832-*: fix emulation issues in mips and powerpc (LP: #2011832)

 -- Christian Ehrhardt <email address hidden> Thu, 23 Mar 2023 08:18:28 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:7.0+dfsg-7ubuntu2.3

---------------
qemu (1:7.0+dfsg-7ubuntu2.3) kinetic; urgency=medium

  * d/p/u/lp-1999885-s390x-tod-kvm-don-t-save-restore-the-TOD-in-PV-guest.patch:
    avoid timer issues in s390x secure execution guests (LP: #1999885)
  * d/p/u/lp-2011832-*: fix emulation issues in mips (LP: #2011832)

 -- Christian Ehrhardt <email address hidden> Thu, 23 Mar 2023 08:18:28 +0100