Fails to boot cirros QEMU image with tuned running

[Martin Pitt]

A recent security update broke booting of some images, particularly CirrOS [3]:

$ wget https://download.cirros-cloud.net/0.3.5/cirros-0.3.5-i386-disk.img
$ qemu-system-x86_64 -enable-kvm -nographic cirros-0.3.5-i386-disk.img -snapshot
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]

And then nothing happens at all any more, other than QEMU using 100% CPU. This also affects version 0.4.0 and x86_64, so it's not particularly sensitive to guest changes.

I'm testing this with (nested) KVM inside an Ubuntu 18.04 LTS VM.

When going back to an older image that was built 5 days ago [4], it works:

# qemu-system-x86_64 -enable-kvm -nographic cirros-0.3.5-i386-disk.img -snapshot
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.2.0-80-virtual (buildd@komainu) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #116-Ubuntu SMP Mon Mar 23 17:48:17 UTC 2015 (Ubuntu 3.2.0-80.116-virtual 3.2.68)
[...]

This shows that the "ECX.svm" warning already happened before and seems to be unrelated.

The most recent security update of Qemu 2.11+dfsg-1ubuntu7.2 [1] and Linux 4.15.0-22.24 [2] are already on that previous image, so it seems this is some dependency update in between these stacks. I. e. this is *not* a regression in QEMU itself or Linux. I file it against qemu for now, but I will bisect which particular update was responsible.

[1] https://launchpad.net/ubuntu/+source/qemu/1:2.11+dfsg-1ubuntu7.2
[2] https://launchpad.net/ubuntu/+source/linux/4.15.0-22.24
[3] https://download.cirros-cloud.net/
[4] https://github.com/cockpit-project/cockpit/pull/9221

[Martin Pitt (pitti)]

Found it. This happens as soon as you install tuned, which auto-starts tuned.service. After "systemctl stop tuned", QEMU works again. Retitled accordingly, so this is not a regression.

[Scott Moser (smoser)]

@Martin,
Can you file this upstream with 'tuned' ? or is it not a problem with upstream?

[Martin Pitt (pitti)]

I tested this on Fedora 28, which also has tuned 2.9.0, but a slightly newer QEMU (2.11.1, as opposed to 2.11 on Ubuntu 18.04), and a newer kernel (4.16.10 instead of 4.15.0 in Ubuntu 18.04).

The invocation in the unit (/usr/bin/python -Es /usr/sbin/tuned -l -P) is exactly the same, so is the active profile ("virtual-guest"), profile_mode ("auto"), and the config file tuned-main.conf, which is bit by bit the same between the two.

However, this also affects Debian testing, which has Linux 4.16.5 and a newer QEMU (2.12). I filed a bug there as well now.

[Christian Ehrhardt  (paelzer)]

Hi Martin,
tuned is usually not installed, at least not on the new guest you are booting, so to be sure are we talking about this setup:

Host
  - KVM-Host
  - not so important, but what OS is this?
Guest1
  - KVM-Guest
  - runs an Ubuntu Cosmic Image
  - also KVM-Nested-Host
  - tuned runs here in virt-guest profile
Guest2
  - nested KVM-Guest
  - runs CirrOS image
  - Booting this guest hangs with the newer tuned

I assume tuned runs in the default config here.
So it spawns its service, has a dbus listener but none of the specials (like kernel commandline changes).
But if I not miss something the virt-guest profile should then by default only change vm.dirty_ratio and vm.swappiness - none of these should be super important.

I'm trying to recreate now with above assumptions, let me know if anything differs.

[Christian Ehrhardt  (paelzer)]

Setup as assumed above:
1. Host Bionic
2. Guest1 Cosmic
3. Guest2 Cosmic or CirrOS

Guest1 is at:
linux-image-virtual 4.15.0.20.23
libvirt-daemon-system 4.0.0-1ubuntu11
qemu-system-x86 1:2.11+dfsg-1ubuntu10

Test 1 - no tuned on Guest1, to know we are good in the general setup
 ok - libvirt/uvtool guest
 ok - CirrOS guest directly in qemu-system

Test 2 - install tuned in Guest1 - log at /var/log/tuned/tuned.log
 - I see base and cpu pluging initializing
 - re-applying systemctl
 - apply tuning of virtual-guest config
   I compared sysctl changes and found kernel.sched_min_granularity_ns and kernel.sched_wakeup_granularity_ns bumped up by ~4-5, but that should not break anything.
   Also I found the expected dirty/swappiness ratio - also should not break anything

Test 3 - run guests with tuned installed
 ok - libvirt/uvtool guest
 ok - CirrOS guest directly in qemu-system

Ok, I'm not able to reproduce yet, so I think my assumptions on your setup were not correct.
Let me know what is different please.

[Martin Pitt (pitti)]

Sorry, mixed up the project, I didn't actually test that yet with qemu.

[Martin Pitt (pitti)]

I re-tested this on current Ubuntu 22.04 and 23.10, and it seems to work again.