| 2019-05-23 18:25:24 |
Dan Streetman |
bug |
|
|
added bug |
| 2019-05-23 18:51:11 |
Dan Streetman |
description |
[impact]
libvirt caches the 'nested vmx' capability of the host and does not update that even if the host's capability to handle nested vmx changes. Having this domcapability missing means no guests are able to start any nested, kvm-accelerated, guests. Additionally, since openstack live migration requires matching cpu features, this makes migrating guests that do have vmx enabled impossible to hosts where libvirt thinks nested vmx is disabled.
Once the kernel module (kvm_intel) is reloaded with 'nested' enabled, libvirt does not update its domcapabilities cache, even over a libvirtd restart, or even over an entire system reboot. Only certain conditions cause libvirt to update its capabilities cache (possibly libvirt upgrade, or qemu upgrade, or kernel upgrade...I haven't verified any of those yet)
libvirt creates caches for its domcapabilities at /var/cache/libvirt/qemu/capabilities/.
removing the cache xml files there and restarting libvirtd will cause the caches to be recreated with the correct current values.
[test case]
check the kvm_intel module nested parameter:
$ cat /sys/module/kvm_intel/parameters/nested
Y
it can be Y or N. make sure libvirt agrees with the current setting:
$ virsh domcapabilities | grep vmx
<feature policy='require' name='vmx'/>
if 'nested' is Y, domcapabilities should include a vmx feature line; if 'nested' is N, it should have no output (i.e. vmx not supported in guests).
Then, change the kernel nested setting, and re-check domcapabilities. Restarting libvirtd doesn't update the cache, and even rebooting the entire system doesn't update the cache.
$ virsh domcapabilities | grep vmx
$ cat /sys/module/kvm_intel/parameters/nested
N
$ sudo rmmod kvm_intel
$ sudo modprobe kvm_intel nested=1
$ cat /sys/module/kvm_intel/parameters/nested
Y
$ virsh domcapabilities | grep vmx
$ sudo systemctl restart libvirtd
$ virsh domcapabilities | grep vmx
$
[regression potential]
TBD, but this should only require better invalidating of the caps cache, which seems very low for any regression potential. More overhead to regenerate the cache more often maybe.
[other info] |
[impact]
libvirt caches the 'nested vmx' capability of the host and does not update that even if the host's capability to handle nested vmx changes. Having this domcapability missing means no guests are able to start any nested, kvm-accelerated, guests. Additionally, since openstack live migration requires matching cpu features, this makes migrating guests that do have vmx enabled impossible to hosts where libvirt thinks nested vmx is disabled.
Once the kernel module (kvm_intel) is reloaded with 'nested' enabled, libvirt does not update its domcapabilities cache, even over a libvirtd restart, or even over an entire system reboot. Only certain conditions cause libvirt to update its capabilities cache (possibly libvirt upgrade, or qemu upgrade, or kernel upgrade...I haven't verified any of those yet)
libvirt creates caches for its domcapabilities at /var/cache/libvirt/qemu/capabilities/.
removing the cache xml files there and restarting libvirtd will cause the caches to be recreated with the correct current values.
[test case]
check the kvm_intel module nested parameter:
$ cat /sys/module/kvm_intel/parameters/nested
Y
it can be Y or N. make sure libvirt agrees with the current setting:
$ virsh domcapabilities | grep vmx
<feature policy='require' name='vmx'/>
if 'nested' is Y, domcapabilities should include a vmx feature line; if 'nested' is N, it should have no output (i.e. vmx not supported in guests).
Then, change the kernel nested setting, and re-check domcapabilities. Restarting libvirtd doesn't update the cache, and even rebooting the entire system doesn't update the cache.
$ virsh domcapabilities | grep vmx
$ cat /sys/module/kvm_intel/parameters/nested
N
$ sudo rmmod kvm_intel
$ sudo modprobe kvm_intel nested=1
$ cat /sys/module/kvm_intel/parameters/nested
Y
$ virsh domcapabilities | grep vmx
$ sudo systemctl restart libvirtd
$ virsh domcapabilities | grep vmx
$
[regression potential]
TBD, but this should only require better invalidating of the caps cache, which seems very low for any regression potential. More overhead to regenerate the cache more often maybe.
[other info]
related RH bugs, though no changes appear to have resulted from either:
https://bugzilla.redhat.com/show_bug.cgi?id=1474874
https://bugzilla.redhat.com/show_bug.cgi?id=1650950 |
|
| 2019-05-23 23:38:45 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2019-05-24 04:55:19 |
Christian Ehrhardt |
libvirt (Ubuntu): status |
New |
Triaged |
|
| 2019-05-24 04:55:21 |
Christian Ehrhardt |
libvirt (Ubuntu): importance |
Undecided |
Medium |
|
| 2019-05-27 09:13:03 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Cosmic |
|
| 2019-05-27 09:13:03 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Cosmic) |
|
| 2019-05-27 09:13:03 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Xenial |
|
| 2019-05-27 09:13:03 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Xenial) |
|
| 2019-05-27 09:13:03 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Bionic |
|
| 2019-05-27 09:13:03 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Bionic) |
|
| 2019-05-27 09:13:08 |
Christian Ehrhardt |
libvirt (Ubuntu): status |
Triaged |
Fix Released |
|
| 2019-05-28 05:44:02 |
Christian Ehrhardt |
summary |
libvirt caches nested vmx capability (in domcapabilities) |
Use more triggers to refresh libvirt capability cache |
|
| 2019-05-28 06:06:44 |
Christian Ehrhardt |
summary |
Use more triggers to refresh libvirt capability cache |
Use changed nested VMX attribute as trigger to refresh libvirt capability cache |
|
| 2019-05-28 06:07:00 |
Christian Ehrhardt |
libvirt (Ubuntu Cosmic): status |
New |
Triaged |
|
| 2019-05-28 06:07:02 |
Christian Ehrhardt |
libvirt (Ubuntu Bionic): status |
New |
Triaged |
|
| 2019-05-28 06:22:34 |
Christian Ehrhardt |
description |
[impact]
libvirt caches the 'nested vmx' capability of the host and does not update that even if the host's capability to handle nested vmx changes. Having this domcapability missing means no guests are able to start any nested, kvm-accelerated, guests. Additionally, since openstack live migration requires matching cpu features, this makes migrating guests that do have vmx enabled impossible to hosts where libvirt thinks nested vmx is disabled.
Once the kernel module (kvm_intel) is reloaded with 'nested' enabled, libvirt does not update its domcapabilities cache, even over a libvirtd restart, or even over an entire system reboot. Only certain conditions cause libvirt to update its capabilities cache (possibly libvirt upgrade, or qemu upgrade, or kernel upgrade...I haven't verified any of those yet)
libvirt creates caches for its domcapabilities at /var/cache/libvirt/qemu/capabilities/.
removing the cache xml files there and restarting libvirtd will cause the caches to be recreated with the correct current values.
[test case]
check the kvm_intel module nested parameter:
$ cat /sys/module/kvm_intel/parameters/nested
Y
it can be Y or N. make sure libvirt agrees with the current setting:
$ virsh domcapabilities | grep vmx
<feature policy='require' name='vmx'/>
if 'nested' is Y, domcapabilities should include a vmx feature line; if 'nested' is N, it should have no output (i.e. vmx not supported in guests).
Then, change the kernel nested setting, and re-check domcapabilities. Restarting libvirtd doesn't update the cache, and even rebooting the entire system doesn't update the cache.
$ virsh domcapabilities | grep vmx
$ cat /sys/module/kvm_intel/parameters/nested
N
$ sudo rmmod kvm_intel
$ sudo modprobe kvm_intel nested=1
$ cat /sys/module/kvm_intel/parameters/nested
Y
$ virsh domcapabilities | grep vmx
$ sudo systemctl restart libvirtd
$ virsh domcapabilities | grep vmx
$
[regression potential]
TBD, but this should only require better invalidating of the caps cache, which seems very low for any regression potential. More overhead to regenerate the cache more often maybe.
[other info]
related RH bugs, though no changes appear to have resulted from either:
https://bugzilla.redhat.com/show_bug.cgi?id=1474874
https://bugzilla.redhat.com/show_bug.cgi?id=1650950 |
[impact]
libvirt caches the 'nested vmx' capability of the host and does not update that even if the host's capability to handle nested vmx changes. Having this domcapability missing means no guests are able to start any nested, kvm-accelerated, guests. Additionally, since openstack live migration requires matching cpu features, this makes migrating guests that do have vmx enabled impossible to hosts where libvirt thinks nested vmx is disabled.
Once the kernel module (kvm_intel) is reloaded with 'nested' enabled, libvirt does not update its domcapabilities cache, even over a libvirtd restart, or even over an entire system reboot. Only certain conditions cause libvirt to update its capabilities cache (possibly libvirt upgrade, or qemu upgrade, or kernel upgrade...I haven't verified any of those yet)
libvirt creates caches for its domcapabilities at /var/cache/libvirt/qemu/capabilities/.
removing the cache xml files there and restarting libvirtd will cause the caches to be recreated with the correct current values.
The fix backports the upstream fix:
https://libvirt.org/git/?p=libvirt.git;a=commit;h=b183a753
Which makes it always check the current vs the last stored attribute.
[test case]
check the kvm_intel module nested parameter:
$ cat /sys/module/kvm_intel/parameters/nested
Y
it can be Y or N. make sure libvirt agrees with the current setting:
$ virsh domcapabilities | grep vmx
<feature policy='require' name='vmx'/>
if 'nested' is Y, domcapabilities should include a vmx feature line; if 'nested' is N, it should have no output (i.e. vmx not supported in guests).
Then, change the kernel nested setting, and re-check domcapabilities. Restarting libvirtd doesn't update the cache, and even rebooting the entire system doesn't update the cache.
$ virsh domcapabilities | grep vmx
$ cat /sys/module/kvm_intel/parameters/nested
N
$ sudo rmmod kvm_intel
$ sudo modprobe kvm_intel nested=1
$ cat /sys/module/kvm_intel/parameters/nested
Y
$ virsh domcapabilities | grep vmx
$ sudo systemctl restart libvirtd
$ virsh domcapabilities | grep vmx
$
Not only should it work, but further configurung libvirt debug [1] the fix should leave a message like this when triggering:
VIR_DEBUG("Outdated capabilities for '%s': kvm kernel nested "
"value changed from %d",)
Test #2:
- restart libvirtd
- call `virsh domcapabilities`
- repeat the above
- this should later on use the cache (faster)
- If it always regenerates the cache (see spawned qemu's and new file
dates) the detection is wrong
[regression potential]
This will make libvirt refresh the capability cache more often. This is a quite expensive tasks (depending on the number of qemu's installed which can be anything from none to all arch emulators and the kvm based ones ~10. Those will be forked and probed again. The new code now adds a rather safe detection as the nested attribute would usually only change on a reboot or a module reload. So it should be rather safe. The one real regression would be if the detection would be wrong and always trigger.
I added Test #2 above to check for that.
[other info]
related RH bugs, though no changes appear to have resulted from either:
https://bugzilla.redhat.com/show_bug.cgi?id=1474874
https://bugzilla.redhat.com/show_bug.cgi?id=1650950 |
|
| 2019-05-28 06:53:36 |
Christian Ehrhardt |
libvirt (Ubuntu Xenial): status |
New |
Triaged |
|
| 2019-05-28 06:58:08 |
Christian Ehrhardt |
description |
[impact]
libvirt caches the 'nested vmx' capability of the host and does not update that even if the host's capability to handle nested vmx changes. Having this domcapability missing means no guests are able to start any nested, kvm-accelerated, guests. Additionally, since openstack live migration requires matching cpu features, this makes migrating guests that do have vmx enabled impossible to hosts where libvirt thinks nested vmx is disabled.
Once the kernel module (kvm_intel) is reloaded with 'nested' enabled, libvirt does not update its domcapabilities cache, even over a libvirtd restart, or even over an entire system reboot. Only certain conditions cause libvirt to update its capabilities cache (possibly libvirt upgrade, or qemu upgrade, or kernel upgrade...I haven't verified any of those yet)
libvirt creates caches for its domcapabilities at /var/cache/libvirt/qemu/capabilities/.
removing the cache xml files there and restarting libvirtd will cause the caches to be recreated with the correct current values.
The fix backports the upstream fix:
https://libvirt.org/git/?p=libvirt.git;a=commit;h=b183a753
Which makes it always check the current vs the last stored attribute.
[test case]
check the kvm_intel module nested parameter:
$ cat /sys/module/kvm_intel/parameters/nested
Y
it can be Y or N. make sure libvirt agrees with the current setting:
$ virsh domcapabilities | grep vmx
<feature policy='require' name='vmx'/>
if 'nested' is Y, domcapabilities should include a vmx feature line; if 'nested' is N, it should have no output (i.e. vmx not supported in guests).
Then, change the kernel nested setting, and re-check domcapabilities. Restarting libvirtd doesn't update the cache, and even rebooting the entire system doesn't update the cache.
$ virsh domcapabilities | grep vmx
$ cat /sys/module/kvm_intel/parameters/nested
N
$ sudo rmmod kvm_intel
$ sudo modprobe kvm_intel nested=1
$ cat /sys/module/kvm_intel/parameters/nested
Y
$ virsh domcapabilities | grep vmx
$ sudo systemctl restart libvirtd
$ virsh domcapabilities | grep vmx
$
Not only should it work, but further configurung libvirt debug [1] the fix should leave a message like this when triggering:
VIR_DEBUG("Outdated capabilities for '%s': kvm kernel nested "
"value changed from %d",)
Test #2:
- restart libvirtd
- call `virsh domcapabilities`
- repeat the above
- this should later on use the cache (faster)
- If it always regenerates the cache (see spawned qemu's and new file
dates) the detection is wrong
[regression potential]
This will make libvirt refresh the capability cache more often. This is a quite expensive tasks (depending on the number of qemu's installed which can be anything from none to all arch emulators and the kvm based ones ~10. Those will be forked and probed again. The new code now adds a rather safe detection as the nested attribute would usually only change on a reboot or a module reload. So it should be rather safe. The one real regression would be if the detection would be wrong and always trigger.
I added Test #2 above to check for that.
[other info]
related RH bugs, though no changes appear to have resulted from either:
https://bugzilla.redhat.com/show_bug.cgi?id=1474874
https://bugzilla.redhat.com/show_bug.cgi?id=1650950 |
[impact]
libvirt caches the 'nested vmx' capability of the host and does not update that even if the host's capability to handle nested vmx changes. Having this domcapability missing means no guests are able to start any nested, kvm-accelerated, guests. Additionally, since openstack live migration requires matching cpu features, this makes migrating guests that do have vmx enabled impossible to hosts where libvirt thinks nested vmx is disabled.
Once the kernel module (kvm_intel) is reloaded with 'nested' enabled, libvirt does not update its domcapabilities cache, even over a libvirtd restart, or even over an entire system reboot. Only certain conditions cause libvirt to update its capabilities cache (possibly libvirt upgrade, or qemu upgrade, or kernel upgrade...I haven't verified any of those yet)
libvirt creates caches for its domcapabilities at /var/cache/libvirt/qemu/capabilities/.
removing the cache xml files there and restarting libvirtd will cause the caches to be recreated with the correct current values.
The fix backports the upstream fix:
https://libvirt.org/git/?p=libvirt.git;a=commit;h=b183a753
Which makes it always check the current vs the last stored attribute.
[test case]
check the kvm_intel module nested parameter:
$ cat /sys/module/kvm_intel/parameters/nested
Y
it can be Y or N. make sure libvirt agrees with the current setting:
$ virsh domcapabilities | grep vmx
<feature policy='require' name='vmx'/>
if 'nested' is Y, domcapabilities should include a vmx feature line; if 'nested' is N, it should have no output (i.e. vmx not supported in guests).
Then, change the kernel nested setting, and re-check domcapabilities. Restarting libvirtd doesn't update the cache, and even rebooting the entire system doesn't update the cache.
$ virsh domcapabilities | grep vmx
$ cat /sys/module/kvm_intel/parameters/nested
N
$ sudo rmmod kvm_intel
$ sudo modprobe kvm_intel nested=1
$ cat /sys/module/kvm_intel/parameters/nested
Y
$ virsh domcapabilities | grep vmx
$ sudo systemctl restart libvirtd
$ virsh domcapabilities | grep vmx
$
Not only should it work, but further configurung libvirt debug [1] the fix should leave a message like this when triggering:
VIR_DEBUG("Outdated capabilities for '%s': kvm kernel nested "
"value changed from %d",)
Test #2:
- restart libvirtd
- call `virsh domcapabilities`
- repeat the above
- this should later on use the cache (faster)
- If it always regenerates the cache (see spawned qemu's and new file
dates) the detection is wrong
Test #3:
- some arches (e.g. s390x) don't have this attribute, check on one of those how their behavior changes.
[regression potential]
This will make libvirt refresh the capability cache more often. This is a quite expensive tasks (depending on the number of qemu's installed which can be anything from none to all arch emulators and the kvm based ones ~10. Those will be forked and probed again. The new code now adds a rather safe detection as the nested attribute would usually only change on a reboot or a module reload. So it should be rather safe. The one real regression would be if the detection would be wrong and always trigger.
I added Test #2 above to check for that.
[other info]
related RH bugs, though no changes appear to have resulted from either:
https://bugzilla.redhat.com/show_bug.cgi?id=1474874
https://bugzilla.redhat.com/show_bug.cgi?id=1650950 |
|
| 2019-06-03 12:03:30 |
Christian Ehrhardt |
libvirt (Ubuntu Bionic): status |
Triaged |
In Progress |
|
| 2019-06-03 12:03:31 |
Christian Ehrhardt |
libvirt (Ubuntu Cosmic): status |
Triaged |
In Progress |
|
| 2019-06-03 12:51:14 |
Christian Ehrhardt |
libvirt (Ubuntu Xenial): status |
Triaged |
Invalid |
|
| 2019-06-07 22:13:58 |
Steve Langasek |
libvirt (Ubuntu Cosmic): status |
In Progress |
Fix Committed |
|
| 2019-06-07 22:13:59 |
Steve Langasek |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-06-07 22:14:03 |
Steve Langasek |
bug |
|
|
added subscriber SRU Verification |
| 2019-06-07 22:14:08 |
Steve Langasek |
tags |
|
verification-needed verification-needed-cosmic |
|
| 2019-06-13 10:34:28 |
Christian Ehrhardt |
tags |
verification-needed verification-needed-cosmic |
verification-done verification-done-cosmic |
|
| 2019-06-17 08:39:56 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2019-06-17 08:39:55 |
Launchpad Janitor |
libvirt (Ubuntu Cosmic): status |
Fix Committed |
Fix Released |
|
| 2019-06-17 08:58:59 |
Łukasz Zemczak |
libvirt (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2019-06-17 08:59:00 |
Łukasz Zemczak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-06-17 08:59:05 |
Łukasz Zemczak |
tags |
verification-done verification-done-cosmic |
verification-done-cosmic verification-needed verification-needed-bionic |
|
| 2019-06-17 13:54:08 |
Christian Ehrhardt |
tags |
verification-done-cosmic verification-needed verification-needed-bionic |
verification-done verification-done-bionic verification-done-cosmic |
|
| 2019-06-24 23:52:20 |
Launchpad Janitor |
libvirt (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
