| 2016-09-23 11:03:00 |
Rafael David Tinoco |
bug |
|
|
added bug |
| 2016-09-23 11:05:51 |
Rafael David Tinoco |
qemu: status |
New |
In Progress |
|
| 2016-09-23 11:05:55 |
Rafael David Tinoco |
qemu: assignee |
|
Rafael David Tinoco (inaddy) |
|
| 2016-11-02 08:30:57 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2016-11-03 09:58:50 |
Ivan Suzdal |
bug |
|
|
added subscriber Ivan Suzdal |
| 2016-11-09 07:45:28 |
Christian Ehrhardt |
bug |
|
|
added subscriber ChristianEhrhardt |
| 2016-11-18 10:04:39 |
Rafael David Tinoco |
bug task added |
|
qemu (Ubuntu) |
|
| 2016-11-18 10:04:47 |
Rafael David Tinoco |
qemu (Ubuntu): status |
New |
In Progress |
|
| 2016-11-18 10:04:49 |
Rafael David Tinoco |
qemu (Ubuntu): assignee |
|
Rafael David Tinoco (inaddy) |
|
| 2016-11-18 10:06:56 |
Louis Bouchard |
nominated for series |
|
Ubuntu Yakkety |
|
| 2016-11-18 10:06:56 |
Louis Bouchard |
bug task added |
|
qemu (Ubuntu Yakkety) |
|
| 2016-11-18 10:06:56 |
Louis Bouchard |
nominated for series |
|
Ubuntu Zesty |
|
| 2016-11-18 10:06:56 |
Louis Bouchard |
bug task added |
|
qemu (Ubuntu Zesty) |
|
| 2016-11-18 10:06:56 |
Louis Bouchard |
nominated for series |
|
Ubuntu Xenial |
|
| 2016-11-18 10:06:56 |
Louis Bouchard |
bug task added |
|
qemu (Ubuntu Xenial) |
|
| 2016-11-18 10:07:19 |
Rafael David Tinoco |
qemu (Ubuntu Xenial): status |
New |
In Progress |
|
| 2016-11-18 10:07:22 |
Rafael David Tinoco |
qemu (Ubuntu Yakkety): status |
New |
In Progress |
|
| 2016-11-18 10:07:25 |
Rafael David Tinoco |
qemu (Ubuntu Xenial): assignee |
|
Rafael David Tinoco (inaddy) |
|
| 2016-11-18 10:07:27 |
Rafael David Tinoco |
qemu (Ubuntu Yakkety): assignee |
|
Rafael David Tinoco (inaddy) |
|
| 2016-11-18 11:31:57 |
Billy Olsen |
bug task added |
|
cloud-archive |
|
| 2016-11-18 11:32:25 |
Billy Olsen |
nominated for series |
|
cloud-archive/mitaka |
|
| 2016-11-18 11:32:25 |
Billy Olsen |
nominated for series |
|
cloud-archive/newton |
|
| 2016-11-18 11:32:59 |
Rafael David Tinoco |
cloud-archive: status |
New |
In Progress |
|
| 2016-11-18 11:33:01 |
Rafael David Tinoco |
cloud-archive: assignee |
|
Rafael David Tinoco (inaddy) |
|
| 2016-11-22 10:01:03 |
Rafael David Tinoco |
attachment added |
|
xenial_qemu_2.5+dfsg-5ubuntu10.7.debdiff https://bugs.launchpad.net/qemu/+bug/1626972/+attachment/4781425/+files/xenial_qemu_2.5+dfsg-5ubuntu10.7.debdiff |
|
| 2016-11-22 11:47:46 |
Rafael David Tinoco |
attachment added |
|
yakkety_qemu_2.6.1+dfsg-0ubuntu5.2.debdiff https://bugs.launchpad.net/qemu/+bug/1626972/+attachment/4781464/+files/yakkety_qemu_2.6.1+dfsg-0ubuntu5.2.debdiff |
|
| 2016-11-22 12:08:04 |
Rafael David Tinoco |
description |
And, when libvirt starts using apparmor, and creating apparmor profiles for every virtual machine created in the compute nodes, mitaka qemu (2.5 - and upstream also) uses a fallback mechanism for creating shared memory for live-migrations. This fall back mechanism, on kernels 3.13 - that don't have memfd_create() system-call, try to create files on /tmp/ directory and fails.. causing live-migration not to work.
Trusty with kernel 3.13 + Mitaka with qemu 2.5 + apparmor capability = can't live migrate.
From qemu 2.5, logic is on :
void *qemu_memfd_alloc(const char *name, size_t size, unsigned int seals, int *fd)
{
if (memfd_create)... ### only works with HWE kernels
else ### 3.13 kernels, gets blocked by apparmor
tmpdir = g_get_tmp_dir
...
mfd = mkstemp(fname)
}
And you can see the errors:
From the host trying to send the virtual machine:
2016-08-15 16:36:26.160 1974 ERROR nova.virt.libvirt.driver [req-0cac612b-8d53-4610-b773-d07ad6bacb91 691a581cfa7046278380ce82b1c38ddd 133ebc3585c041aebaead8c062cd6511 - - -] [instance: 2afa1131-bc8c-43d2-9c4a-962c1bf7723e] Migration operation has aborted
2016-08-15 16:36:26.248 1974 ERROR nova.virt.libvirt.driver [req-0cac612b-8d53-4610-b773-d07ad6bacb91 691a581cfa7046278380ce82b1c38ddd 133ebc3585c041aebaead8c062cd6511 - - -] [instance: 2afa1131-bc8c-43d2-9c4a-962c1bf7723e] Live Migration failure: internal error: unable to execute QEMU command 'migrate': Migration disabled: failed to allocate shared memory
From the host trying to receive the virtual machine:
Aug 15 16:36:19 tkcompute01 kernel: [ 1194.356794] type=1400 audit(1471289779.791:72): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" pid=12565 comm="apparmor_parser"
Aug 15 16:36:19 tkcompute01 kernel: [ 1194.357048] type=1400 audit(1471289779.791:73): apparmor="STATUS" operation="profile_load" profile="unconfined" name="qemu_bridge_helper" pid=12565 comm="apparmor_parser"
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.877027] type=1400 audit(1471289780.311:74): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" pid=12613 comm="apparmor_parser"
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.904407] type=1400 audit(1471289780.343:75): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="qemu_bridge_helper" pid=12613 comm="apparmor_parser"
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.973064] type=1400 audit(1471289780.407:76): apparmor="DENIED" operation="mknod" profile="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" name="/tmp/memfd-tNpKSj" pid=12625 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=107 ouid=107
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.979871] type=1400 audit(1471289780.411:77): apparmor="DENIED" operation="open" profile="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" name="/tmp/" pid=12625 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.979881] type=1400 audit(1471289780.411:78): apparmor="DENIED" operation="open" profile="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" name="/var/tmp/" pid=12625 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
When leaving libvirt without apparmor capabilities (thus not confining virtual machines on compute nodes, the live migration works as expected, so, clearly, apparmor is stepping into the live migration). I'm sure that virtual machines have to be confined and that this isn't the desired behaviour... |
[Impact]
* Updated QEMU (from UCA) live migration doesn't work with 3.13 kernels.
* QEMU code checks if it can create /tmp/memfd-XXX files wrongly.
* Apparmor will block access to /tmp/ and QEMU will fail migrating.
[Test Case]
* Install 2 Ubuntu Trusty (3.13) + UCA Mitaka + apparmor rules.
* Try to live-migration from one to another.
* Apparmor will block creation of /tmp/memfd-XXX files.
[Regression Potential]
Pros:
* Exhaustively tested this.
* Worked with upstream on this fix.
* I'm implementing new vhost log mechanism for upstream.
* One line change to a blocker that is already broken.
Cons:
* To break live migration in other circumstances.
[Other Info]
* Christian Ehrhardt has been following this.
ORIGINAL DESCRIPTION:
When libvirt starts using apparmor, and creating apparmor profiles for every virtual machine created in the compute nodes, mitaka qemu (2.5 - and upstream also) uses a fallback mechanism for creating shared memory for live-migrations. This fall back mechanism, on kernels 3.13 - that don't have memfd_create() system-call, try to create files on /tmp/ directory and fails.. causing live-migration not to work.
Trusty with kernel 3.13 + Mitaka with qemu 2.5 + apparmor capability = can't live migrate.
From qemu 2.5, logic is on :
void *qemu_memfd_alloc(const char *name, size_t size, unsigned int seals, int *fd)
{
if (memfd_create)... ### only works with HWE kernels
else ### 3.13 kernels, gets blocked by apparmor
tmpdir = g_get_tmp_dir
...
mfd = mkstemp(fname)
}
And you can see the errors:
From the host trying to send the virtual machine:
2016-08-15 16:36:26.160 1974 ERROR nova.virt.libvirt.driver [req-0cac612b-8d53-4610-b773-d07ad6bacb91 691a581cfa7046278380ce82b1c38ddd 133ebc3585c041aebaead8c062cd6511 - - -] [instance: 2afa1131-bc8c-43d2-9c4a-962c1bf7723e] Migration operation has aborted
2016-08-15 16:36:26.248 1974 ERROR nova.virt.libvirt.driver [req-0cac612b-8d53-4610-b773-d07ad6bacb91 691a581cfa7046278380ce82b1c38ddd 133ebc3585c041aebaead8c062cd6511 - - -] [instance: 2afa1131-bc8c-43d2-9c4a-962c1bf7723e] Live Migration failure: internal error: unable to execute QEMU command 'migrate': Migration disabled: failed to allocate shared memory
From the host trying to receive the virtual machine:
Aug 15 16:36:19 tkcompute01 kernel: [ 1194.356794] type=1400 audit(1471289779.791:72): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" pid=12565 comm="apparmor_parser"
Aug 15 16:36:19 tkcompute01 kernel: [ 1194.357048] type=1400 audit(1471289779.791:73): apparmor="STATUS" operation="profile_load" profile="unconfined" name="qemu_bridge_helper" pid=12565 comm="apparmor_parser"
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.877027] type=1400 audit(1471289780.311:74): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" pid=12613 comm="apparmor_parser"
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.904407] type=1400 audit(1471289780.343:75): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="qemu_bridge_helper" pid=12613 comm="apparmor_parser"
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.973064] type=1400 audit(1471289780.407:76): apparmor="DENIED" operation="mknod" profile="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" name="/tmp/memfd-tNpKSj" pid=12625 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=107 ouid=107
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.979871] type=1400 audit(1471289780.411:77): apparmor="DENIED" operation="open" profile="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" name="/tmp/" pid=12625 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.979881] type=1400 audit(1471289780.411:78): apparmor="DENIED" operation="open" profile="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" name="/var/tmp/" pid=12625 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
When leaving libvirt without apparmor capabilities (thus not confining virtual machines on compute nodes, the live migration works as expected, so, clearly, apparmor is stepping into the live migration). I'm sure that virtual machines have to be confined and that this isn't the desired behaviour... |
|
| 2016-11-22 12:09:32 |
Rafael David Tinoco |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2016-11-22 12:29:00 |
Rafael David Tinoco |
attachment added |
|
zesty_qemu_2.6.1+dfsg-0ubuntu7.debdiff https://bugs.launchpad.net/qemu/+bug/1626972/+attachment/4781485/+files/zesty_qemu_2.6.1+dfsg-0ubuntu7.debdiff |
|
| 2016-11-22 13:49:21 |
Christian Ehrhardt |
bug |
|
|
added subscriber Ubuntu Server Team |
| 2016-11-23 11:27:05 |
Christian Ehrhardt |
attachment added |
|
Collection of extra test logs if we have to search for anything in them later on. https://bugs.launchpad.net/qemu/+bug/1626972/+attachment/4781992/+files/bug-1626972-migration-fix-tinoco-sts-extraverifications.tgz |
|
| 2016-11-23 15:26:05 |
Launchpad Janitor |
qemu (Ubuntu Zesty): status |
In Progress |
Fix Released |
|
| 2016-11-23 22:37:44 |
Martin Pitt |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2016-11-23 22:37:53 |
Martin Pitt |
qemu (Ubuntu Xenial): status |
In Progress |
Fix Committed |
|
| 2016-11-23 22:37:55 |
Martin Pitt |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2016-11-23 22:37:57 |
Martin Pitt |
bug |
|
|
added subscriber SRU Verification |
| 2016-11-23 22:38:03 |
Martin Pitt |
tags |
|
verification-needed |
|
| 2016-11-24 08:35:27 |
Thomas Huth |
qemu: status |
In Progress |
Fix Committed |
|
| 2016-11-28 20:36:02 |
James Page |
cloud-archive: status |
In Progress |
Fix Committed |
|
| 2016-12-01 14:56:55 |
James Page |
cloud-archive: status |
Fix Committed |
Fix Released |
|
| 2016-12-01 19:10:43 |
Brian Murray |
qemu (Ubuntu Yakkety): status |
In Progress |
Fix Committed |
|
| 2016-12-08 09:29:22 |
James Page |
bug task added |
|
cloud-archive/mitaka |
|
| 2016-12-08 09:29:38 |
James Page |
cloud-archive/mitaka: status |
New |
Fix Committed |
|
| 2016-12-08 09:29:41 |
James Page |
cloud-archive: status |
Fix Released |
Invalid |
|
| 2017-01-11 13:48:08 |
Rafael David Tinoco |
tags |
verification-needed |
verification-done |
|
| 2017-01-16 11:39:32 |
Thomas Huth |
qemu: status |
Fix Committed |
Fix Released |
|
| 2017-01-19 16:03:59 |
Launchpad Janitor |
qemu (Ubuntu Yakkety): status |
Fix Committed |
Fix Released |
|
| 2017-01-19 16:04:10 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2017-01-25 10:23:13 |
Launchpad Janitor |
qemu (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2017-08-07 14:51:13 |
James Page |
cloud-archive/mitaka: status |
Fix Committed |
Fix Released |
|
| 2018-08-28 06:30:32 |
ctct-sup |
bug |
|
|
added subscriber ctct-sup |
