Mitaka + Trusty (kernel 3.13) not using apparmor capability by default, when it does, live migration doesn't work (/tmp/memfd-XXX can't be created)

[Rafael David Tinoco]

In my environment: Trusty (3.13) + JuJu (1.25) w/ latest charms + Kilo upgraded to Mitaka (already using non-tunnelled live migrations, after latest SRU to disable tunnelled live migrations)

BUG #1

My compute nodes are NOT loading "apparmor" libvirt capability by default:

inaddy@tkcompute01:~$ virsh capabilities | grep apparmor | echo $?
1
inaddy@tkcompute02:~$ virsh capabilities | grep apparmor | echo $?
1
inaddy@tkcompute03:~$ virsh capabilities | grep apparmor | echo $?
1

Because "libvirt" is loaded before apparmor profile is loaded and qemu.conf doesn't specify 'security_driver = "apparmor' in its file. If you try to add the security driver to the file, libvirt and nova-compute won't start because apparmor isn't started when they start. For trusty, apparmor is started as a legacy SYS-V init script, at the end of initialisation, causing this problem.

After re-starting libvirt-bin service, apparmor starts being used:

inaddy@tkcompute01:~$ sudo service libvirt-bin restart
libvirt-bin stop/waiting
libvirt-bin start/running, process 7031
inaddy@tkcompute01:~$ virsh capabilities | grep apparmor | echo $?
0

inaddy@tkcompute02:~$ sudo service libvirt-bin restart
libvirt-bin stop/waiting
libvirt-bin start/running, process 7031
inaddy@tkcompute02:~$ virsh capabilities | grep apparmor | echo $?
0

inaddy@tkcompute03:~$ sudo service libvirt-bin restart
libvirt-bin stop/waiting
libvirt-bin start/running, process 7031
inaddy@tkcompute03:~$ virsh capabilities | grep apparmor | echo $?
0

BUG #2 (after fixing BUG #1)

And, when libvirt starts using apparmor, and creating apparmor profiles for every virtual machine created in the compute nodes, mitaka qemu (2.5) uses a fallback mechanism for creating shared memory for live-migrations. This fall back mechanism, on kernels 3.13 - that don't have memfd_create() system-call, try to create files on /tmp/ directory and fails.. causing live-migration not to work.

Trusty with kernel 3.13 + Mitaka with qemu 2.5 + apparmor capability = can't live migrate.

From qemu 2.5, logic is on :

void *qemu_memfd_alloc(const char *name, size_t size, unsigned int seals, int *fd)
{
    if (memfd_create)... ### only works with HWE kernels

    else ### 3.13 kernels, gets blocked by apparmor
       tmpdir = g_get_tmp_dir
       ...
       mfd = mkstemp(fname)
}

And you can see the errors:

From the host trying to send the virtual machine:

2016-08-15 16:36:26.160 1974 ERROR nova.virt.libvirt.driver [req-0cac612b-8d53-4610-b773-d07ad6bacb91 691a581cfa7046278380ce82b1c38ddd 133ebc3585c041aebaead8c062cd6511 - - -] [instance: 2afa1131-bc8c-43d2-9c4a-962c1bf7723e] Migration operation has aborted
2016-08-15 16:36:26.248 1974 ERROR nova.virt.libvirt.driver [req-0cac612b-8d53-4610-b773-d07ad6bacb91 691a581cfa7046278380ce82b1c38ddd 133ebc3585c041aebaead8c062cd6511 - - -] [instance: 2afa1131-bc8c-43d2-9c4a-962c1bf7723e] Live Migration failure: internal error: unable to execute QEMU command 'migrate': Migration disabled: failed to allocate shared memory

From the host trying to receive the virtual machine:

Aug 15 16:36:19 tkcompute01 kernel: [ 1194.356794] type=1400 audit(1471289779.791:72): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" pid=12565 comm="apparmor_parser"
Aug 15 16:36:19 tkcompute01 kernel: [ 1194.357048] type=1400 audit(1471289779.791:73): apparmor="STATUS" operation="profile_load" profile="unconfined" name="qemu_bridge_helper" pid=12565 comm="apparmor_parser"
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.877027] type=1400 audit(1471289780.311:74): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" pid=12613 comm="apparmor_parser"
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.904407] type=1400 audit(1471289780.343:75): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="qemu_bridge_helper" pid=12613 comm="apparmor_parser"
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.973064] type=1400 audit(1471289780.407:76): apparmor="DENIED" operation="mknod" profile="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" name="/tmp/memfd-tNpKSj" pid=12625 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=107 ouid=107
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.979871] type=1400 audit(1471289780.411:77): apparmor="DENIED" operation="open" profile="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" name="/tmp/" pid=12625 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Aug 15 16:36:20 tkcompute01 kernel: [ 1194.979881] type=1400 audit(1471289780.411:78): apparmor="DENIED" operation="open" profile="libvirt-2afa1131-bc8c-43d2-9c4a-962c1bf7723e" name="/var/tmp/" pid=12625 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=0

When leaving libvirt without apparmor capabilities (thus not confining virtual machines on compute nodes, the live migration works as expected, so, clearly, apparmor is stepping into the live migration). I'm sure that virtual machines have to be confined and that this isn't the desired behaviour...

Still trying to figure out rules from /etc/apparmor.d/abstraction/libvirt-qemu that are not allowing the live migration to work.

[Morgan Fainberg (mdrnstm)]

This looks like a bug in Libvirt / Qemu and/or packaging for libvirt / Qemu, not really something that can be fixed in Nova unless you have a recommendation on a change Nova can make as a call to qemu that would change this interaction with AppArmor.

I don't think Nova itself ships apparmor profiles (or SELinux for that matter), which is left to packagers (Ubuntu, Debian, Red Hat, etc).

[Morgan Fainberg (mdrnstm)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Rafael David Tinoco (rafaeldtinoco)]

I do agree that this looks like a bug in Libvirt.. since both files:

/etc/apparmor.d/abstractions/libvirt-qemu
/etc/apparmor.d/libvirt/TEMPLATE.qemu

come from libvirt-bin.

We have other users facing the same problem: I could reproduce problem described by other 2 or 3 end-users (Openstack upgrade, problems with live migration using kernel 3.13: qemu complains about memory allocation during live migration). Still struggling with apparmor, not my specialty.

[Rafael David Tinoco (rafaeldtinoco)]

Okay, so here is what is happening after some more investigation:

- qemu only + libvirt w/ apparmor capability + kernel 3.13 = ok
- kvm + libvirt w/ apparmor capability + kernel 3.13 = not ok

Even when enabling writes to "/tmp/memfd-XXX" file, I still get this error:

# from sender:

/build/qemu-uiagOj/qemu-2.5+dfsg/nbd.c:nbd_receive_reply():L898: read failed

# from receiver:

2016-08-16T16:48:56.996296Z qemu-system-x86_64: Not a migration stream
2016-08-16T16:48:56.996954Z qemu-system-x86_64: load of migration failed: Invalid argument

If I disable libvirt I don't get this. I think that apparmor is blocking the nbd transfer somehow.

[Rafael David Tinoco (rafaeldtinoco)]

Sorry, if i disable apparmor from libvirt I don't get those errors AND the live migration happens, that is what i meant to say.

[Rafael David Tinoco (rafaeldtinoco)]

Ok, so I narrowed this down and it was easier than I thought.

Live migrations work after an initial boot because libvirt doesn't use apparmor capability. After restarting libvirt-bin... it starts using apparmor capability and we get the following error, on the compute node, when starting a virtual machine:

mkstemp: No such file or directory

If I got this right, this error won't allow this instance to migrate its devices using libvirt when we try to tell nova to live-migrate.

If you change /etc/apparmor.d/abstractions/libvirt-qemu:

  # silence spurious denials (see lp#1403648)
  # deny /tmp/{,**} r,
  /tmp/{,**} rw,
  deny /var/tmp/{,**} r,

And stop the virtual machine, start it again and try the live migration, it works.

[Rafael David Tinoco (rafaeldtinoco)]

O.. after the change to the abstraction you have to:

$ sudo service apparmor restart
$ sudo service libvirt-bin restart

And then start the virtual machine again. It shall work.

[Rafael David Tinoco (rafaeldtinoco)]

This is how to reproduce and how to mitigate the issue:

####
#### To reproduce the issue again:
####

- Start all compute nodes with kernel 3.13

root@tkcompute01:~# uname -a
Linux tkcompute01 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

- Rotate logs:

root@tkcompute01:~# logrotate -f /etc/logrotate.conf

- Make sure "apparmor" is enabled in libvirt:

root@tkcompute01:~# virsh capabilities | grep apparmor

If there is no output from the above, restart libvirt and check again:

root@tkcompute01:~# service libvirt-bin restart
libvirt-bin stop/waiting
libvirt-bin start/running, process 37824

root@tkcompute01:~# virsh capabilities | grep apparmor
<model>apparmor</model>

- Start a virtual machine using nova:

inaddy@tkjuju:~$ nova start trustyblock

- Try to live-migrate the machine:

inaddy@tkjuju:~$ nova live-migration --block-migrate trustyblock

Check if it was migrated or not (possibly not).

####
#### To mitigate the situation
####

- Stop all virtual machines to be live-migrated

- IN ALL compute nodes, edit following file:

root@tkcompute01:~# vi /etc/apparmor.d/abstractions/libvirt-qemu

And change:

# silence spurious denials (see lp#1403648)
deny /tmp/{,**} r,
deny /var/tmp/{,**} r,

For:

# silence spurious denials (see lp#1403648)
# deny /tmp/{,**} r,
/tmp/{,**} rw,
deny /var/tmp/{,**} r,

Restarting apparmor and libvirt-bin services right after:

root@tkcompute01:~# service apparmor restart
* Reloading AppArmor profiles [ OK ]
Skipping profile in /etc/apparmor.d/disable: usr.bin.nova-compute
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd

root@tkcompute01:~# service libvirt-bin restart
libvirt-bin stop/waiting
libvirt-bin start/running, process 39905

- Rotate logs

root@tkcompute01:~# logrotate -f /etc/logrotate.conf

- Start the virtual machine using nova:

inaddy@tkjuju:~$ nova start trustyblock

- Try to live-migrate the machine:

inaddy@tkjuju:~$ nova live-migration --block-migrate trustyblock

Check if it was migrated, possibly yes.

[Jeremy Stanley (fungi)]

I wonder if this should have been filed against an Ubuntu package rather than against the upstream OpenStack Nova bug tracker. James Page, looks like you were Cc'd on the original filing... any opinion? The OpenStack VMT is hesitant to make the call as to whether this bug can be switched to public, but is pretty sure this is not something upstream OpenStack developers will be able to fix.

[James Page (james-page)]

I agree with Jeremy - this looks like a problem with libvirt/qemu in Ubuntu, not with Nova itself; I'll raise appropriate Ubuntu tasks and I'm good with marking the OpenStack project tasks as invalid.

[James Page (james-page)]

Attempting reproduction on trusty-mitaka (note that I've tested live migration on xenial-mitaka and did not hit this issue, likely due to the newer kernel version).

[Jeremy Stanley (fungi)]

Thanks, James. I've switched the upstream OpenStack Nova and Security Advisory tasks to invalid.

[Rafael David Tinoco (rafaeldtinoco)]

James,

Note that there are 2 bugs here.. you said you tried to reproduce with xenial-mitaka..

1) libvirt starts before apparmor (not loading apparmor policy).

After fixing (1) we have:

2) libvirt abstractions block /tmp/memfd-XXX during qemu initialization -> will block live migrations.

In trusty happens (1).
In 3.13 happens (2).

[Rafael David Tinoco (rafaeldtinoco)]

For (1): $ virsh capabilities | grep app

$ sudo update-rc.d -f apparmor remove
$ sudo update-rc.d -f libvirt-bin remove
$ cd /etc/init ; mv libvirt-bin.override libvirt-bin.override.old ; echo manual | sudo tee libvirt-bin.override ; cd -
$ sudo update-rc.d -f apparmor defaults 20
$ sudo update-rc.d -f libvirt-bin defaults 21
$ sudo reboot

OBS: deactivating upstart script and re-activating sys-v with proper order solves the issue:

$ virsh capabilities | grep app
      <model>apparmor</model>

[Rafael David Tinoco (rafaeldtinoco)]

Okay, so altering libvirt-bin.conf for upstart:

pre-start script
    [ -r /etc/default/libvirt-bin ] && . /etc/default/libvirt-bin
    [ ! "x$start_libvirtd" = "xyes" ] && { stop; exit 0; }
    mkdir -p /var/run/libvirt
    # Clean up a pidfile that might be left around
    rm -f /var/run/libvirtd.pid
    # Make sure libvirt is confined (LP: #1613423)
    /lib/init/apparmor-profile-load usr.sbin.libvirtd
end script

Does the trick for (1),

And altering /etc/apparmor.d/abstractions/libvirt-qemu

# silence spurious denials (see lp#1403648)
deny /tmp/{,**} r,
deny /var/tmp/{,**} r,

For:

# silence spurious denials (see lp#1403648)
# deny /tmp/{,**} r,
/tmp/{,**} rw,
deny /var/tmp/{,**} r,

Does the trick for (2).

I'll provide the debdiffs.

[Jamie Strandboge (jdstrand)]

Adjusting /etc/apparmor.d/abstractions/libvirt-qemu means that every VM gets access to the /tmp directory and since qemu runs under the same user for each VM, that means that we aren't enforcing guest isolation and a malicious guest could tamper with files of another guest.

Can you determine what is using /tmp and adjust it to use a VM-specific directory instead?

[Rafael David Tinoco (rafaeldtinoco)]

Affects:

(1) Trusty + libvirt (1.2.2-0ubuntu13.1.19):

# virsh capabilities | grep -q apparmor ; echo $?
1

# dpkg -l | grep libvirt | awk '{print $2." "$3}'
libvirt-bin 1.2.2-0ubuntu13.1.19
libvirt0 1.2.2-0ubuntu13.1.19

After changing /etc/init/libvirt-bin.conf:

# virsh capabilities | grep -q apparmor ; echo $?
0

SO, considering EOL, it affects:

Trusty, Kilo UCA, Liberty UCA, Xenial, Yakkety
(Icehouse UCA, Mitaka UCA and Newton UCA should be synced)

Not sure about Yakkety (Newton) since Trusty is the last upstart based.

[Rafael David Tinoco (rafaeldtinoco)]

Jamie,

Only if I change qemu code. It uses /tmp/memfd-XXX for memory map files (to be used in future live-migrations). I could change qemu code to use something else, but, that would make qemu to behave differently from upstream and harder to maintain.

Instead what we could do would have :

deny /tmp/{,**^memfd-*} rw,

I'm trying to limit the allow rule to /tmp/memfd-XXX only.

[Rafael David Tinoco (rafaeldtinoco)]

For all those interested:

/etc/apparmor.d/abstractions/libvirt-qemu:

# silence spurious denials (see lp#1403648)
# memfd to allow live migrations on 3.13 (see lp#1613423)
/tmp/memfd* rw,
deny /tmp/{,**^memfd*} r,
deny /var/tmp/{,**} r,

I was able to live migrate using that approach for (2).

Providing the SRU for both problems...

[Jamie Strandboge (jdstrand)]

Your rule isn't quite right. This is what you want:

/tmp/memfd* rw,
deny /tmp/{,[^m]**,m[^e]**,me[^m]**,mem[^f]**,memf[^d]**,m,me,mem,memf} r,
deny /var/tmp/{,**} r,

That said-- have you approached upstream? qemu and libvirt work very closely together and apparmor in libvirt is used by a lot of distros besides Ubuntu. Ideally we would not distro patch this but instead get upstream input (people in CDO have been working with Debian and upstream to reduce Ubuntu's delta for some time). The proper fix would be for the file to be dynamically added to the VM's file in /etc/apparmor.d/libvirt/libvirt-<uuid>.files using virt-aa-helper in libvirt (see src/security/virt-aa-helper.c in libvirt). There are a couple of ways to do this:

- have libvirt tell qemu to use '/tmp/memfd-<vmname>-XXXXXX" and adjust main() in src/security/virt-aa-helper.c to do:
  virBufferAsprintf(&buf, " \"/tmp/memfd-%s-*\" rw,\n"),
- have libvirt and qemu coordinate on the file to use and adjust get_files() to extract that and add it to the policy

There might be other options. I can say that live migration used to work and I think it would be best to work with upstream on why it isn't now instead of opening up the security policy in this manner.

[Jamie Strandboge (jdstrand)]

I obviously meant:

virBufferAsprintf(&buf, " \"/tmp/memfd-%s-*\" rw,\n", ctl->def->name);

[Rafael David Tinoco (rafaeldtinoco)]

Hello Jamie,

I see what you mean now.

I wasn't aware on how qemu implemented the security driver. I read your security/security_apparmor, security/security_driver & virt-aa-helper. I'll open upstream discussions on how to enable libvirt <-> qemu to exchange memfd filename (vhost log file), allowing the security driver to correctly create a rule to allow that *specific-to-instance* vhost log file (without letting every qemu process to access each other's).

Will get back soon.

[Rafael David Tinoco (rafaeldtinoco)]

I must say that, for a SRU, your first approach:

- have libvirt tell qemu to use '/tmp/memfd-<vmname>-XXXXXX" and adjust main() in src/security/virt-aa-helper.c to do:

virBufferAsprintf(&buf, " \"/tmp/memfd-%s-*\" rw,\n", ctl->def->name);

seems more straight forward.

Specially because RedHat's selinux driver might also have to deal with this - if their kernels are not backporting memfd_create().

[Rafael David Tinoco (rafaeldtinoco)]

Hello Jamie,

I came up with this patch for QEMU:

http://paste.ubuntu.com/23217056/

I'm finishing libvirt patch so I can propose upstream QEMU already sure that libvirt will benefit from this change. Right after I'll propose libvirt upstream patch (changing vert-aa-helper logic).

Cheers!

Rafael

[Rafael David Tinoco (rafaeldtinoco)]

Improved it a little bit: http://paste.ubuntu.com/23217333/

Will start doing all checks and proposes. Will get back after finishing everything.

[Jamie Strandboge (jdstrand)]

Thanks for this!

Note that 'fname = g_strdup_printf("%s/memfd-%s", tmpdir, info->UUID);' is a predictable filename-- what if you did this instead:

fname = g_strdup_printf("%s/memfd-%s-XXXXXX", tmpdir, info->UUID);
if((mfd = mkstemp(fname)) == -1) {
...

ie, treat the info->UUID case exactly the same as the other two: allows have some trailing random bits at the end. This if fine for profiling in virt-aa-helper since you can use a glob rule.

[Rafael David Tinoco (rafaeldtinoco)]

https://bugs.launchpad.net/qemu/+bug/1626972

[Rafael David Tinoco (rafaeldtinoco)]

I'll follow to see if patch was accepted upstream:

https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg06191.html
https://<email address hidden>/msg400892.html

[Rafael David Tinoco (rafaeldtinoco)]

Discussion going on in:

https://bugs.launchpad.net/qemu/+bug/1626972

[Rafael David Tinoco (rafaeldtinoco)]

Preferred way for this fix was:

commit 0d34fbabc13891da41582b0823867dc5733fffef
Author: Rafael David Tinoco <email address hidden>
Date: Mon Oct 24 15:35:03 2016

    vhost: migration blocker only if shared log is used

    Commit 31190ed7 added a migration blocker in vhost_dev_init() to
    check if memfd would succeed. It is better if this blocker first
    checks if vhost backend requires shared log. This will avoid a
    situation where a blocker is added inappropriately (e.g. shared
    log allocation fails when vhost backend doesn't support it).

    Signed-off-by: Rafael David Tinoco <email address hidden>
    Reviewed-by: Marc-André Lureau <email address hidden>
    Reviewed-by: Michael S. Tsirkin <email address hidden>
    Signed-off-by: Michael S. Tsirkin <email address hidden>

And accepted upstream. I'm closing this bug.

[Rafael David Tinoco (rafaeldtinoco)]

There wasn't any action in libvirt side (so I'm making this incomplete).

[Rafael David Tinoco (rafaeldtinoco)]

QEMU BUG: https://bugs.launchpad.net/qemu/+bug/1626972

[Rafael David Tinoco (rafaeldtinoco)]

Let me clarify better why I'm flagging this as incomplete:

14:20 <rafaeldtinoco> my upstream fix was to qemu
14:20 <rafaeldtinoco> and it fixed the issue by not having the memfd file created
14:20 <rafaeldtinoco> for live migrations.. but there might be cases where there should be one
14:21 <rafaeldtinoco> in those cases, libvirt should be creating the apparmor rule to allow the memfd backing file creation

14:22 <andreas> so the bug is valid for libvirt?
14:22 <andreas> that's question #1
14:22 <andreas> in which case, it's not incomplete, but new, or confirmed, or triaged
14:22 <andreas> question #2 is if it's trusty only

14:22 <rafaeldtinoco> because memfd is not used for all live migration cases we have with openstack
14:22 <rafaeldtinoco> memfd backing file creation was pretty new back then
14:23 <rafaeldtinoco> there was a big discussion whether libvirt should ever touch that
14:23 <rafaeldtinoco> (changing the apparmor to guarantee memfd file creation)

14:24 <rafaeldtinoco> andreas: https://github.com/rafaeldtinoco/work/blob/master/sources/scratch/oldstuff/qemu-idea-patches/0004-vhost-secure-vhost-shared-log-files-using-argv-parem.patch
14:24 <rafaeldtinoco> i tried to create the file through qemu
14:24 <rafaeldtinoco> and there was discussion saying yes and saying no

14:24 <rafaeldtinoco> thats why I think its incomplete
14:24 <rafaeldtinoco> i dont think it was ever decided
14:24 <rafaeldtinoco> and its a corner case not being used
14:24 <andreas> ok

[Launchpad Janitor (janitor)]

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]