Charm sets up rbd cache in a way that doesn't allow libvirt to access the admin socket

[Tilman Baumann]

The charm sets up rbd cache with hard-coded values and file permissions that don't work with libvirt and AppArmor
https://github.com/openstack/charm-nova-compute/blob/c744e052347d8ddfae88804a0ad0bdfdf4f5ae0d/hooks/nova_compute_context.py#L320

The admin socket file is not accessible to libvirt via AppArmor. There is a bug for that in libvirt. https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1779674

The other problem is, that quemu libvirt doesn't have access to the directory created in the same code block as it only belongs to root.

[Tilman Baumann (tilmanbaumann)]

There is the possibility that this causes rbd cache not to work. But it's likely just the admin socket that fails.

May 23 10:06:38 var0tf1a-cmp3s40d2yl-hr nova-compute: 2018-05-23 10:06:38.972 55598 WARNING nova.compute.manager [req-40e3686c-d70b-4d0b-8e65-9b6ec1847903 - - - - -] [instance: c364f41a-a2df-40e5-be43-1e47dd4e4fd7] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4
May 23 10:06:46 var0tf1a-cmp3s40d2yl-hr /usr/share/filebeat/bin/filebeat[10378]: log.go:91: Harvester started for file: /var/log/upstart/nova-compute.log
May 23 10:06:46 var0tf1a-cmp3s40d2yl-hr /usr/share/filebeat/bin/filebeat[10378]: log.go:91: Harvester started for file: /var/log/nova/nova-compute.log
May 23 10:06:50 var0tf1a-cmp3s40d2yl-hr kernel: [10110228.305439] audit: type=1400 audit(1527070010.408:172758): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-c364f41a-a2df-40e5-be43-1e47dd4e4fd7" pid=24777 comm="apparmor_parser"
May 23 10:06:50 var0tf1a-cmp3s40d2yl-hr kernel: [10110228.305762] audit: type=1400 audit(1527070010.408:172759): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-c364f41a-a2df-40e5-be43-1e47dd4e4fd7//qemu_bridge_helper" pid=24777 comm="apparmor_parser"
May 23 10:06:50 var0tf1a-cmp3s40d2yl-hr qemu-system-x86_64: 2018-05-23 10:06:50.530151 7f5c1da45ac0 -1 asok(0x561ffd079ee0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/rbd-client-24780.asok': (13) Permission denied
May 23 10:06:50 var0tf1a-cmp3s40d2yl-hr kernel: [10110228.421988] audit: type=1400 audit(1527070010.524:172760): apparmor="DENIED" operation="mknod" profile="libvirt-c364f41a-a2df-40e5-be43-1e47dd4e4fd7" name="/run/ceph/rbd-client-24780.asok" pid=24780 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=64055 ouid=64055
May 23 10:06:50 var0tf1a-cmp3s40d2yl-hr qemu-system-x86_64: 2018-05-23 10:06:50.531159 7f5c1da45ac0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.nova-compute-ext.keyring: (13) Permission denied

Needs confirmation. Not sure how to test...

[Tilman Baumann (tilmanbaumann)]

I can fix the code with the directory permissions...

[Tilman Baumann (tilmanbaumann)]

Looks like the qemu processes below go group 'kvm'.
I will give the directory to that group

[Tilman Baumann (tilmanbaumann)]

Oh, I filed this big in the wrong project. Sorry about that.
Should, of course, be in https://launchpad.net/charm-nova-compute

[Tilman Baumann (tilmanbaumann)]

Filed in the wrong project

[Tilman Baumann (tilmanbaumann)]

https://bugs.launchpad.net/charm-cinder-ceph/+bug/1779676

[Tilman Baumann (tilmanbaumann)]

The Gerrit bot didn't post this here

https://review.openstack.org/#/c/580145/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/580145
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=dd1d2daeb9119ee93e54ef0b6fa09a9a4352b608
Submitter: Zuul
Branch: master

commit dd1d2daeb9119ee93e54ef0b6fa09a9a4352b608
Author: Tilman Baumann <email address hidden>
Date: Tue Jul 3 13:31:56 2018 +0200

    Fixing /var/run/ceph/ directory permissions

    In previous versions of the charm the directory /var/run/ceph/
    was created with access permissions only for root.
    Consequently libvirt had no access to the rbd admin socket.

    Prior installations are also fixed through update-charm.

    Change-Id: I7f8054a404de9910bc070b288e1df1ce8dcf754e
    Closes-Bug: #1779676