AppArmor does not permit access to rbd admin socket hardcoded in OpenStack charms

[Tilman Baumann]

The OpenStack charm nova-compute sets up rbd with hardcoded paths which libvirt has no access too when confined by AppArmor

The charm sets up
'admin socket': '/var/run/ceph/rbd-client-$pid.asok'
via
https://github.com/openstack/charm-nova-compute/blob/c744e052347d8ddfae88804a0ad0bdfdf4f5ae0d/hooks/nova_compute_context.py#L320

But libvirt has no exception for this path in the AppArmor profile.

Please add /run/ceph/rbd-client-*.asok rw, to /etc/apparmor.d/abstractions/libvirt-qemu to allow access to that file.

Log file excerpt:

May 23 10:06:38 var0tf1a-cmp3s40d2yl-hr nova-compute: 2018-05-23 10:06:38.972 55598 WARNING nova.compute.manager [req-40e3686c-d70b-4d0b-8e65-9b6ec1847903 - - - - -] [instance: c364f41a-a2df-40e5-be43-1e47dd4e4fd7] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4
May 23 10:06:46 var0tf1a-cmp3s40d2yl-hr /usr/share/filebeat/bin/filebeat[10378]: log.go:91: Harvester started for file: /var/log/upstart/nova-compute.log
May 23 10:06:46 var0tf1a-cmp3s40d2yl-hr /usr/share/filebeat/bin/filebeat[10378]: log.go:91: Harvester started for file: /var/log/nova/nova-compute.log
May 23 10:06:50 var0tf1a-cmp3s40d2yl-hr kernel: [10110228.305439] audit: type=1400 audit(1527070010.408:172758): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-c364f41a-a2df-40e5-be43-1e47dd4e4fd7" pid=24777 comm="apparmor_parser"
May 23 10:06:50 var0tf1a-cmp3s40d2yl-hr kernel: [10110228.305762] audit: type=1400 audit(1527070010.408:172759): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-c364f41a-a2df-40e5-be43-1e47dd4e4fd7//qemu_bridge_helper" pid=24777 comm="apparmor_parser"
May 23 10:06:50 var0tf1a-cmp3s40d2yl-hr qemu-system-x86_64: 2018-05-23 10:06:50.530151 7f5c1da45ac0 -1 asok(0x561ffd079ee0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/rbd-client-24780.asok': (13) Permission denied
May 23 10:06:50 var0tf1a-cmp3s40d2yl-hr kernel: [10110228.421988] audit: type=1400 audit(1527070010.524:172760): apparmor="DENIED" operation="mknod" profile="libvirt-c364f41a-a2df-40e5-be43-1e47dd4e4fd7" name="/run/ceph/rbd-client-24780.asok" pid=24780 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=64055 ouid=64055
May 23 10:06:50 var0tf1a-cmp3s40d2yl-hr qemu-system-x86_64: 2018-05-23 10:06:50.531159 7f5c1da45ac0 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.nova-compute-ext.keyring: (13) Permission denied

[Tilman Baumann (tilmanbaumann)]

Partner bug in Charm https://bugs.launchpad.net/charm-cinder-ceph/+bug/1779676

[Andreas Hasenack (ahasenack)]

What is the consequence of this access being denied? Is this admin socket something used by the charm, or admins logging in on the node?

[Tilman Baumann (tilmanbaumann)]

I'm still testing the full extent of the impact. It could be that just the admin socket isn't set up and the nasty error message.
Customer claims it's disabling the whole cache. But I'm not sure if that is actually true. And not quite sure how to test it TBH.

Best case, it's just ugly cosmetics.

[Christian Ehrhardt  (paelzer)]

Per our IRC discussion. It is a non default config being enabled by the charm that then makes it need that access.
As discussed the charm should in that case also add the apparmor line to allow the path that was added.

Setting to incomplete for now as the assumption on IRC was that the fix would have to be in the charm.

[Tilman Baumann (tilmanbaumann)]

Honestly, I would only modify package maintained files from a charm as the very last resort.
Using those paths is not only something strange in the charm. Anyone using ceph rbd would use those paths.

IMO, the question is, should a properly confined qemu allow rbd or not?

This has btw precedence.
/etc/apparmor.d/abstractions/libvirt-qemu
  # allow access to charm-specific ceph config and silence spurious
  # denials (LP: #1403648).
  /var/lib/charm/*/ceph.conf r,
  deny /tmp/{,**} r,
  deny /var/tmp/{,**} r,

This was even more charm specific.

[Christian Ehrhardt  (paelzer)]

Ok - other than I thought to remember our discussion, but that might only mean my vacation was good :-)
I'm fine adding:
  /run/ceph/rbd-client-*.asok rw,

Can you in advance verify that this would be sifficient by manually adding it?

[Tilman Baumann (tilmanbaumann)]

Ah, sounds like good holidays.
Yes, we tested that line and it worked.
The charm had additional issues with the way it created the directory, but that was just a distraction.

What I don't know 100% is the impact of not having this socket. Because in order to test if rbd works, I need that socket. (AFAIK)
Could be that rbd cache works and just not the admin socket.

I lost my test env. But I will try again to verify that. But nevertheless, it would be nice if it just worked.

There could be an argument made that the glob I chose is perhaps a little tight. But even all the instructions I saw online use that path pattern.

[Christian Ehrhardt  (paelzer)]

Tagged it up to be resolved on the Cosmic merge of Libvirt

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 4.6.0-2ubuntu1

---------------
libvirt (4.6.0-2ubuntu1) cosmic; urgency=medium

  * Merged with Debian unstable (LP: #1786957).
    Among many other new features and fixes this includes fixes
    for (LP: #1754871), Remaining changes:
    - Disable libssh2 support (universe dependency)
    - Disable firewalld support (universe dependency)
    - Set qemu-group to kvm (for compat with older ubuntu)
    - Additional apport package-hook
    - Autostart default bridged network (As upstream does, but not Debian).
      In addition to just enabling it our solution provides:
      + do not autostart if subnet is already taken (e.g. in guests).
      + iterate some alternative subnets before giving up
    - d/p/ubuntu/Allow-libvirt-group-to-access-the-socket.patch: This is
      the group based access to libvirt functions as it was used in Ubuntu
      for quite long.
      + d/p/ubuntu/daemon-augeas-fix-expected.patch fix some related tests
        due to the group access change.
      + d/libvirt-daemon-system.postinst: add users in sudo to the libvirt
        group.
    - ubuntu/parallel-shutdown.patch: set parallel shutdown by default.
    - d/p/ubuntu/enable-kvm-spice.patch: compat with older Ubuntu qemu/kvm
      which provided a separate kvm-spice.
    - Xen related
      - d/p/ubuntu/ubuntu-libxl-qemu-path.patch: this change was split. The
        section that adapts the path of the emulator to the Debian/Ubuntu
        packaging is kept.
      - d/p/ubuntu/ubuntu-libxl-Fix-up-VRAM-to-minimum-requirements.patch: auto
        set VRAM to minimum requirements
      - d/p/ubuntu/xen-default-uri.patch: set default URI on xen hosts
      - Add libxl log directory
      - libvirt-uri.sh: Automatically switch default libvirt URI for users on
        Xen dom0 via user profile (was missing on changelogs before)
    - d/p/ubuntu/apibuild-skip-libvirt-common.h: drop libvirt-common.h from
      included_files to avoid build failures due to duplicate definitions.
    - Update README.Debian with Ubuntu changes
    - Convert libvirt0, libnss_libvirt and libvirt-dev to multi-arch.
    - Enable some additional features on ppc64el and s390x (for arch parity)
      + systemtap, zfs, numa and numad on s390x.
      + systemtap on ppc64el.
    - d/t/control, d/t/smoke-qemu-session: fixup smoke-qemu-session by making
      vmlinuz available and accessible (Debian bug 848314)
    - d/t/control, d/t/smoke-lxc: fix up lxc smoke test isolation
    - Add dnsmasq configuration to work with system wide dnsmasq (drop >18.04,
      no more UCA onto Xenial then which has global dnsmasq by default).
    - d/p/ubuntu/ubuntu_machine_type.patch: accept ubuntu types as pci440fx
    - Further upstreamed apparmor Delta, especially any new one
      Our former delta is split into logical pieces and is either Ubuntu only
      or is part of a continuous upstreaming effort.
      Listing related remaining changes in debian/patches/ubuntu-aa/:
      + 0001-apparmor-Allow-pygrub-to-run-on-Debian-Ubuntu.patch: apparmor:
        Allow pygrub to run on Debian/Ubuntu
      + 0003-apparmor-libvirt-qemu-Allow-read-access-to-overcommi.patch:
        ...