When set use-multipath apparmor denies running /usr/sbin/multipathd

[Bartosz Woronicz]

There's missing configurations of apparmon so nova can run /usr/bin/multipathd
when use-multipath option is set to charm
and one runs openstack server add volume...

```
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
...

2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connector.py", line 251, in get_connector_properties
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server connector.get_connector_properties(
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/base.py", line 54, in get_connector_properties
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server linuxscsi.LinuxSCSI.is_multipath_running(
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/linuxscsi.py", line 172, in is_multipath_running
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server out, _err = execute(*cmd, run_as_root=True,
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/privileged/rootwrap.py", line 186, in execute
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server raise putils.ProcessExecutionError(
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: [Errno 13] Permission denied
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server Command: multipathd show status
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server Exit code: -
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server Stdout: None
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server Stderr: None
2021-04-22 12:52:57.696 2032868 ERROR oslo_messaging.rpc.server

```

dmesg:
```
[510617.168463] audit: type=1400 audit(1619095977.187:9276): apparmor="DENIED" operation="exec" profile="/usr/bin/nova-compute" name="/usr/sbin/multipathd" pid=2094222 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
[510617.168473] audit: type=1400 audit(1619095977.187:9277): apparmor="DENIED" operation="exec" profile="/usr/bin/nova-compute" name="/usr/sbin/multipathd" pid=2094222 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
```

The workaround for it is to set app armor profile to complain
$ aa-complain /usr/bin/nova-compute

Then attaching works as expected
```
2021-04-22 13:03:03.879 2032868 INFO nova.virt.libvirt.driver [req-45993c5f-53ba-4b63-b796-4bcfd2dae2d7 36344934288b44b98282cce9095c60ca b0027274fda94bdd9ffebab10d6d9bdc - 2fba3120cb5542ffb124b7296cf22e71 2fba3120cb5542ffb124b7296cf22e71] [instance: 3c00f4ae-0d56-480b-839d-a8cf02019b72] Ignoring supplied device name: /dev/vdb
2021-04-22 13:03:04.127 2032868 INFO nova.compute.manager [req-45993c5f-53ba-4b63-b796-4bcfd2dae2d7 36344934288b44b98282cce9095c60ca b0027274fda94bdd9ffebab10d6d9bdc - 2fba3120cb5542ffb124b7296cf22e71 2fba3120cb5542ffb124b7296cf22e71] [instance: 3c00f4ae-0d56-480b-839d-a8cf02019b72] Attaching volume 3ed8c5e8-cf02-4e5e-bff8-04c7af3c89b5 to /dev/vdb
2021-04-22 13:03:06.937 2032868 INFO os_brick.initiator.connectors.fibre_channel [-] Fibre Channel volume device not yet found. Will rescan & retry. Try number: 0.
2021-04-22 13:03:08.973 2032868 INFO os_brick.initiator.linuxscsi [req-45993c5f-53ba-4b63-b796-4bcfd2dae2d7 36344934288b44b98282cce9095c60ca b0027274fda94bdd9ffebab10d6d9bdc - 2fba3120cb5542ffb124b7296cf22e71 2fba3120cb5542ffb124b7296cf22e71] Find Multipath device file for volume WWN 360002ac0000000000000001a00015717
```

[Pedro Guimarães (pguimaraes)]

The dmesg DENIED:

[510617.168463] audit: type=1400 audit(1619095977.187:9276): apparmor="DENIED" operation="exec" profile="/usr/bin/nova-compute" name="/usr/sbin/multipathd" pid=2094222 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
[510617.168473] audit: type=1400 audit(1619095977.187:9277): apparmor="DENIED" operation="exec" profile="/usr/bin/nova-compute" name="/usr/sbin/multipathd" pid=2094222 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0

State that profile for nova-compute should be set for /usr/sbin, not /sbin/multipathd as seen on:
https://github.com/openstack/charm-nova-compute/blob/master/templates/usr.bin.nova-compute#L87

[Liam Young (gnuoy)]

It seems that as of Disco *1 /sbin is a symlink to /usr/sbin. So I'm guessing that from an apparmor pov it expands the symlink so /usr/sbin/multipathd should be in the policy file.

*1 https://lists.ubuntu.com/archives/ubuntu-devel-announce/2018-November/001253.html

[Bartosz Woronicz (mastier1)]

Oh my... you are right, Liam! I have not know it.
I see it even on my Focal-based desktop machine

mastier@graf:~$ ls -la /sbin
lrwxrwxrwx 1 root root 8 kwi 15 21:37 /sbin -> usr/sbin/
mastier@graf:~$ cat /etc/issue
Ubuntu 20.04.2 LTS \n \l

So the profiles actually must be updated.

[Liam Young (gnuoy)]

https://review.opendev.org/c/openstack/charm-nova-compute/+/788000

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/788000
Committed: https://opendev.org/openstack/charm-nova-compute/commit/120235f3590d3a1272061320a4f76a06f407ff34
Submitter: "Zuul (22348)"
Branch: master

commit 120235f3590d3a1272061320a4f76a06f407ff34
Author: Liam Young <email address hidden>
Date: Mon Apr 26 13:51:06 2021 +0000

    Add apparmor rule to support /usr/sbin

    It seems that as of Disco *1 /sbin is a symlink to /usr/sbin. This
    patch adds support for file in either location.

    *1 https://lists.ubuntu.com/archives/ubuntu-devel-announce/2018-November/001253.html

    Change-Id: I66fa27f3f5e29d83cfea0f1afb33374303ab4669
    Closes-Bug: #1925511

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/21.04
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/791109

[David Ames (thedac)]

Started the back port process [0] as we saw this in the wild specifically around iscsiadm:

2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 77, in wrapped
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server _emit_exception_notification(
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server raise value
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1456, in decorated_function
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 216, in decorated_function
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-13 20:40:42.051 1044576 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-13 20...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/791109
Committed: https://opendev.org/openstack/charm-nova-compute/commit/9ec1e1f5082440107e05a21cf4990362c57b5b3a
Submitter: "Zuul (22348)"
Branch: stable/21.04

commit 9ec1e1f5082440107e05a21cf4990362c57b5b3a
Author: Liam Young <email address hidden>
Date: Mon Apr 26 13:51:06 2021 +0000

    Add apparmor rule to support /usr/sbin

    It seems that as of Disco *1 /sbin is a symlink to /usr/sbin. This
    patch adds support for file in either location.

    *1 https://lists.ubuntu.com/archives/ubuntu-devel-announce/2018-November/001253.html

    Change-Id: I66fa27f3f5e29d83cfea0f1afb33374303ab4669
    Closes-Bug: #1925511
    (cherry picked from commit 120235f3590d3a1272061320a4f76a06f407ff34)

[Chris MacNaughton (chris.macnaughton)]

This change was release to the stable charms with the above review, so I'm marking this fix-released.