usr.bin.nova-compute apparmor profile is blocking nova-compute operations

[Marcin Wilk]

Env: focal/yoga CIS hardened env, 'libvirt-image-backend' charm config option left empty (default qcow2), fix for [2] applied.

Creating a VM using flavor with --ephemeral not empty fails with:
2022-12-13 13:57:16.435 4064024 DEBUG oslo_concurrency.processutils [req-2b02f9a8-c014-4932-8812-c3d5d73c05f1 566d015a12a848eaaa90882e0a321ed9 73a75802773d40b387b146c43970f3a2 - 663ae7365105439f8aede922311749
52 663ae7365105439f8aede92231174952] 'mkfs -t ext4 -F -L ephemeral0 /var/lib/nova/instances/_base/ephemeral_4000_40d1d2c' failed. Not Retrying. execute /usr/lib/python3/dist-packages/oslo_concurrency/processu
tils.py:473
2022-12-13 13:57:17.617 4064024 ERROR nova.compute.manager [req-2b02f9a8-c014-4932-8812-c3d5d73c05f1 566d015a12a848eaaa90882e0a321ed9 73a75802773d40b387b146c43970f3a2 - 663ae7365105439f8aede92231174952 663ae7
365105439f8aede92231174952] [instance: 6be06476-547f-4843-90cc-fc64b302b46d] Failed to build and run instance: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: mkfs -t ext4 -F -L ephemeral0 /var/lib/nova/instances/_base/ephemeral_4000_40d1d2c
Exit code: 1
Stdout: ''
Stderr: 'mkfs: failed to execute mkfs.ext4: Permission denied\n'

journalctl clearly shows that the Apparmor is blocking this operation:
Dec 13 13:57:16 satpac54s020020 audit[4067457]: AVC apparmor="DENIED" operation="exec" profile="/usr/bin/nova-compute" name="/usr/sbin/mke2fs" pid=4067457 comm="mkfs" requested_mask="x" denied_mask="x" fsuid=64060 ouid=0

Allowing '/usr/sbin/mke2fs' in the '/etc/apparmor.d/usr.bin.nova-compute' (end reloading the profiles) solves the problem.

In addition to that, scanning journalctl for the apparmor DENIED shows more actions denied for the nova-compute profiles:

Dec 12 12:42:43 satpac54s020020 audit[2264780]: AVC apparmor="DENIED" operation="open" profile="/usr/bin/nova-compute" name="/etc/ssh/ssh_config.d/" pid=2264780 comm="ssh" requested_mask="r" denied_mask="r" fsuid=64060 ouid=0
Dec 12 13:08:39 satpac54s020020 audit[2295009]: AVC apparmor="DENIED" operation="exec" profile="/usr/bin/nova-compute" name="/usr/sbin/blkid" pid=2295009 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Dec 12 13:08:39 satpac54s020020 audit[2295003]: AVC apparmor="DENIED" operation="mkdir" profile="/usr/bin/nova-compute" name="/etc/nvme/" pid=2295003 comm="privsep-helper" requested_mask="c" denied_mask="c" fsuid=0 ouid=0

So apparently there are other functionalities impacted.
The 'mke2fs' case has been already identified in [1]

Looks like the apparmor nova-compute profile need revision and/or updates.

[1] https://bugs.launchpad.net/charm-nova-compute/+bug/1960231/comments/1
[2] https://bugs.launchpad.net/charm-nova-compute/+bug/1992386

[Alex Kavanagh (ajkavanagh)]

Hi Marcin, is this bug essentially a duplicate of https://bugs.launchpad.net/charm-nova-compute/+bug/1960231, or is it subtly different? I wanted to check prior to setting it as a duplicate. It may be that the information in the bug would be a good comment on the potential dup?

[Marcin Wilk (wilkmarcin)]

Hi Alex,
This bug is related to lp1960231. But I had an impression that lp1960231 was opened to explicitly solve virt_mkfs config and mkfs.ext4 rule for nova-compute profile.
While this bug, apart from allowing mke2fs also raises a problem with other actions blocked for nova-compute by apparmor as seen in the audit log above.
Apart from the rule fix for mke2fs it would be good to further investigate what is impacted by these denies:
Dec 12 12:42:43 satpac54s020020 audit[2264780]: AVC apparmor="DENIED" operation="open" profile="/usr/bin/nova-compute" name="/etc/ssh/ssh_config.d/" pid=2264780 comm="ssh" requested_mask="r" denied_mask="r" fsuid=64060 ouid=0
Dec 12 13:08:39 satpac54s020020 audit[2295009]: AVC apparmor="DENIED" operation="exec" profile="/usr/bin/nova-compute" name="/usr/sbin/blkid" pid=2295009 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Dec 12 13:08:39 satpac54s020020 audit[2295003]: AVC apparmor="DENIED" operation="mkdir" profile="/usr/bin/nova-compute" name="/etc/nvme/" pid=2295003 comm="privsep-helper" requested_mask="c" denied_mask="c" fsuid=0 ouid=0

Hence I opened a separate bug report. If you think that all of the above could be handled in the lp1960231, feel free to mark it as duplicate.

Cheers,
Marcin

[Nobuto Murata (nobuto)]

nvme and blkid are for NVMeoF related.
https://bugs.launchpad.net/charm-nova-compute/+bug/1979812/comments/1

We need to eventually update the policy for NVMeoF use cases but it's not a blocker for usual cases if I'm not mistaken.

[Marcin Wilk (wilkmarcin)]

The bug related to the 'mke2fs' issue has been fixed:
https://bugs.launchpad.net/charm-nova-compute/+bug/2008391
https://opendev.org/openstack/charm-nova-compute/commit/7e3ead3389b3824ba2a9fb5467cf03869cd59fb2

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/902047