OpenStack Nova Compute Charm

When attaching multiattach volumes apparmor nova-compute profile blocks some operations

Bug #2039161 reported by Giuseppe Petralia on 2023-10-12

This bug affects 2 people

	Status	Importance	Assigned to
OpenStack Nova Compute Charm	Fix Committed	Undecided	Felipe Reyes
2023.1	Fix Released	Undecided	Unassigned
2023.2	Fix Released	Undecided	Unassigned
Yoga	Fix Committed	Undecided	Unassigned
Zed	Fix Released	Undecided	Unassigned
nova-compute (Ubuntu)	Invalid	Undecided	Unassigned

Bug Description

This is happening in jammy nova-compute version 3:25.2.0-0ubuntu1

When attaching multiattach volumes the following warnings are logged:

2023-10-12 09:45:25.723 3906368 WARNING os_brick.initiator.connectors.nvmeof [req-9bfe2fce-9a78-4df3-8200-0e5901016e72 db682b8df0304e36b91345b7ce594aff 12b52497ff32492f888ae0ba837c2ae6 - 16b5db126dbb42fd804296790a3f9f6a 16b5db126dbb42fd804296790a3f9f6a] Process execution error in _get_host_uuid: [Errno 13] Permission denied
Command: blkid /dev/mapper/vg0-lvroot -s UUID -o value
Exit code: -
Stdout: None
Stderr: None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 13] Permission denied
2023-10-12 09:45:25.800 3906368 WARNING os_brick.initiator.connectors.nvmeof [req-9bfe2fce-9a78-4df3-8200-0e5901016e72 db682b8df0304e36b91345b7ce594aff 12b52497ff32492f888ae0ba837c2ae6 - 16b5db126dbb42fd804296790a3f9f6a 16b5db126dbb42fd804296790a3f9f6a] Unknown error when checking presence of nvme: [Errno 13] Permission denied: 'nvme': PermissionError: [Errno 13] Permission denied: 'nvme'
2023-10-12 09:45:25.806 958579 WARNING os_brick.privileged.nvmeof [-] Could not generate host nqn: [Errno 13] Permission denied: 'nvme'

and in syslog I can see:

Oct 12 09:13:59 machine1 kernel: [18324599.319817] audit: type=1400 audit(1697102039.363:696303): apparmor="DENIED" operation="exec" class="file" profile="/usr/bin/nova-compute" name="/usr/sbin/blkid" pid=4085668 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Oct 12 09:13:59 machine1 kernel: [18324599.319844] audit: type=1400 audit(1697102039.363:696304): apparmor="DENIED" operation="exec" class="file" profile="/usr/bin/nova-compute" name="/usr/sbin/blkid" pid=4085668 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Oct 12 09:13:59 machine1 kernel: [18324599.346662] audit: type=1400 audit(1697102039.387:696305): apparmor="DENIED" operation="exec" class="file" profile="/usr/bin/nova-compute" name="/usr/sbin/nvme" pid=4085671 comm="nova-compute" requested_mask="x" denied_mask="x" fsuid=64060 ouid=0
Oct 12 09:13:59 machine1 kernel: [18324599.346745] audit: type=1400 audit(1697102039.387:696306): apparmor="DENIED" operation="exec" class="file" profile="/usr/bin/nova-compute" name="/usr/sbin/nvme" pid=4085671 comm="nova-compute" requested_mask="x" denied_mask="x" fsuid=64060 ouid=0
Oct 12 09:13:59 machine1 kernel: [18324599.364823] audit: type=1400 audit(1697102039.407:696307): apparmor="DENIED" operation="open" class="file" profile="/usr/bin/nova-compute" name="/etc/nvme/hostnqn" pid=1343874 comm="nova-compute" requested_mask="r" denied_mask="r" fsuid=64060 ouid=0
Oct 12 09:13:59 machine1 kernel: [18324599.369262] audit: type=1400 audit(1697102039.411:696308): apparmor="DENIED" operation="exec" class="file" profile="/usr/bin/nova-compute" name="/usr/sbin/nvme" pid=4085672 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Oct 12 09:13:59 machine1 kernel: [18324599.369299] audit: type=1400 audit(1697102039.411:696309): apparmor="DENIED" operation="exec" class="file" profile="/usr/bin/nova-compute" name="/usr/sbin/nvme" pid=4085672 comm="privsep-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0

Warnings are gone if I set the apparmor profile to complain with:

aa-complain /etc/apparmor.d/usr.bin.nova-compute

See original description

Tags:

Giuseppe Petralia (peppepetra) on 2023-10-12

description:

updated

Giuseppe Petralia (peppepetra) on 2023-10-12

description:

updated

Revision history for this message

Nobuto Murata (nobuto) wrote on 2023-10-12:

Fwiw, when the last time I looked into, access denial for blkid and nvme wasn't in the critical path although it was a valid issue for NVMEoF use case.
https://bugs.launchpad.net/charm-nova-compute/+bug/1979812

Revision history for this message

Felipe Reyes (freyes) wrote on 2023-10-12:

Hi Giuseppe, can you check the version of the charms you are running?, because the bug Nobuto pointed out has been fixed and backported all the way back to Ussuri - https://review.opendev.org/q/message:1979812

I'm marking the "nova" package task as invalid since the apparmor is owned by the charm.

Changed in nova-compute (Ubuntu):
status:	New → Invalid
Changed in charm-nova-compute:
status:	New → Incomplete

Revision history for this message

Giuseppe Petralia (peppepetra) wrote on 2023-10-12 (last edit on 2023-10-12):

Hi Felipe, we are using nova-compute from yoga/stable rev. 664

rev. 664 has the update to apparmor from https://review.opendev.org/q/message:1979812

Checked on one machine

```
root@machine5:~# grep scsi/device /etc/apparmor.d/usr.bin.nova-compute
/sys/bus/scsi/devices/ r,

root@machine5:~# grep run/lock/nova /etc/apparmor.d/usr.bin.nova-compute
/run/lock/nova/* wk,

```

I checked latest yoga/stable and I can't see any update related to blkid

Checked with
```
mkdir nova-compute
juju download ch:nova-compute --channel yoga/stable
unzip nova-compute_02670ee.charm -d nova-compute
grep blkid nova-compute/templates/usr.bin.nova-compute
```

can you clarify if latest yoga/stable will allow the Denied in the description and if we need them to have multiattach to work?

Thanks

Revision history for this message

Giuseppe Petralia (peppepetra) wrote on 2023-10-12:

I did some tests and apparently even with the warnings multiattach volumes work as expected.

Felipe Reyes (freyes) on 2023-10-12

Changed in charm-nova-compute:
status:	Incomplete → New

Revision history for this message

Felipe Reyes (freyes) wrote on 2023-10-12:

it looks like os-brick has a fallback mechanism - https://opendev.org/openstack/os-brick/src/branch/master/os_brick/privileged/nvmeof.py#L33 - but I will enable these in the AA profile anyway.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-10-12: Fix proposed to charm-nova-compute (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/898127

Changed in charm-nova-compute:
status:	New → In Progress

Revision history for this message

DUFOUR Olivier (odufourc) wrote on 2023-11-28:

I updated the apparmor profile and was able to confirm on my lab that the issue has definitely disappeared with apparmor in enforced mode.

Edward Hope-Morley (hopem) on 2024-01-29

Changed in charm-nova-compute:
assignee:	nobody → Felipe Reyes (freyes)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-02-14: Fix merged to charm-nova-compute (master)

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/898127
Committed: https://opendev.org/openstack/charm-nova-compute/commit/0f9c730817b4f175e617ab5ce362bf9ff5157092
Submitter: "Zuul (22348)"
Branch: master

commit 0f9c730817b4f175e617ab5ce362bf9ff5157092
Author: Felipe Reyes <email address hidden>
Date: Thu Oct 12 11:50:30 2023 -0300

AppArmor policy update for NVMeoF

    When using NVMeoF feature with nova-compute apparmor in enforce
    mode, nova-compute is denied from running /usr/sbin/nvme and
    /usr/sbin/blkid, and reading /etc/nvme/hostnqn.

Change-Id: Ia23fbf341d5b7ad469337d8a0c65c18ec519a891
Closes-Bug: #2039161

Changed in charm-nova-compute:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-02-14: Fix proposed to charm-nova-compute (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/908936

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-05: Fix merged to charm-nova-compute (stable/2023.2)

#10

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/908936
Committed: https://opendev.org/openstack/charm-nova-compute/commit/557c47f37baa83e96f5618ae0a46a554897977b4
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 557c47f37baa83e96f5618ae0a46a554897977b4
Author: Felipe Reyes <email address hidden>
Date: Thu Oct 12 11:50:30 2023 -0300

AppArmor policy update for NVMeoF

    When using NVMeoF feature with nova-compute apparmor in enforce
    mode, nova-compute is denied from running /usr/sbin/nvme and
    /usr/sbin/blkid, and reading /etc/nvme/hostnqn.

    Change-Id: Ia23fbf341d5b7ad469337d8a0c65c18ec519a891
    Closes-Bug: #2039161
    (cherry picked from commit 0f9c730817b4f175e617ab5ce362bf9ff5157092)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-05: Fix proposed to charm-nova-compute (stable/2023.1)

#11

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/911066

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-06: Fix merged to charm-nova-compute (stable/2023.1)

#12

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/911066
Committed: https://opendev.org/openstack/charm-nova-compute/commit/c41b443548ecd91c16539924fb432dca8a6519b2
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit c41b443548ecd91c16539924fb432dca8a6519b2
Author: Felipe Reyes <email address hidden>
Date: Thu Oct 12 11:50:30 2023 -0300

AppArmor policy update for NVMeoF

    When using NVMeoF feature with nova-compute apparmor in enforce
    mode, nova-compute is denied from running /usr/sbin/nvme and
    /usr/sbin/blkid, and reading /etc/nvme/hostnqn.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-06: Fix proposed to charm-nova-compute (stable/zed)

#13

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/911601

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-07: Fix merged to charm-nova-compute (stable/zed)

#14

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/911601
Committed: https://opendev.org/openstack/charm-nova-compute/commit/721529036bc2baec8dd21ad3da86aeb453e01f2a
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 721529036bc2baec8dd21ad3da86aeb453e01f2a
Author: Felipe Reyes <email address hidden>
Date: Thu Oct 12 11:50:30 2023 -0300

AppArmor policy update for NVMeoF

    When using NVMeoF feature with nova-compute apparmor in enforce
    mode, nova-compute is denied from running /usr/sbin/nvme and
    /usr/sbin/blkid, and reading /etc/nvme/hostnqn.

    Change-Id: Ia23fbf341d5b7ad469337d8a0c65c18ec519a891
    Closes-Bug: #2039161
    (cherry picked from commit 0f9c730817b4f175e617ab5ce362bf9ff5157092)
    (cherry picked from commit 557c47f37baa83e96f5618ae0a46a554897977b4)
    (cherry picked from commit c41b443548ecd91c16539924fb432dca8a6519b2)

tags:

added: in-stable-zed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-07: Fix proposed to charm-nova-compute (stable/yoga)

#15

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-nova-compute/+/911610

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-19: Fix merged to charm-nova-compute (stable/yoga)

#16

Reviewed: https://review.opendev.org/c/openstack/charm-nova-compute/+/911610
Committed: https://opendev.org/openstack/charm-nova-compute/commit/3881433b2b7f45e6b899037a8f8f741faddc2fef
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 3881433b2b7f45e6b899037a8f8f741faddc2fef
Author: Felipe Reyes <email address hidden>
Date: Thu Oct 12 11:50:30 2023 -0300

AppArmor policy update for NVMeoF

    When using NVMeoF feature with nova-compute apparmor in enforce
    mode, nova-compute is denied from running /usr/sbin/nvme and
    /usr/sbin/blkid, and reading /etc/nvme/hostnqn.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.