snap.maas.supervisor DENIED

[Robert Tilley]

This appears that it may be related in some way to a previous issue Bug #1867571. I am constantly getting some version of the following error message. It isn't always /dev/sda but could be also ="/etc/gss/mech.d/" for example. Log attached.

Aug 13 20:57:49 utl01 kernel: [898463.471232] audit: type=1400 audit(1628888269.041:222847): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sda" pid=2245069 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

snap 2.51.3
snapd 2.51.3
series 16
ubuntu 20.04
kernel 5.4.0-81-generic

[Ian Johnson (anonymouse67)]

I'm not sure that this is a snapd bug, it depends on why MAAS is trying to access /dev/sda

[Alberto Donato (ack)]

Could you paste the output of `snap connections maas`?

[James Simpson (jsimpso)]

I've just noticed similar messages on a MAAS server:

"kernel: [8227300.867247] audit: type=1400 audit(1653363233.557:970746): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/vda" pid=1645448 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0"

"vda" is the root (and only) disk on this particular machine.

We're also seeing this for "/etc/gss/mech.d/":
jsimpso@maas:~$ grep 'apparmor="DENIED"' /var/log/syslog | cut -d ' ' -f 12,13 | sort | uniq
profile="snap.maas.supervisor" name="/dev/vda"
profile="snap.maas.supervisor" name="/etc/gss/mech.d/"

Here's the output of "snap connections maas" as previously requested:
Interface Plug Slot Notes
avahi-observe maas:avahi-observe :avahi-observe -
content[maas-cli] maas:maas-cli maas-cli:maas-cli -
content maas:test-db-socket - -
hardware-observe maas:hardware-observe :hardware-observe -
home maas:home :home -
kernel-module-observe maas:kernel-module-observe :kernel-module-observe -
mount-observe maas:mount-observe :mount-observe -
network maas:network :network -
network-bind maas:network-bind :network-bind -
network-control maas:network-control :network-control -
network-observe maas:network-observe :network-observe -
system-observe maas:system-observe :system-observe -
time-control maas:time-control :time-control -

[Alberto Donato (ack)]

The /dev/vda error is quite strange, as MAAS should be allowed to access block devices.

For other paths, it could be that apps used inside the maas snap (e.g. nginx, wget, ...) try to access default paths for configs, which are not allowed as they're outside of the snap. This is usually harmless as MAAS has its own configs.

Do you see any wrong behavior in MAAS that could be related to the error?

[Launchpad Janitor (janitor)]

[Expired for MAAS because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for snapd because there has been no activity for 60 days.]

[Junien F (axino)]

Hi,

I'm not seeing any wrong behaviour in MAAS, but the log spamming is quite bad :
https://pastebin.ubuntu.com/p/jwT9cHy2TT/

$ snap connections maas
Interface Plug Slot Notes
avahi-observe maas:avahi-observe :avahi-observe -
content[maas-cli] maas:maas-cli maas-cli:maas-cli -
content maas:test-db-socket - -
hardware-observe maas:hardware-observe :hardware-observe -
home maas:home :home -
kernel-module-observe maas:kernel-module-observe :kernel-module-observe -
mount-observe maas:mount-observe :mount-observe -
network maas:network :network -
network-bind maas:network-bind :network-bind -
network-control maas:network-control :network-control -
network-observe maas:network-observe :network-observe -
system-observe maas:system-observe :system-observe -
time-control maas:time-control :time-control -

[Christian Grabowski (cgrabowski)]

Hi there, can you please provide what version of MAAS you are using and what version of Ubuntu the machine running the snap is using?

[Junien F (axino)]

Sure ! MAAS snap 3.1.0-10901-g.f1f8f1505, Ubuntu 20.04, kernel 5.4.0-120-generic

[Jerzy Husakowski (jhusakowski)]

MAAS attempts retrieving information that is not available from inside of a snap. This leads to snapd log spam. MAAS should not attempt retrieving such information.

[Alan Baghumian (alanbach)]

I am running MAAS 3.3.4/3.3.5 and also do see these in the logs.

[Heather Lemon (hypothetical-lemon)]

Hello,

I ran a kprobe on apparmor.

# kprobe command
perf probe common_lsm_audit
# perf command & tail syslog to watch for warning messages from aa_
perf record -e probe:common_lsm_audit -a -g sleep 400

amd64 115533 [002] 83559.450382: probe:common_lsm_audit: (ffffffffb467d2f0)
        ffffffffb467d2f1 common_lsm_audit+0x1 ([kernel.kallsyms])
        ffffffffb46a3fa7 aa_audit_file+0x127 ([kernel.kallsyms])
        ffffffffb46a46b1 __aa_path_perm+0xa1 ([kernel.kallsyms])
        ffffffffb46a47a9 profile_path_perm.part.0+0x79 ([kernel.kallsyms])
        ffffffffb46a48ad aa_path_perm+0xdd ([kernel.kallsyms])
        ffffffffb46a1470 apparmor_file_open+0x1e0 ([kernel.kallsyms])
        ffffffffb465012b security_file_open+0x2b ([kernel.kallsyms])
        ffffffffb44cf5ef do_dentry_open+0xdf ([kernel.kallsyms])
        ffffffffb44d0fdd vfs_open+0x2d ([kernel.kallsyms])
        ffffffffb44e5d64 do_last+0x194 ([kernel.kallsyms])
        ffffffffb44e655d path_openat+0x8d ([kernel.kallsyms])
        ffffffffb44e7bd1 do_filp_open+0x91 ([kernel.kallsyms])
        ffffffffb44d12fe do_sys_open+0x17e ([kernel.kallsyms])
        ffffffffb44d1490 __x64_sys_openat+0x20 ([kernel.kallsyms])
        ffffffffb4204fe7 do_syscall_64+0x57 ([kernel.kallsyms])
        ffffffffb4e000a4 entry_SYSCALL_64_after_hwframe+0x5c ([kernel.kallsyms])
                  4abaea [unknown] (/snap/maas/33860/usr/share/maas/machine-resources/amd64)
                  4d14bb [unknown] (/snap/maas/33860/usr/share/maas/machine-resources/amd64)
                  4cffc5 [unknown] (/snap/maas/33860/usr/share/maas/machine-resources/amd64)
                  6a479a [unknown] (/snap/maas/33860/usr/share/maas/machine-resources/amd64)

# binary
/snap/maas/33860/usr/share/maas/machine-resources/amd64

# MAAS code
https://git.launchpad.net/maas/tree/src/host-info/Makefile?h=3.5
https://git.launchpad.net/maas/tree/src/host-info/pkg/info/info.go?h=3.5
func getResources() (*lxdapi.Resources, error) {
 return resources.GetResources()
}
https://github.com/casual-lemon/maas/blob/master/src/host-info/pkg/info/info.go#L167
https://github.com/canonical/lxd/blob/aab7941c0270a21586ed1923adf6ec4f5a0b8cf3/lxd/resources/resources.go#L10
https://github.com/canonical/lxd/blob/aab7941c0270a21586ed1923adf6ec4f5a0b8cf3/lxd/resources/storage.go#L133
// Detect all block devices
 if sysfsExists(sysClassBlock) {
  entries, err := os.ReadDir(sysClassBlock)

I followed the Maas source code, and it uses the LXD API to make a call to storage.go which tries to access any block devices on the host.

Cheers,
Heather Lemon

[Heather Lemon (hypothetical-lemon)]

Is there an option in LXD API to not scan all block devices?

[Anton Troyanov (troyanov)]

One possible solution might be just to add allowance rule to the AppArmor profile of `snap.maas.supervisor`
However, if you do this MAAS can potentially read all your data!

On your host machine you can edit `/var/lib/snapd/apparmor/profiles/snap.maas.supervisor`:
1. Navigate to the section `profile "snap.maas.supervisor" (attach_disconnected,mediate_deleted)`
2. Add the following line inside the scope `/dev/sda r,`

[Heather Lemon (hypothetical-lemon)]

Hi Anton,

Yes, I have tested the workaround and replacing the line in snap.maas.supervisor does work, but I don't think it's safe for the reasons you mentioned - MAAS reading data and #2 without this we see > 17,000 log lines in syslog from apparmor complaining about the access rights.

Thanks,
Heather Lemon

[Simon Déziel (sdeziel)]

On LXD, the needed Apparmor rules/snap interfaces allow reading block devices so that such errors are not logged when using the same package that MAAS uses. As such, maybe MAAS' snap is missing a plug to the `block-devices` interface https://snapcraft.io/docs/block-devices-interface?

Irrespective of that, I don't know why MAAS makes that check so frequently (every 30s according to the logs).

[Heather Lemon (hypothetical-lemon)]

Hi Simon, thanks for this.

I don't see a block-devices interface listed after a fresh maas-latest install.

sudo snap connections maas

Interface Plug Slot Notes
avahi-observe maas:avahi-observe :avahi-observe -
content - maas:maas-logs -
content[db-socket] maas:test-db-socket maas-test-db:db-socket -
hardware-observe maas:hardware-observe :hardware-observe -
home maas:home :home -
kernel-module-observe maas:kernel-module-observe :kernel-module-observe -
mount-observe maas:mount-observe :mount-observe -
network maas:network :network -
network-bind maas:network-bind :network-bind -
network-control maas:network-control :network-control -
network-observe maas:network-observe :network-observe -
snap-refresh-control maas:snap-refresh-control :snap-refresh-control -
system-observe maas:system-observe :system-observe -
time-control maas:time-control :time-control -

Do you have an example of this in the LXD code?
> the needed Apparmor rules/snap interfaces allow reading block devices so that such errors are not logged when using the same package that MAAS uses.

sudo snap connections lxd

Interface Plug Slot Notes
lxd - lxd:lxd -
lxd-support lxd:lxd-support :lxd-support -
network lxd:network :network -
network-bind lxd:network-bind :network-bind -
system-observe lxd:system-observe :system-observe -

Cheers,
Heather Lemon

[Heather Lemon (hypothetical-lemon)]

Fixed formatting here: https://pastebin.canonical.com/p/QH4zmXsXTr/

[Jerzy Husakowski (jhusakowski)]

MAAS checks for changes in interfaces, which needs to be picked up quickly, hence the 30s interval. LXD API for obtaining host information tries to retrieve comprehensive information about the host, which includes enumeration of block devices. This triggers the logspam from apparmor when this happens inside of a snap.

Snapd does offer a connector that would allow such enumeration without logspam - the user can connect `block-devices` plug and it should silence the apparmor notification, which is what LXD does (and it's not considered a security issue).
Alternatively, comment #15 shows how to update the apparmor rules on the host to silence the message.

The mechanism works as designed, and there are at least two methods to remove the unwanted apparmor warnings. Making the retrieval of host information aware of the environment it runs in and conditionally exclude accessing certain types of devices would be the wrong place to address the apparmor warnings and introduce unnecessary complexity.

[Junien F (axino)]

What if you don't want to allow MAAS to access your devices though ? You're stuck with spam in your logs ?