Services fail to start in noble deployed with TPM+FDE

[James Paton-Smith]

What's known so far:
- 24.04 desktop deployed with TPM+FDE shows this bug
- services confined with apparmor that need to access something in /run/systemd (like the notify socket) fail to do so, even if the apparmor profile is in complain mode. And the apparmor profile does already have rules to allow that access
- only after running aa-disable <path> can the service start fine
- paths logged by the apparmor DENIED or ALLOWED messages are missing the "/run" prefix from "/run/systemd/......".
- When we add rules to the profile using "/systemd/...." (i.e., also dropping the /run prefix), then it works
- other access in /run/systemd/ are also blocked, but the most noticeable one is the notify mechanism
- comment #2 also states that azure CVM images are also impacted
- comment #4 has instructions on how to create such a VM locally with LXD vms

Original description follows:

This might be related to #2064088

The rsyslog service is continually timing out and restarting. If I use a service drop-in file and change the 'Type' from 'notify' to 'simple', the service starts and appears to work normally.

In the journal, I can see the attached apparmor errors. I can't make sense of them, but if it's a similar issue to #2064088, then I suspect apparmor is preventing the systemd notify function from alerting systemd that the service is up and running.

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: rsyslog 8.2312.0-3ubuntu9
ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1
Uname: Linux 6.8.0-31-generic x86_64
ApportVersion: 2.28.1-0ubuntu2
Architecture: amd64
CasperMD5CheckMismatches: ./boot/grub/grub.cfg
CasperMD5CheckResult: fail
CurrentDesktop: ubuntu:GNOME
Date: Mon Apr 29 10:37:46 2024
ProcEnviron:
 LANG=en_GB.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
SourcePackage: rsyslog
UpgradeStatus: No upgrade log present (probably fresh install)

[Gauthier Jolly (gjolly)]

Azure CVM images are impacted by the same issue. I see on #2064088 that this is an tpm-backed FDE system. So I think it's the same problem here if those desktop images use an systemd-based initramfs.

For now I suspect that the issue is due to systemd starting services and setting up UNIX sockets (eg /run/systemd/journal/dev-log, /run/systemd/notify and others) before doing the pivot_root and reexec. Then, when apparmor tries to resolve the path of the peer socket it fails here[1] I believe.

[1] https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble/tree/fs/d_path.c#n125

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Simon Déziel (sdeziel)]

Here's how to reproduce this in a LXD VM:

Download Ubuntu 24.04 Desktop image into ~/Downloads

Import the ISO
$ lxc storage volume import default ~/Downloads/ubuntu-24.04-desktop-amd64.iso 24.04-desktop --type=iso

Prepare a LXD VM
$ lxc init --empty --vm lxd-noble-fde -c limits.memory=6GiB -c limits.cpu=4 -d root,size=32GiB
$ lxc config device add lxd-noble-fde iso-volume disk pool=default source=24.04-desktop boot.priority=10
$ lxc config device add lxd-noble-fde tpm tpm
$ lxc start --console=vga lxd-noble-fde

Go through the manual install process but at the "Disk Setup" step, select "Erase disk and install Ubuntu" and click "Advanced features...".
Select "Enable hardware-backed full disk encryption" then click "OK"

Once the installation is done, force the LXD VM to stop
$ lxc stop --force lxd-noble-fde

Remove the ISO
$ lxc config device remove lxd-noble-fde iso-volume

Start the VM back
$ lxc start lxd-noble-fde

Once logged in, rsyslog should eventually fail to start and the same Apparmor denials should show up in `journalctl -k`.

[James Paton-Smith (jamesps)]

I've just found that the cups.service is also experiencing the same behaviour. Again it has the service type 'notify'.

I suspect other services using this type will have the same problem.

[Andreas Hasenack (ahasenack)]

So far it's rsyslog, cups, and sssd (just confirmed via https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/2064088).

[Sergio Durigan Junior (sergiodj)]

Andreas and I spent some time this afternoon investigating this issue. Here are our findings.

First, we noticed that the paths being reported by apparmor on dmesg appear to be relative to /run. This is just an impression, though: I believe that, for some reason, apparmor/systemd/something-else is actually seeing the paths as "/systemd/notify" instead of "/run/systemd/notify". Therefore, we decided to try to list those paths inside the apparmor profile, like:

  /systemd/journal/dev-log rwkl,
  /systemd/notify rwkl,

Note that we're using "rwkl" just because we don't want to deal with limiting the scope of each access.

After adding these paths to /etc/apparmor.d/usr.sbin.rsyslogd and reloading the profile, the service can finally be (re)started. This indicates that there's a discrepancy between the paths seeing by apparmor/systemd/Linux and those seeing by the userspace application.

With that in mind, our next idea was to try to use "systemd-run" to mimic what's happening with rsyslogd. This could help us determine which component is problematic, but unfortunately we were unable to make the failure happen. We tried many combinations of commands; some of them are listed below:

# Try to "ls" the notify socket using different paths
systemd-run -p AppArmorProfile=rsyslogd ls /run/systemd/notify
systemd-run -p AppArmorProfile=rsyslogd ls /systemd/notify

# Likewise, but running the command using the syslog user
systemd-run --uid 102 -p AppArmorProfile=rsyslogd ls /run/systemd/notify
systemd-run --uid 102 -p AppArmorProfile=rsyslogd ls /systemd/notify

Strangely, "ls" was able to properly list the contents of /run/systemd/notify on both cases (which it shouldn't, because the apparmor profile doesn't allow it). It also reported that "/systemd/notify", which is correct but unexpected (because we thought that systemd might be the problematic component which doesn't use "/run" in the paths). We also double checked and confirmed that the processes started by "systemd-run" have "systemd" as their parent, so in theory we should have seen the same problem here.

There is also the fact that these file accesses are being denied even when the apparmor profile is running in complain mode. AFAIU, this shouldn't happen. Unless apparmor is affecting the path resolution that happens when the service tries to connect to the socket, effectively mangling the final path... but that would be very weird, I believe.

Either way, it is unclear:

1) Why we're seeing these "partial" paths in the logs.

2) Why these accesses are being denied even when the apparmor profile is in complain mode.

3) Why "systemd-run" can't seem to reproduce the problem.

[James Paton-Smith (jamesps)]

Do you think we should mark #2064088 as a duplicate of this (or vice-versa), if we're confident this is the same underlying issue?

There are some outstanding questions for me on that bug, but it might make sense to focus our comments in one place going forward.

[Andreas Hasenack (ahasenack)]

I think neither rsyslog, nor cups, or sssd, are the correct packages for this bug, but I agree that spreading out the investigation over two different bugs is not good.

I'll mark https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/2064088 as a duplicate, and add cups and sssd tasks to this one for tracking purposes, but I suspect the fix will be elsewhere in the end.

[Christian Ehrhardt  (paelzer)]

Thanks for the great debug work so far already, I think it is "apparmor or kernel" enough that we should add those packages and subscribe a few folks we know dealing with those details - I'd start with jjohansen as he'd be the best to map us to either knowledge or a known case.

[James Paton-Smith (jamesps)]

Answering questions from #2064088

Q: Did you install this fde/tpm setup using the ubuntu desktop noble installer? Or was hit some manual setup?
A: The install was performed using autoinstall with the desktop ISO. Attached is a copy of the autoinstall yaml

[James Paton-Smith (jamesps)]

Answering questions from #2064088

Q: Can you also show the output of: ps fauxwZ

A: See attached

[Andreas Hasenack (ahasenack)]

> /usr/lib/systemd/systemd --switched-root --system --deserialize=40 splash

Ok, --switched-root is there in PID 1, it's something I was looking to confirm.

[John Johansen (jjohansen)]

Does the profile have the attach_disconnected flag set?

Does the profile have the attach_disconnected flag set while in complain mode?

It looks to me that we are looking at open file descriptors that exist out of the current namespace. This will result in a partial unattached path that will not be allowed in complain mode. The denied path will not start with /.

If the attach_disconnected flag is add, that will attach the disconnected path to the root of the current mount namespace. Which is what I believe is happening with

  /systemd/...
vs
  /run/systemd/......".

Unless unconfined is involved, both the ends of a socket are required to exist in the namespace for v7/v8 unix socket mediation (what is in noble). Unconfined is special in that it can delegate access to an open fd which is not generically allowed atm.

If all the above is correct then you can use the attach_disconnected.path flag to attach the accesses to disconnected fds.

The full flags parameter to apparmor would then look like

  profile example flags=(attach_disonnected attach_disconnected.path=/run/) { ...)

and for complain mode

  profile example flags=(complain attach_disonnected attach_disconnected.path=/run/) { ...)

This of course is a less than satisfactory work around. There is work to address the above better but none of it is in noble.

[James Paton-Smith (jamesps)]

On my test VM I can see the cupsd profile DOES have 'attach_disconnected' flag, but not 'attach_disconnected.path=/run/'
If I add it and restart the cups.service, it starts successfully.

rsyslogd and sssd apparmor profiles do not have either these flags.

Could an apparmor abstraction work for this? i.e. if the system has 'boot-managed-by-snapd' package we could pull in an apparmor abstraction which adds these flags? Not sure what would be appropriate.

[John Johansen (jjohansen)]

Unfortunately there isn't a way to do this via abstractions or configs. It would be possible to add a patch to the userspace and SRU it. This would be the quickest solution while we work on the necessary kernel changes to make the use of attach_disconnected unnecessary.

[Gauthier Jolly (gjolly)]

To help with the investigations: I was able to reproduce the issue by simply installing dracut on a normal (non tpm-backed FDE) VM. Dracut replaces initramfs-tools and build a systemd-base initramfs.

# start the lxd VM
$ lxc launch --vm ubuntu:24.04 noble-vm

# in the VM install dracut and reboot
$ sudo apt update && sudo apt install -y dracut && sudo reboot

# check the logs after reboot
$ sudo journalctl -b0 | grep DENIED

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

We are seeing this issue with Ubuntu Core 24 too, for instance when running logger from a confined snap. The workaround of adding "attach_disonnected attach_disconnected.path=/run/" does work as well.

As an interesting data point, I have done some experiments and I see that using systemd 253.5-1ubuntu1 (an older version that was at some point in noble) with kernel 6.8 does not show the issue, while systemd 255.4-1ubuntu8 with kernel 6.8 triggers it. So some change in systemd triggers this.

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

(note that the version of systemd in my comment refers to the version in the UC initramfs)

[Nick Rosbrook (enr0n)]

Yes, I found that systemd switched from using MS_MOVE to MS_BIND | MS_REC when moving /run (and other filesystems) during the switch root. Although this is ultimately a shortcoming in AppArmor, this change in systemd is why we are seeing the issue now.

Discussing with upstream in https://github.com/systemd/systemd/pull/32645.