bad restriction: apparmor="DENIED" [...] namespace="root//lxd-n_<var-snap-lxd-common-lxd>" profile="rsyslogd" name="/run/systemd/journal/dev-log"

[Paride Legovini]

On my Questing system running LXD containers, my kernel log is full of messages like:

[ 129.551382] audit: type=1400 audit(1757925628.229:1005): apparmor="DENIED" operation="sendmsg" class="file" namespace="root//lxd-q_<var-snap-lxd-common-lxd>" profile="rsyslogd" name="/run/systemd/journal/dev-log" pid=5370 comm="systemd-journal" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000

One of my containers is named "q", hence the "root//lxd-q...". Some actual functionality is likely broken in the container.

[John Johansen (jjohansen)]

This is the rsyslogd profile in the container not allowing access to /run/systemd/journal/dev-log.

[Ryan Lee (rlee287)]

This looks like an issue with the rsyslogd profile, which is shipped as part of the rsyslog package on Ubuntu instead of the AppArmor package.

[Andreas Hasenack (ahasenack)]

UPDATE: I was mistaken before, and I didn't see it in my reproducer initially.

The questing LXD images seem to have fallen behind again, last uploaded on September 3rd, 2025.

My versions after the dist-upgrade:
ii apparmor 5.0.0~alpha1-0ubuntu6 amd64 user-space parser utility for AppArmor
ii rsyslog 8.2504.0-1ubuntu1 amd64 reliable system and kernel logging daemon
ii systemd 257.9-0ubuntu1 amd64 system and service manager

Was this a plain questing container, from September 3rd? Dist-upgraded or not? And no special workload, just lxc launch ubuntu-daily:questing?

[Andreas Hasenack (ahasenack)]

the rsyslog apparmor profile has a rule for /dev/log, which is a symlink:

lrwxrwxrwx 1 root root 28 Sep 15 16:39 /dev/log -> /run/systemd/journal/dev-log

But the base abstraction allows writing to that symlink's target:
/etc/apparmor.d/abstractions/base: @{run}/systemd/journal/dev-log w,

And the rsyslog profile includes the base abstraction.

[Andreas Hasenack (ahasenack)]

I reproduced it with the current questing lxd VM image, and launching a questing lxd inside that. I then dist-upgraded the host (the vm), and I don't see the denials anymore. Paride, can you please check if your questing host was also "old", and dist-upgrade it to be sure?

[Paride Legovini (paride)]

My host is a fully up-to-date Questing, and freshly rebooted. Guests are Noble and Questing containers.

Which package do you think could have made a difference in your case? The latest rsyslog in Questing migrated over one month ago.

[Andreas Hasenack (ahasenack)]

I don't know, systemd? Can you reproduce it with any new questing container on that host? Have you rebooted your host?

[Andreas Hasenack (ahasenack)]

I tried with a questing VM acting as the host, but I'm willing to install a bare metal host with questing today or tomorrow, given the upcoming beta.

[Paride Legovini (paride)]

I can reproduce the issue even after rebooting the host, and with new containers (i.e. I deleted the existing ones and recreated them: the issue immediately shows up.) Maybe the version of the lxd snap can play a role? I'm running 6.5-22da890 from latest/stable.

[Andreas Hasenack (ahasenack)]

> Maybe the version of the lxd snap can play a role? I'm running 6.5-22da890 from latest/stable.

Interesting, my questing VM (which is the lxd host for this test) installed lxd 5.21.4-8b5e998 (35624) from 5.21/stable/ubuntu-25.10

[Andreas Hasenack (ahasenack)]

But now a day later, launching a new questing container (and it downloaded a new image, btw), I see the issue, with the same 5.21.4 lxd:

[Tue Sep 16 13:44:06 2025] audit: type=1400 audit(1758030246.890:584): apparmor="DENIED" operation="sendmsg" class="file" namespace="root//lxd-q_<var-snap-lxd-common-lxd>" profile="rsyslogd" name="/run/systemd/journal/dev-log" pid=9412 comm="systemd-journal" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000
[Tue Sep 16 13:44:07 2025] audit: type=1400 audit(1758030247.977:585): apparmor="DENIED" operation="sendmsg" class="file" namespace="root//lxd-q_<var-snap-lxd-common-lxd>" profile="rsyslogd" name="/run/systemd/journal/dev-log" pid=9412 comm="systemd-journal" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000

[Andreas Hasenack (ahasenack)]

Several things on unpack in that DENIED message:

a) "sendmsg": something is trying to send a message. Given that "name" is a unix socket, it's presumably about writing to that socket.

b) rsyslogd profile is denying that operation

c) comm(and) is "systemd-journal". That doesn't make sense. rsyslogd would not exec systemd-journal: that is a daemon. rsyslogd does link with libsystemd, though, so perhaps the library is doing something?

d) denied_mask="r". What was denied was a read, not a write. That's weird. "sendmsg" implies a write, but a read was denied. The rsyslogd profile does allow a write to that dev-log socket (via the base abstraction), and not a read. But something tried to read from it. systemd-journal looks like, but why would that be confined by the rsyslog profile??

e) To check for (c), I ran execsnoop on the vm do see if I could catch an exec of systemd-journal, but no such exec happened.

So current theory is that somehow by virtue of linking with libsystemd, rsyslogd is trying to read from dev-log and that is being denied.

[Andreas Hasenack (ahasenack)]

I straced the rsyslogd process that is running inside the container while I was triggering the events that result in the DENIED message. strace() stayed put(!).

This is the current DENIED message that shows up in the questing host (where the questing lxd is running):

[Tue Sep 16 13:55:01 2025] audit: type=1400 audit(1758030901.984:1192): apparmor="DENIED" operation="sendmsg" class="file" namespace="root//lxd-q_<var-snap-lxd-common-lxd>" profile="rsyslogd" name="/run/systemd/journal/dev-log" pid=10991 comm="systemd-journal" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000

I logout and login (which is what triggers the DENIED messages), and that PID is always the same: 10991.

PID 10991 is the systemd-journald daemon from the CONTAINER, but as seen from the host:

  10919 ? Ss 0:00 [lxc monitor] /var/snap/lxd/common/lxd/containers q
  10926 ? Ss 0:00 \_ /sbin/init
  10991 ? Ss 0:00 \_ /usr/lib/systemd/systemd-journald

It shows up as confined like this (from the host):

lxd-q_</var/snap/lxd/common/lxd>//&:lxd-q_<var-snap-lxd-common-lxd>:unconfined (enforce) 1000000 10991 0.0 0.7 34524 14384 ? Ss 13:45 0:00 \_ /usr/lib/systemd/systemd-journald

So why would the rsyslog profile be the culprit for denying systemd-journald pid 10991 from reading dev-log?

When I strace that PID 10991 (systemd-journal from the guest lxd), I can see several denials of this form:

sendmsg(3, {msg_name={sa_family=AF_UNIX, sun_path="/run/systemd/journal/syslog"}, msg_namelen=30, msg_iov=[{iov_base="<38>", iov_len=4}, {iov_base="Sep 16 14:11:18 ", iov_len=16}, {iov_base="systemd-logind", iov_len=14}, {iov_base="[360]: ", iov_len=7}, {iov_base="Removed session c7.", iov_len=19}], msg_iovlen=5, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=360, uid=0, gid=0}}], msg_controllen=28, msg_flags=0}, MSG_NOSIGNAL) = -1 EACCES (Permission denied)

[Andreas Hasenack (ahasenack)]

This might still be something in apparmor itself. Reopening that task for evaluation.

[Seth Arnold (seth-arnold)]

One aspect of AppArmor IPC mediation is a "crosscheck" that requires a sending domain to have policy to allow sending and also requires the receiver to have policy to allow receiving. If either one fails, then the operation is failed as early as possible. (I'm not entirely sure how I would expect it to show up in the logs when they aren't in the same namespace, but this feels about what I would expect.)

Perhaps the Unix Domain Socket changes in newer versions of AppArmor require changes to the policy? I have a vague memory that previous versions of AppArmor allow file rules to give access to unix domain sockets in the filesystem but newer versions of AppArmor require explicit unix rules. (Worse yet, don't know what to add to the rsyslog policy to allow this access.)

[Andreas Hasenack (ahasenack)]

> Worse yet, don't know what to add to the rsyslog policy to allow this access.

I was stracing rsyslog, and it didn't get any DENIED error. But when I straced systemd-journald, the DAEMON, that's what was denied access. And by the rsyslog profile. This is what I cannot understand. It's another process. How can I add rules to the rsyslog profile about what another process should be able to do or not. There is something else going on here. systemd-journald is unconfined from the POV of the container, but it's definitely confined from the POV of the host. Maybe this is similar to LP: #2121552?

[Georgia Garcia (georgiag)]

Andreas, you are right that this is related to LP: #2121552. I'll try to explain:

This is a unix operation, and as Seth mentioned, we crosscheck -at the same time- if
1. sender is allowed to send to receiver
2. receiver is allowed to receive from sender

That's why unix rules have a peer component to them:
unix (receive) peer=(label=unconfined),

This rule is included in abstractions/base, so rsyslog is allowed to receive unix sockets from unconfined, which is the case here. There's another detail though: since this is a named unix socket, AppArmor also does a filesystem check using file rules. That's the denial we are seeing due to the bug in 2121552. Since systemd-journald (owner of the unix socket and unconfined) is running in the container in a (apparmor) virtualized stack, rsyslog is not being allowed delegation of the unconfined fd by default as expected.

This will require a kernel fix for a permanent solution.
Meanwhile,
/run/systemd/journal/dev-log r,
in rsyslog should allow it.

[Andreas Hasenack (ahasenack)]

After I add

/run/systemd/journal/dev-log r,

and reload the profile with apparmor_parser -r -W -T /etc/apparmor.d/usr.sbin.rsyslogd

Looks like I need to reboot the container. Restarting rsyslog or systemd-journal does't cut it.

[Andreas Hasenack (ahasenack)]

Uploaded to questing unapproved:

Uploading rsyslog_8.2504.0-1ubuntu2.dsc
Uploading rsyslog_8.2504.0-1ubuntu2.debian.tar.xz
Uploading rsyslog_8.2504.0-1ubuntu2_source.buildinfo
Uploading rsyslog_8.2504.0-1ubuntu2_source.changes

[Launchpad Janitor (janitor)]

This bug was fixed in the package rsyslog - 8.2504.0-1ubuntu2

---------------
rsyslog (8.2504.0-1ubuntu2) questing; urgency=medium

  * d/usr.sbin.rsyslogd: allow reading of systemd-journal's dev-log
    unix socket (LP: #2123821)

 -- Andreas Hasenack <email address hidden> Wed, 17 Sep 2025 17:14:42 -0300

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.