Lots of services of systemd 256 fail to start in nested LXD containers

[Jose Manuel Santamaria Lema]

Hi,

since 256 packages entered oracular-proposed I noticed they don't work in nested containers.

This is for me very easy to reproduce:
1. As LXD server I'm jammy and using the LXD 5.0 version from the snap.
2. I create a vm or a container
3. if it's a container set the security.nesting option as true
4. inside the container or vm created in 2. create another container (with "lxc launch ubuntu-daily:oracular test" for example)
5. I get into the container and 'apt dist-upgrade'
6. the system is broken, please see below how

The first noticeable thing is that right in the package configuration, we can see how the network and journal services failed:

Setting up systemd (256-1ubuntu1) ...
Installing new version of config file /etc/systemd/journald.conf ...
Installing new version of config file /etc/systemd/logind.conf ...
Installing new version of config file /etc/systemd/networkd.conf ...
Installing new version of config file /etc/systemd/sleep.conf ...
Installing new version of config file /etc/systemd/system.conf ...
/usr/lib/tmpfiles.d/legacy.conf:13: Duplicate line for path "/run/lock", ignoring.
Created symlink '/run/systemd/system/tmp.mount' → '/dev/null'.
/usr/lib/tmpfiles.d/legacy.conf:13: Duplicate line for path "/run/lock", ignoring.
Job for systemd-networkd.service failed because the control process exited with error code.
See "systemctl status systemd-networkd.service" and "journalctl -xeu systemd-networkd.service" for details.
Job for systemd-journald.service failed because the control process exited with error code.
See "systemctl status systemd-journald.service" and "journalctl -xeu systemd-journald.service" for details.

Then we can see the network service exited with 243/CREDENTIALS code:

root@test:~# systemctl status systemd-networkd
× systemd-networkd.service - Network Configuration
     Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; preset: enabled)\
     Active: failed (Result: exit-code) since Fri 2024-07-12 05:40:04 UTC; 5min ago
 Invocation: 00540f4884c44ec7a9f286942b8109a2
TriggeredBy: × systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)\
             man:org.freedesktop.network1(5)\
    Process: 455 ExecStart=/usr/lib/systemd/systemd-networkd (code=exited, status=243/CREDENTIALS)
   Main PID: 455 (code=exited, status=243/CREDENTIALS)
   FD Store: 0 (limit: 512)

Same for the journal service:

root@test:~# systemctl status systemd-journald.service
× systemd-journald.service - Journal Service
     Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static)\
    Drop-In: /usr/lib/systemd/system/systemd-journald.service.d
             └─nice.conf\
     Active: failed (Result: exit-code) since Fri 2024-07-12 05:31:39 UTC; 16min ago
 Invocation: 13bc72060e6c4d588869721d57fdba8a
TriggeredBy: × systemd-journald-dev-log.socket
             × systemd-journald.socket
             ○ systemd-journald-audit.socket
       Docs: man:systemd-journald.service(8)\
             man:journald.conf(5)\
    Process: 181 ExecStart=/usr/lib/systemd/systemd-journald (code=exited, status=243/CREDENTIALS)
   Main PID: 181 (code=exited, status=243/CREDENTIALS)
   FD Store: 0 (limit: 4224)

And, well, obviously the network doesn't work, neither the journal.

[Jose Manuel Santamaria Lema (panfaust)]

Another few things:

I tested the system package from here https://launchpad.net/~enr0n/+archive/ubuntu/systemd and it still has the same problem.

Executing /usr/lib/systemd/systemd-networkd just like that resurrects the network.

Commenting out certain things in /usr/lib/systemd/system/systemd-networkd.service make the network service work again - this is obviously not a solution, but I'm mentioning it in case it helps to debug the problem. The things you have to do with that file to "fix" the problem with -networkd are:
1. comment out the "ImportCredential=network.wireguard.*" line
2. comment out all the "Protect" lines
3. remove "systemd-networkd-persistent-storage.service" from "Wants="
4. systemctl daemon-reload
5. systemctl restart systemd-networkd

Also please note there's also many other services broken, network and journal are just the most 2 noticeable examples.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.