wait_booted() times out on systems with broken sysvinit compat

[Nick Rosbrook]

The wait_booted() function in lib/VirtSubproc.py uses `runlevel` to get the current SysV runlevel, and only considers the test bed booted if `runlevel` returns a valid number. This is fine in most cases, but causes the test bed to timeout if this SysVinit compatibility is broken for some reason. Since Ubuntu runs systemd, it would be better to also check e.g. if multi-user.target is active, since SysVinit programs/interfaces are only left around in systemd for compatibility.

I came across this in a PPA autopkgtest where privileged LXD containers broke systemd-tmpfiles-setup.service, which is responsible for creating /var/run/utmp, which is needed for utmpx functions (i.e. how runlevel is stored on the system). Even though the container was booted and running "OK", autopkgtest timed out waiting for the test bed to boot: https://autopkgtest.ubuntu.com/results/autopkgtest-lunar-enr0n-systemd/lunar/amd64/s/systemd/20221130_112430_66c56@/log.gz(look for the line '<VirtSubproc>: failure: timed out waiting for testbed to start; last runlevel "unknown').

I plan on providing a patch for this.

[Paride Legovini (paride)]

Hi, thanks for reporting this bug. A couple of comments:

1. I am not sure I agree we should skip over a test that breaks very basic sysv compatibility, specifically by making the system never reach a valid (= numeric) runlevel. That's what wait_booted() checks before checking systemd stuff, if systemd is available (see the await-boot.sh script). In other words I think that a package that prevents runlevel from returning a runlevel should be considered to be compromising boot, so a test failure looks correct to me.

2. The aim for the autopkgtest package is to not diverge from Debian. Ubuntu delta in the devel release in temporary. This means that that changes you're proposed should land upstream [1]. I strongly doubt Debian will switch away from using runlevel there. The autopkgtest tooling has some "if Ubuntu ..." specific code, but I think we should the very basic parts like wait_booted() in common with Debian.

[1] https://salsa.debian.org/ci-team/autopkgtest/

[Martin Pitt (pitti)]

For the record, there's a very high chance that I am to blame for that "runlevel" check. Back then that was simply the lowest common denominator between sysvinit, upstart, and systemd. These days I agree that `systemctl is-system-running` would be more appropriate.

[Brian Murray (brian-murray)]

One thing to keep in mind is that we are still running autopkgtests for Trusty (ESM!) and that does not have systemctl.

[Lukas Märdian (slyon)]

This is apparently related to Ubuntu's apparmor confinement of the LXC process. Running containers unconfined seems to help:

```
sudo apparmor_parser -R /etc/apparmor.d/usr.bin.lxc-start
sudo ln -s /etc/apparmor.d/usr.bin.lxc-start /etc/apparmor.d/disable/
echo "lxc.apparmor.profile = unconfined" | sudo tee -a /etc/lxc/default.conf
```

See: https://discourse.ubuntu.com/t/containers-lxc/11526 (Apparmor section)

[Paride Legovini (paride)]

Hi Lukas, did you actually intend to comment to this bug? I fail to see the link between runlevel(8) returning unknown and apparmor confinement.

[Nick Rosbrook (enr0n)]

Yeah, his comment is relevant. The reason I hit this bug is that systemd-tmpfiles-setup.service failed to start, and that service is responsible for creating /var/run/utmp on Ubuntu(hence utmpx functions cannot store/fetch runlevel). The systemd-tmpfiles-setup.service failure is LXD-specific, and Lukas' comment suggests a workaround that could make systemd-tmpfiles-setup.service run OK (apparently without removing the LoadCredential= entries).