Installing many packages at once lets the nginx service fail to start

[Christian Ehrhardt]

Hi,
in our regular tests one does install a whole bunch of server programs to check various things like space, ports, ... - In that test I've started to see since 21st December a fail.

When installing in plucky nginx fails to start.

After digging through many red herrings (e.g. the collision with apache on port 80) I think this still is a real issue, yet I haven't found the lowest level of root cause.

The following will reproduce the issue:

$ lxc launch ubuntu-daily:p p-metric --ephemeral
$ lxc exec p-metric --env=DEBIAN_FRONTEND=noninteractive -- apt-get update -y
$ lxc exec p-metric --env=DEBIAN_FRONTEND=noninteractive -- apt-get -y install postgresql-all mysql-server libvirt-daemon-system containerd runc nfs-kernel-server samba slapd krb5-kdc sssd haproxy pacemaker memcached chrony nginx apache2 squid python3-django dovecot-imapd dovecot-pop3d postfix openvpn strongswan

The issue we can see in the log then is

06:05:04 Setting up nginx (1.26.0-3ubuntu1) ...
06:05:04 Job for nginx.service failed because the control process exited with error code.
06:05:04
06:05:04 See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.
06:05:04
06:05:04 invoke-rc.d: initscript nginx, action "start" failed.
06:05:04 × nginx.service - A high performance web server and a reverse proxy server
06:05:04 Loaded: loaded (]8;;file://metric-server-simple-plucky-vm-c1-m1/usr/lib/systemd/system/nginx.service\/usr/lib/systemd/system/nginx.service]8;;\; enabled; preset: enabled)
06:05:04 Active: failed (Result: exit-code) since Wed 2025-01-01 05:05:04 UTC; 11ms ago
06:05:04 Invocation: 7ccc5da4a95940d28cc0189485a6b4ec
06:05:04 Docs: ]8;;man:nginx(8)\man:nginx(8)]8;;\
06:05:04 Process: 11158 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=127)
06:05:04 Mem peak: 260K
06:05:04 CPU: 830us
06:05:04
06:05:04 Jan 01 05:05:04 metric-server-simple-plucky-vm-c1-m1 systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server...
06:05:04 Jan 01 05:05:04 metric-server-simple-plucky-vm-c1-m1 systemd[1]: nginx.service: Control process exited, code=exited, status=127/n/a
06:05:04 Jan 01 05:05:04 metric-server-simple-plucky-vm-c1-m1 systemd[1]: [0;1;38:5:185m[0;1;38:5:185mnginx.service: Failed with result 'exit-code'.
06:05:04 Jan 01 05:05:04 metric-server-simple-plucky-vm-c1-m1 systemd[1]: Failed to start nginx.service - A high performance web server and a reverse proxy server.
06:05:04 dpkg: error processing package nginx (--configure):
06:05:04 installed nginx package post-installation script subprocess returned error exit status 1

So we see the config pre-check failing "/usr/sbin/nginx -t -q -g daemon on; master_process on" but it isn't telling us how or why.

A simple "install apache2 + nginx" works fine.

I hope I'm rid of many red herrings that tried to lead me the wrong way and this really is us and will try to simplify the install command to less programs as it might help to understand why it happens.

[Christian Ehrhardt (paelzer)]

Checked further and reduced the set to install to trigger the issue.
And that was ... interesting, imagine a long log that I removed, here is what matters:

1. No other software is needed

I formerly reduced it just to libvirt+nginx, but that only mattered because its dependencies made it upgrade systemd and openssl. Based on the "The following packages will be upgraded:" output I could make it just

$ lxc exec p-metric --env=DEBIAN_FRONTEND=noninteractive -- apt-get -y --no-install-recommends install systemd nginx

Which will effectively do systemd+openssl+nginx:

The following NEW packages will be installed:
  nginx nginx-common openssl-provider-legacy
The following packages will be upgraded:
  libnss-systemd libpam-systemd libssl3t64 libsystemd-shared libsystemd0 libudev1 openssl systemd systemd-cryptsetup systemd-resolved systemd-sysv systemd-timesyncd udev

2. upgrade vs dist-upgrade before the install matter:

$ lxc exec p-metric --env=DEBIAN_FRONTEND=noninteractive -- apt-get update -y
$ lxc exec p-metric --env=DEBIAN_FRONTEND=noninteractive -- apt-get upgrade -y
$ lxc exec p-metric --env=DEBIAN_FRONTEND=noninteractive -- apt-get -y --no-install-recommends install systemd nginx

=> still fails

$ lxc exec p-metric --env=DEBIAN_FRONTEND=noninteractive -- apt-get update -y
$ lxc exec p-metric --env=DEBIAN_FRONTEND=noninteractive -- apt-get dist-upgrade -y
$ lxc exec p-metric --env=DEBIAN_FRONTEND=noninteractive -- apt-get -y --no-install-recommends install systemd nginx

=> works (the same updates were done before)

$ lxc exec p-metric --env=DEBIAN_FRONTEND=noninteractive -- apt-get update -y
$ lxc exec p-metric --env=DEBIAN_FRONTEND=noninteractive -- apt-get upgrade -y
$ lxc exec p-metric --env=DEBIAN_FRONTEND=noninteractive -- apt-get -y --no-install-recommends install nginx

=> works (not pulling in systemd/openssl)

But if I do the upgrade of ssl/systemd in one go with the nginx install it will fail.

[Christian Ehrhardt (paelzer)]

Opening a systemd task as that seems not so much an nginx issue, but a symptom in nginx due to systemd.
Also I assume once 257-2ubuntu1 is in the plucky images right away this issue will be gone.
But I'm not sure about noble/oracular -> plucky upgrades.

Asking for a systemd POV look at this please.

[Christian Ehrhardt (paelzer)]

By being out of proposed this does not immediately fail the tests anymore, but there is a risk that the same will hit e.g. O->P or N->R upgrades, worth a look if you can afford it?

[Simon Chopin (schopin)]

Is needrestart installed on your test container? If so, can you try with it removed?

[Nick Rosbrook (enr0n)]

I need to spend more time on this, but what I have observed so far is that the service failure is due to /usr/bin/nginx not being visible/executable. This is probably a sandboxing issue, but needs more investigation.

Before running the repro steps, I added:

# /etc/systemd/system/nginx.service.d/override.conf
[Service]
ExecStartPre=/usr/bin/true

and that failed as well, hence the thought about sandboxing or something.