nginx wont be started (needs network-online.target?)

[Thomas Schweikle]

Nginx is configured and will start if started manually by "systemctl start nginx"
But it wont start after (re)booting.

# systemctl enable nginx
# systemctl is-enabled nginx
enabled
# reboot

[...]

# systemctl is-enabled nginx
enabled
# systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Tue 2019-10-29 10:03:14 CET; 6s ago
     Docs: man:nginx(8)
  Process: 10406 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCC
  Process: 10026 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 10025 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 10027 (code=exited, status=0/SUCCESS)

Okt 29 09:55:23 nc158-muc systemd[1]: Starting A high performance web server and a reverse proxy server...
Okt 29 09:55:23 nc158-muc systemd[1]: Started A high performance web server and a reverse proxy server.
Okt 29 10:03:14 nc158-muc systemd[1]: Stopping A high performance web server and a reverse proxy server...
Okt 29 10:03:14 nc158-muc systemd[1]: Stopped A high performance web server and a reverse proxy server.
# pgrep nginx
#

# systemctl start nginx
# systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-10-29 10:04:30 CET; 8s ago
     Docs: man:nginx(8)
  Process: 10406 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCC
  Process: 10415 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 10414 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 10416 (nginx)
    Tasks: 5 (limit: 4915)
   CGroup: /system.slice/nginx.service
           ├─10416 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           ├─10417 nginx: worker process
           ├─10418 nginx: worker process
           ├─10419 nginx: worker process
           └─10420 nginx: worker process

Okt 29 10:04:30 nc158-muc systemd[1]: Starting A high performance web server and a reverse proxy server...
Okt 29 10:04:30 nc158-muc systemd[1]: Started A high performance web server and a reverse proxy server.
# pgrep nginx
10416
10417
10418
10419
10420
#

I'd awaited nginx to be started after network is up!

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: nginx 1.14.0-0ubuntu1.6
ProcVersionSignature: Ubuntu 4.15.0-67.76-generic 4.15.18
Uname: Linux 4.15.0-67-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.7
Architecture: amd64
Date: Tue Oct 29 09:56:19 2019
Dependencies:

InstallationDate: Installed on 2014-01-31 (2096 days ago)
InstallationMedia: Ubuntu-Server 13.10 "Saucy Salamander" - Release amd64 (20131016)
PackageArchitecture: all
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
SourcePackage: nginx
UpgradeStatus: Upgraded to bionic on 2018-11-23 (339 days ago)

[Bryce Harrington (bryce)]

It may be that your system has been upgraded from an earlier Ubuntu version and some of its config/service files have desync'd from correct defaults, but to check this we'll need to look at some of these files. To the best of my knowledge, just running `systemctl enable nginx` as you've done should be enough to get it starting on boot, so I'm guessing there's some other missing config.

Please attach the following to this bug report:

/lib/systemd/system/nginx.service
/etc/nginx/nginx.conf
Output of `dpkg -l | grep -i nginx`
Output of `sudo journalctl -u nginx`
Output of `sudo nginx -t`

I *think* that should provide enough info to figure out what's going on. If there are any other logs or config files you notice while collecting these that look relevant, please include them as well.

The fix is likely going to be some tweak to your nginx.service file. If you'd like to tinker directly, you might find LP: #1818574 of interest; it's a slightly different problem but probably a similar fix. There's some good systemd configuration guides on the web you can refer to for ideas. Otherwise, add the above files here and one of us will take a look.

[Christian Ehrhardt  (paelzer)]

From your log:
Okt 17 15:28:57 nc158-muc nginx[1296]: nginx: [emerg] bind() to 172.18.8.1:80 failed (99: Cannot assign requested address)
Okt 17 15:28:57 nc158-muc nginx[1296]: nginx: configuration file /etc/nginx/nginx.conf test failed
Okt 17 15:28:57 nc158-muc systemd[1]: nginx.service: Control process exited, code=exited status=1
Okt 17 15:28:57 nc158-muc systemd[1]: nginx.service: Failed with result 'exit-code'.

Maybe the server just starts to early.
Can you replace for a test
After=network.target
with
After=network-online.target
IIRC?

There is a whole category of such issues and it eventually comes down to user-config, but it might be worth to check if that dependency would make it work for you.
You can even wait for the device itself.

[Launchpad Janitor (janitor)]

[Expired for nginx (Ubuntu) because there has been no activity for 60 days.]

[Bryce Harrington (bryce)]

If this just needs network-online.target added, then perhaps this can be tested synthetically in lxc? Would be helpful to get original reporter to test this, but barring that the bug may be actionable (or at least upstreamable to Debian) at this point, if the issue is reproducible.

[Thomas Ward (teward)]

Bryce: If this is done by the original reporter, that's something they ahve to do in their local overrides. The server team has already NACK'd the request to change the target in the service file.

[Thomas Ward (teward)]

Bryce: Refer to https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1666368

The SystemD spec says daemons should rely on network.target, not network-online.target. Following this spec, we continue to ship the service file that is in the package as network.target, NOT network-online.target. As such, such changes aren't going to be package-level bugs and are going to need overrides at the local system level as was suggested.

[Thomas Ward (teward)]

Specifically, from https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ which explains this a bit better, under the `network-online.target` description:

> It is strongly recommended not to pull in this target too liberally: for example network server software should generally not pull this in (since server software generally is happy to accept local connections even before any routable network interface is up), its primary purpose is network client software that cannot operate without network.

... as you can see, the recommendation is to NOT have server software pull in network-online.target by default. As such, we're still following that, we won't be 'fixing' the target in the SystemD unit. If you *must* have nginx start up AFTER your networking is up, that's a configuration change you can apply with systemd overrides and snippets on the end systems themselves, but we won't provide that change as part of the package.