Nginx cannot bind static IPv6 address on boot

Bug #1818574 reported by Vihai
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
nginx (Debian)
Fix Released
Unknown
nginx (Ubuntu)
Won't Fix
Low
Unassigned

Bug Description

Hello,

I have a Ubuntu cosmic box with static IPv4 and IPv6 addresses. It noticed that on boot nginx is not able to bind to the explicit IPv6 address. However, trying few seconds after boot completed it starts fine.

Mar 04 21:02:10 gems systemd[1]: Starting A high performance web server and a reverse proxy server...
Mar 04 21:02:10 gems nginx[640]: nginx: [emerg] bind() to [2a09:62c0:0:de::133]:80 failed (99: Cannot assign requested address)
Mar 04 21:02:10 gems nginx[640]: nginx: configuration file /etc/nginx/nginx.conf test failed

The network is configured with systemd-networkd and netplan.io

Apparently nginx is started before the network is fully stable.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Thanks for filing this bug in Ubuntu.

There are similar cases out there with other services, when a specific IP address is selected for the service to bind to. openssh, for example, has this bug upstream: https://bugzilla.mindrot.org/show_bug.cgi?id=2512

A few solutions, or workarounds, exist for this. The only one that seems to work for all use cases is to use the IP_FREEBIND socket option:
       IP_FREEBIND (since Linux 2.4)
              If enabled, this boolean option allows binding to an IP address that is nonlocal or does not (yet) exist. This permits listening on a socket, without requiring the underlying network inter‐
              face or the specified dynamic IP address to be up at the time that the application is trying to bind to it. This option is the per-socket equivalent of the ip_nonlocal_bind /proc interface
              described below.

That requires an upstream change, however.

In the meantime, users affected by this are probably better off by having nginx starting later in the boot process by adding a dependency to network-online.target. It will likely make the boot slower, though, that's why it's not a good solution to be applied generally.

You can try this perhaps:
sudo systemctl edit nginx.service

Then in the empty file that opens, type:
[Unit]
After=network-online.target
Requires=network-online.target

Then save. You can view your addition in the output of this command now:

sudo systemctl cat nginx.service

There appear to be more fine-grained ways of achieving this, like specifying which interface to wait for. https://www.freedesktop.org/software/systemd/man/systemd-networkd-wait-online.html documents something like that, but I haven't tried it.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I found this thread:
https://forum.nginx.org/read.php?29,248807,248807#msg-248807

Looks like something similar was proposed already, but not applied. This thread also had a discussion: http://mailman.nginx.org/pipermail/nginx-devel/2014-March/005138.html

Changed in nginx (Ubuntu):
status: New → Triaged
importance: Undecided → Low
Revision history for this message
Thomas Ward (teward) wrote :

Note that this type of "free binding" behavior is not desirable in MOST setups. Discussing in the Ubuntu Server IRC channel, in MOST setups, if you can't do something like bind to an IP it should error; slow-to-configure IPv6 aside, the most desirable thing would be for improper IP/bind configurations to error out in most default configurations.

------

Free binding and non-local binding behavior is, in my opinion, "Opt-In" behavior. I would not alter the SystemD file currently to use anything but network.target, as prior discussions on that matter with the Server Team have suggested that this would be improper, especially on local test systems where there is no 'networking' that would be online and in other edge cases.

------

There are several confirmed workarounds tested today by the Server Team to make sure they actually work, as well as *historically* have been suggested here already:

(1) Have nginx start up later in the cycle by overriding the SystemD unit to use network-online.target. This was suggested by Andreas, so look earlier in the bug comments for how to do this.

Execute: sudo systemctl edit nginx.service

Add this content:

[Unit]
After=network-online.target
Requires=network-online.target

------

(2) sysctl changes (Linux 4.3+ kernels)

Set net.ipv6.ip_nonlocal_bind which will implement "Free Binding" behavior like IP_FREEBIND has.

If you don't want to set this systemwide or persistently, or have a case where sysctl changes just don't persist (like in some containerization mechanisms), then...

------

(3) Set SystemD overrides for ExecStartPre for the service rather than changing your sysctl lines in the chance they don't persist.

Execute: sudo systemctl edit nginx.service

Add this content:

[Service]
ExecStartPre=-/sbin/sysctl -w net.ipv4.ip_nonlocal_bind=1
ExecStartPre=-/sbin/sysctl -w net.ipv6.ip_nonlocal_bind=1

This will do the same behavior.

------

Short of Upstream NGINX implementing IP_FREEBIND, or short of your manual insertion of overrides here, there's not much we can do to fix this bug in Ubuntu at this time...

Revision history for this message
Vihai (daniele-orlandi) wrote :

Thank you Andreas, thank you Thomas, the workaround to depend on network-online.target is fine for me.

I agree with Thomas that freebinding is not the appropriate way to cope with binding to address that should deterministically be present on start of the service.

You have surely made all the detailed evaluations on whether to depend on network.target or network-online.target however in my situation the "principle of least surprise" has somewhat been defeated. I hope you will find a way to prevent other people in analog situations to fall in the same pitfalls.

Changed in nginx (Debian):
status: Unknown → New
Thomas Ward (teward)
Changed in nginx (Ubuntu):
status: Triaged → Won't Fix
Changed in nginx (Debian):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.