Nginx 1.14.0 fails to start if ipv6 support on host is disabled

[Dave Bevan]

See bug report at https://trac.nginx.org/nginx/ticket/1821#comment:1

# uname -a
Linux bevand10-HP-EliteBook-840-G3 4.15.0-54-generic #58 Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

# nginx -V
nginx version: nginx/1.14.0 (Ubuntu) built with OpenSSL 1.1.1 11 Sep 2018 TLS SNI support enabled configure arguments: --with-cc opt='-g -O2 -fdebug-prefix-map=/build/nginx-pTuC1b/nginx-1.14.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module

# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500

  inet 192.168.249.1 netmask 255.255.255.0 broadcast 192.168.249.255
  ether 02:42:19:ba:e6:23 txqueuelen 0 (Ethernet)
  RX packets 0 bytes 0 (0.0 B)
  RX errors 0 dropped 0 overruns 0 frame 0
  TX packets 0 bytes 0 (0.0 B)
  TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

docker_gwbridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

  inet 192.168.250.1 netmask 255.255.255.0 broadcast 192.168.250.255
  ether 02:42:19:d7:3f:8e txqueuelen 0 (Ethernet)
  RX packets 0 bytes 0 (0.0 B)
  RX errors 0 dropped 0 overruns 0 frame 0
  TX packets 1033 bytes 172294 (172.2 KB)
  TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp0s31f6: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500

  ether c8:d3:ff:69:38:89 txqueuelen 1000 (Ethernet)
  RX packets 1334863 bytes 1058768699 (1.0 GB)
  RX errors 0 dropped 0 overruns 0 frame 0
  TX packets 380949 bytes 134078135 (134.0 MB)
  TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
  device interrupt 16 memory 0xe1200000-e1220000

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536

  inet 127.0.0.1 netmask 255.0.0.0
  loop txqueuelen 1000 (Local Loopback)
  RX packets 4019381 bytes 2054803911 (2.0 GB)
  RX errors 0 dropped 0 overruns 0 frame 0
  TX packets 4019381 bytes 2054803911 (2.0 GB)
  TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

veth3b18948: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

  ether ba:7a:7f:04:bc:2d txqueuelen 0 (Ethernet)
  RX packets 0 bytes 0 (0.0 B)
  RX errors 0 dropped 0 overruns 0 frame 0
  TX packets 1033 bytes 172294 (172.2 KB)
  TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

wlp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

  inet 10.56.110.15 netmask 255.255.252.0 broadcast 10.56.111.255
  ether f0:d5:bf:01:b7:4c txqueuelen 1000 (Ethernet)
  RX packets 17897545 bytes 13685388137 (13.6 GB)
  RX errors 0 dropped 0 overruns 0 frame 0
  TX packets 5316328 bytes 1206667801 (1.2 GB)
  TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: [emerg] socket() [::]:80 failed (97: Address family not supported by protocol)
nginx: configuration file /etc/nginx/nginx.conf test failed

Fragment from /etc/nginx/sites-enabled/default

  server {
    listen 80 default_server;
    listen [::]:80 default_server;

If I comment out the listen [::]:80 ipv6 directive, nginx starts successfully.

This appears to be a recent regression, as previously, absence of ipv6 support on the host would not result is a nginx start error.

For info, this occurred during a today's (2019-07-25) Ubuntu 18.04 upgrade:

nginx/bionic-updates,bionic-updates 1.14.0-0ubuntu1.3 all [upgradable from: 1.14.0-0ubuntu1.2]
nginx-common/bionic-updates,bionic-updates 1.14.0-0ubuntu1.3 all [upgradable from: 1.14.0-0ubuntu1.2]
nginx-core/bionic-updates 1.14.0-0ubuntu1.3 amd64 [upgradable from: 1.14.0-0ubuntu1.2]

The running nginx instance was halted because of the defect. What should have been a seamless upgrade was service affecting.

The NGINX team response:

Resolution set to invalid
Status changed from new to closed
You may want to report this to maintainers of the package you are using - by default nginx does not listen on IPv6 addresses.

[Andreas Hasenack (ahasenack)]

Thanks for filing this bug in Ubuntu.

I installed nginx 1.14.0-0ubuntu1 from the release pocket of Bionic, i.e., no updates:
root@bionic-nginx:~# apt-cache policy nginx
nginx:
  Installed: 1.14.0-0ubuntu1
  Candidate: 1.14.0-0ubuntu1
  Version table:
 *** 1.14.0-0ubuntu1 500
        500 http://br.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

And it already has the listen configuration for ipv6:
root@bionic-nginx:~# grep listen /etc/nginx/sites-enabled/default
 listen 80 default_server;
 listen [::]:80 default_server;
 # listen 443 ssl default_server;
 # listen [::]:443 ssl default_server;
# listen 80;
# listen [::]:80;

I then did a quick "git blame" on that file in the git repository, and that configuration has been there since at least 2014, if I'm reading this correctly:
10:01 $ git blame debian/conf/sites-available/default|grep listen
659cb474c (Kartik Mistry 2013-04-25 12:51:45 +0530 22) listen 80 default_server;
4f4fcce28 (Christos Trochalakis 2014-10-16 13:34:29 +0300 23) listen [::]:80 default_server;
4f4fcce28 (Christos Trochalakis 2014-10-16 13:34:29 +0300 27) # listen 443 ssl default_server;
4f4fcce28 (Christos Trochalakis 2014-10-16 13:34:29 +0300 28) # listen [::]:443 ssl default_server;
4f4fcce28 (Christos Trochalakis 2014-10-16 13:34:29 +0300 80) # listen 80;
4f4fcce28 (Christos Trochalakis 2014-10-16 13:34:29 +0300 81) # listen [::]:80;

Furthermore, the change introduced in 1.14.0-0ubuntu1.3 is just a rebuild with the newer openssl library that was updated in bionic:

nginx (1.14.0-0ubuntu1.3) bionic; urgency=medium

  * No changes rebuild (to build against OpenSSL 1.1.1 in Bionic)
    (LP: #1836366)

 -- Thomas Ward <email address hidden> Fri, 12 Jul 2019 14:18:43 -0400

If all of a sudden ipv6 was required in an update to a stable release, I'd agree it's surprising to say the least. Are you sure your server didn't get ipv6 disabled before this update, and it was only noticed when nginx was restarted? Has the server always had ipv6 disabled?

If it wasn't failing before, and started to fail now with just the rebuild with new openssl, that's something that has to be investigated, but I just want to clear up that point before we dig in, given the above details I have given.

Thanks!

[Andreas Hasenack (ahasenack)]

I just disbaled ipv6 on a plain bionic system, with the released package of nginx (no updates), and nginx fails to start with the same error, so this is not a regression introduced in a bionic update:

root@bionic-nginx:~# service nginx status
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2019-07-30 13:19:01 UTC; 16s ago
     Docs: man:nginx(8)
  Process: 700 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)

Jul 30 13:19:01 bionic-nginx systemd[1]: Starting A high performance web server and a reverse proxy server...
Jul 30 13:19:01 bionic-nginx nginx[700]: nginx: [emerg] socket() [::]:80 failed (97: Address family not supported by protocol)
Jul 30 13:19:01 bionic-nginx nginx[700]: nginx: configuration file /etc/nginx/nginx.conf test failed
Jul 30 13:19:01 bionic-nginx systemd[1]: nginx.service: Control process exited, code=exited status=1
Jul 30 13:19:01 bionic-nginx systemd[1]: nginx.service: Failed with result 'exit-code'.
Jul 30 13:19:01 bionic-nginx systemd[1]: Failed to start A high performance web server and a reverse proxy server.

root@bionic-nginx:~# apt-cache policy nginx
nginx:
  Installed: 1.14.0-0ubuntu1
  Candidate: 1.14.0-0ubuntu1
  Version table:
 *** 1.14.0-0ubuntu1 500
        500 http://br.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status

root@bionic-nginx:~# ifconfig
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 192.168.122.176 netmask 255.255.255.0 broadcast 192.168.122.255
        ether 52:54:00:0d:cf:78 txqueuelen 1000 (Ethernet)
        RX packets 301 bytes 27646 (27.6 KB)
        RX errors 0 dropped 40 overruns 0 frame 0
        TX packets 181 bytes 29753 (29.7 KB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        loop txqueuelen 1000 (Local Loopback)
        RX packets 92 bytes 6972 (6.9 KB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 92 bytes 6972 (6.9 KB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

root@bionic-nginx:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.0-55-generic root=UUID=ab05b2c3-a580-4df1-9ef9-0f0c5a183949 ro ipv6.disable=1 console=tty1 console=ttyS0

[Thomas Ward (teward)]

This has been reported before I have to find the bug but IPv6 disabled fully including link local is not a standard setup, I dont think an SRU will fix this since config files are usually not updated by an SRU. Even if we patch this the config file will need manually updated by end users to adjust this even after an update.

[Dave Bevan (dave-bevan)]

Hi.

It's a standard setup on our corporate LAN/WAN where our Information, Network and Data Security team mandate that IPv6 is disabled across the board.

[Andreas Hasenack (ahasenack)]

So to clarify, this is not a regression introduced in an SRU, this configuration has been part of nginx for years now in Ubuntu.

[Andreas Hasenack (ahasenack)]

@teward, feel free to triage further, my initial concern here was that this was a regression introduced in an SRU, but that doesn't seem to be the case.

[Thomas Ward (teward)]

ahasenack:

Yeah, no need to triage further, this alone is a Won't Fix for SRUing unless we need to push another fix out and include this and others along with it.

Users running into this issue can easily fix this by commenting out that line that's listening on v6.