NGINX fails to start/install/upgrade if IPv6 is completely disabled.

Bug #1743592 reported by Thomas Ward on 2018-01-16
24
This bug affects 2 people
Affects Status Importance Assigned to Milestone
nginx (Debian)
New
Unknown
nginx (Ubuntu)
Medium
Thomas Ward
Xenial
Medium
Unassigned
Bionic
Medium
Unassigned
Disco
Medium
Unassigned
Eoan
Medium
Unassigned

Bug Description

[IMPACT]

With current default vhost listening on IPV6, nginx won't start on fully IPV6 disabled system, expecting to connect to a IPV6 socket on port 80.

# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
==> nginx: [emerg] socket() [::]:80 failed (97: Address family not supported by protocol)
nginx: configuration file /etc/nginx/nginx.conf test failed

# cat /etc/nginx/sites-enabled/default
......
server {
 listen 80 default_server;
==> listen [::]:80 default_server;
......

[TEST CASE]
* Disable ipv6
  # cat /proc/cmdline
  .... ipv6.disable=1

* Reboot for the kernel parameter to be taken into account.

* Fresh install of nginx:
  # apt-get install nginx -y

Output:
--------

# apt-get install nginx -y
.......................
Setting up nginx-core (1.14.0-0ubuntu1.6) ...
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xe" for details.
invoke-rc.d: initscript nginx, action "start" failed.
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-10-23 14:01:26 UTC; 6ms ago
     Docs: man:nginx(8)
  Process: 1005 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)

Oct 23 14:01:26 nginxbionictest1 systemd[1]: Starting A high performance web server and a reverse proxy server...
Oct 23 14:01:26 nginxbionictest1 nginx[1005]: nginx: [emerg] socket() [::]:80 failed (97: Address family not supported by protocol)
Oct 23 14:01:26 nginxbionictest1 nginx[1005]: nginx: configuration file /etc/nginx/nginx.conf test failed
Oct 23 14:01:26 nginxbionictest1 systemd[1]: nginx.service: Control process exited, code=exited status=1
Oct 23 14:01:26 nginxbionictest1 systemd[1]: nginx.service: Failed with result 'exit-code'.
Oct 23 14:01:26 nginxbionictest1 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
dpkg: error processing package nginx-core (--configure):
 installed nginx-core package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of nginx:
 nginx depends on nginx-core (<< 1.14.0-0ubuntu1.6.1~) | nginx-full (<< 1.14.0-0ubuntu1.6.1~) | nginx-light (<< 1.14.0-0ubuntu1.6.1~) | nginx-extras (<< 1.14.0-0ubuntu1.6.1~); however:
  Package nginx-core is not configured yet.
  Package nginx-full is not installed.
  Package nginx-light is not installed.
  Package nginx-extras is not installed.
 nginx depends on nginx-core (>= 1.14.0-0ubuntu1.6) | nginx-full (>= 1.14.0-0ubuntu1.6) | nginx-light (>= 1.14.0-0ubuntu1.6) | nginx-extras (>= 1.14.0-0ubuntu1.6); however:
  Package nginx-core is not configured yet.
  Package nginx-full is not installed.
  Package nginx-light is not installed.
  Package nginx-extras is not installed.

dpkg: error processing package nginx (--configure):
 dependency problems - leaving unconfigured
Processing triggers for systemd (237-3ubuntu10.31) ...
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Errors were encountered while processing:
 nginx-core
 nginx
E: Sub-process /usr/bin/dpkg returned an error code (1)

# dpkg -l | grep -i nginx
iU nginx 1.14.0-0ubuntu1.6 all small, powerful, scalable web/proxy server
ii nginx-common 1.14.0-0ubuntu1.6 all small, powerful, scalable web/proxy server - common files
iF nginx-core 1.14.0-0ubuntu1.6 amd64 nginx web/proxy server (standard version)

--------

[REGRESSION POTENTIAL]

LOW.

Other distributions does that (not including ipv6 listen by default) already (e.g. Centos and Upstream)

* nginx.vh.default.conf -> "nginx-1.16.0-1.el8.ngx.src.rpm"
server {
    listen 80;
    server_name localhost;

* Upstream nginx:
https://github.com/nginx/nginx/blob/master/conf/nginx.conf#L36

This won't introduce behaviour change for those who upgrade nginx, but it will introduce a behaviour change for those used to have ipv6 listen in the default vhost at fresh package installation.

But IMHO, the default vhost is a starting point/example, that the admin will anyway have to modify manually or via some automation tool (ansible, chef, puppet, ...)

So this will only imply that the admin will need to update their recipe to add the ipv6 listen in their vhost config (ONLY if needed).

Minus leaving the users to have to uncomment the ipv6 listen in the vhost (if needed) by themselves manually, automation recipes, ... I don't see any major blocker, I personally think it's a good compromise in order to prevent the actual package to fails and be found in a erroneous state if ipv6 is disabled.

Of course, we can debate if disabling ipv6 is a good or wrong, but I think that the package installation should succeed for both ipv4only and ipv6.

[OTHER INFORMATION]

* Debian upstream bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942817

[ORIGINAL DESCRIPTION]
There is a known issue where NGINX will not properly run or install when IPv6 is fully disabled on a system.

Per Bug #1712696 which first detailed this, these are some 'relevant' log entries:

Aug 23 23:52:24 thomas-ThinkPad-T430 nginx[20166]: nginx: [emerg] socket() [::]:80 failed (97: Address family not supported by protocol)
Aug 23 23:52:24 thomas-ThinkPad-T430 nginx[20166]: nginx: configuration file /etc/nginx/nginx.conf test failed
Aug 23 23:52:24 thomas-ThinkPad-T430 systemd[1]: nginx.service: Control process exited, code=exited status=1
Aug 23 23:52:24 thomas-ThinkPad-T430 systemd[1]: Failed to start A high performance web server and a reverse proxy server.

As you can see the system could not bind to Port 80 on IPv6.

While this is a **nonstandard** setup, there are some systems which are configured to not have IPv6 support. It should currently be considered whether we should test for such 'systems' to see whether IPv6 is enabled or not, and modify the configuration file accordingly during install/configure.

This issue happens regardless of the 'supported' operating system status or not, and is a core NGINX package issue, but also a core 'incompatible configuration' issue with systems which disable IPv6.

Tags: sts Edit Tag help
Thomas Ward (teward) on 2018-01-16
description: updated
Thomas Ward (teward) on 2019-07-31
summary: - NGINX fails to install/upgrade if IPv6 is completely disabled.
+ NGINX fails to start/install/upgrade if IPv6 is completely disabled.
Eric Desrochers (slashd) wrote :
tags: added: sts
Changed in nginx (Debian):
status: Unknown → New
Eric Desrochers (slashd) wrote :

So IMHO there is 2 ways to fix this:

(1) - We remove the ipv6 listen from the vhost (just like Centos/Upstream does atm)

# nginx.vh.default.conf -> "nginx-1.16.0-1.el8.ngx.src.rpm"
server {
    listen 80;
    server_name localhost;

So as upstream nginx:
https://github.com/nginx/nginx/blob/master/conf/nginx.conf#L36

This won't introduce behaviour change for those who upgrade nginx, but it will introduce a behaviour change for those used to have ipv6 listen in the default vhost at fresh package installation.

But IMHO, the default vhost is a starting point/example, that the admin will anyway have to modify manually or via some orchestration tool (ansible, chef, puppet, ...)

So this will only imply that the admin will need to update their recipe to add the ipv6 listen in the vhost. I don't see any big blocker here.

(2) We implement a ipv6 detection as describe in the debian bug and provide a noipv6 vhost and/or a ipv6 vhost depending if ipv6 is [en|dis]abled on the system. But for me, that seems hacky.

I have a preference for scenario #1.

I have proposed both scenarios to debian, let's see what they say. In the absence of response from their part, I'll have SRU scenario #1 into Ubuntu.

I'll give debian maintainer team a week or so before.

- Eric

Changed in nginx (Ubuntu):
assignee: Thomas Ward (teward) → Eric Desrochers (slashd)
status: Triaged → In Progress
Eric Desrochers (slashd) on 2019-10-23
description: updated
description: updated
description: updated
Changed in nginx (Ubuntu Eoan):
assignee: nobody → Eric Desrochers (slashd)
Changed in nginx (Ubuntu Disco):
assignee: nobody → Eric Desrochers (slashd)
Changed in nginx (Ubuntu Bionic):
assignee: nobody → Eric Desrochers (slashd)
Changed in nginx (Ubuntu Xenial):
assignee: nobody → Eric Desrochers (slashd)
Eric Desrochers (slashd) on 2019-10-23
description: updated
Thomas Ward (teward) wrote :

I am in favor of solution #1, and will drive this for Focal. However, I have no call in whether this is accepted by the SRU team for SRU.

More to come though, I have a priority on the nginx package in Focal and determining which branch we track heading up to release first.

Eric Desrochers (slashd) wrote :

@teward, sound good I'll let you drive the focal upload since you were already in the middle of it and include solution #1 in it.

For the SRU, I'll take care of the discussion internally, and will keep the bug updated on the outcome.

Eric

Changed in nginx (Ubuntu):
assignee: Eric Desrochers (slashd) → Thomas Ward (teward)
Changed in nginx (Ubuntu Eoan):
status: New → In Progress
importance: Undecided → Medium
Changed in nginx (Ubuntu Disco):
status: New → In Progress
Changed in nginx (Ubuntu Bionic):
status: New → In Progress
importance: Undecided → Medium
Changed in nginx (Ubuntu Disco):
importance: Undecided → Medium
Eric Desrochers (slashd) on 2019-10-26
description: updated
description: updated
Eric Desrochers (slashd) on 2019-10-26
description: updated
Robie Basak (racb) wrote :

If I understand the situation correctly, my subjective opinion is that I don't think a "fix" for this is suitable for an SRU, but I welcome discussion if you disagree.

It's not uncommon to come across a situation where the user configures the system in some non-default way in one configuration file, and something else breaks without the user altering some other configuration file.

A couple of unrelated examples of this pattern:

1) A user configures a service to bind to a specific address, but then finds that the service fails to start because this introduces a dependency on having an interface with that address configured first. In this example the service hasn't been written to handle the interface coming up after service start, which would be the ideal solution to the problem. In the meantime, the user needs to also configure the init system to order the service start after the interface configuration.

2) A user configures a service to use a non-standard filesystem location, but then finds that AppArmor blocks access to that location. The user needs to also adjust the AppArmor profile to permit access to that location.

So here we have (I think):

3) A user disables IPv6. The user also needs to adjust the nginx configuration to not attempt to bind to IPv6 addresses.

In all of these cases, I agree that it would be nice if the appropriate thing could automatically detect the other configuration change and adjust itself accordingly by default. Doing this would certainly be a valuable usability improvement. However, I don't think these are bugs as such; this is expected behaviour albeit that the expected behaviour could be improved upon. While we might be able to make individual enhancements, in the general case it's clearly impossible to infer every possible necessary adjustment automatically. In the meantime, a user who adjusts away from default configuration in one place is expected to adjust other dependent configuration to match.

My conclusion is that this is a valid Wishlist-level bug in that in this specific case it would be nice if the default nginx configuration would continue to work without needing adjustment to match the global IPv6 disablement. However, for the reasons above I don't think an SRU is warranted, since the current behaviour is expected behaviour and doesn't prevent the user from achieving what they want. The currently correct way of handling this is to adjust nginx configuration, which is possible without needing any changes to the existing packaging.

I appreciate the efforts to make this experience better for users. That should take effect in future releases (but not change behaviour in existing stable releases). I therefore propose "Won't Fix" for the Ubuntu stable release bug tasks.

Thomas Ward (teward) on 2019-10-28
Changed in nginx (Ubuntu):
status: In Progress → Triaged
importance: Low → Medium
importance: Medium → Low
Eric Desrochers (slashd) on 2019-10-28
Changed in nginx (Ubuntu Eoan):
status: In Progress → Won't Fix
Changed in nginx (Ubuntu Disco):
status: In Progress → Won't Fix
Changed in nginx (Ubuntu Bionic):
status: In Progress → Won't Fix
Changed in nginx (Ubuntu Xenial):
status: New → Won't Fix
Eric Desrochers (slashd) on 2019-10-28
Changed in nginx (Ubuntu Xenial):
assignee: Eric Desrochers (slashd) → nobody
Changed in nginx (Ubuntu Bionic):
assignee: Eric Desrochers (slashd) → nobody
Changed in nginx (Ubuntu Eoan):
assignee: Eric Desrochers (slashd) → nobody
Changed in nginx (Ubuntu Disco):
assignee: Eric Desrochers (slashd) → nobody
Thomas Ward (teward) on 2019-11-01
Changed in nginx (Ubuntu):
status: Triaged → In Progress
importance: Low → Medium
Changed in nginx (Ubuntu Xenial):
importance: Undecided → Medium
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nginx - 1.17.5-0ubuntu1

---------------
nginx (1.17.5-0ubuntu1) focal; urgency=medium

  * New upstream release (1.17.5) - full changelog available from
    http://nginx.org/en/CHANGES
  * Remaining Ubuntu-specific changes:
    - debian/patches/ubuntu-branding.patch: add Ubuntu branding (refreshed)
    - d/{control,rules,nginx-core.*}: add new binary package for main,
      nginx-core, which contains only source-tarball-included modules
      and no third-party modules.
    - debian/tests/control: add nginx-core test.
    - debian/apport/source_nginx.py: Add apport hooks for additional bug
      information gathering.
    - debian/nginx-common.install: Add install rule for apport hooks.
    - d/nginx-{core,light,full,extras}.postinst: Add checks for whether
      port 80 is in use or not to determine whether or not to attempt
      starting of the NGINX service during install/upgrade
    - d/control: Add dependencies to nginx-{core,light,full,extras} on
      `iproute2` as the postinst scripts now use `ss` to determine if
      Port 80 is open or not.
    - d/rules: Enable --with-compat build option for all nginx package
      flavors
    - d/{control,rules,copyright,modules/http-geoip2*}: Add GeoIP2 third party
      module to nginx-full and nginx-extras (and use proper DEP5 syntax for
      d/copyright).
  * New Ubuntu-specific changes:
    - d/conf/sites-available/default: Update default nginx site configuration
      file to remove the IPv6 listening line so that servers running without
      IPv6 enabled at all on the system will start nginx properly.
      (LP: #1743592)

 -- Thomas Ward <email address hidden> Fri, 01 Nov 2019 11:55:10 -0400

Changed in nginx (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.