nginx.service: Failed to read PID from file /run/nginx.pid: Invalid argument

Bug #1581864 reported by Simon Déziel on 2016-05-14
26
This bug affects 5 people
Affects Status Importance Assigned to Milestone
nginx (Debian)
New
Undecided
Unassigned
nginx (Ubuntu)
Low
Thomas Ward

Bug Description

Nginx logs an error when started on a machine with a single CPU:

systemctl start nginx
systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2016-05-14 19:35:03 UTC; 2s ago
  Process: 1303 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
  Process: 1307 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 1306 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 1308 (nginx)
    Tasks: 5 (limit: 512)
   Memory: 3.1M
      CPU: 81ms
   CGroup: /system.slice/nginx.service
           ├─1308 nginx: master process /usr/sbin/nginx -g daemon on; master_process on
           ├─1309 nginx: worker process
           ├─1310 nginx: worker process
           ├─1311 nginx: worker process
           └─1312 nginx: worker process

May 14 19:35:03 ngx systemd[1]: Starting A high performance web server and a reverse proxy server...
May 14 19:35:03 ngx systemd[1]: nginx.service: Failed to read PID from file /run/nginx.pid: Invalid argument
May 14 19:35:03 ngx systemd[1]: Started A high performance web server and a reverse proxy server.

Bumping the number of CPU available makes the error disappear. This is reproducible on VMs and containers.

Lastly, the PID file is properly created and matches the PID of the master process.

Additional information:

# lsb_release -rd
Description: Ubuntu 16.04 LTS
Release: 16.04

# apt-cache policy nginx-core
nginx-core:
  Installed: 1.10.0-0ubuntu0.16.04.1
  Candidate: 1.10.0-0ubuntu0.16.04.1
  Version table:
 *** 1.10.0-0ubuntu0.16.04.1 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.9.15-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: nginx-core 1.10.0-0ubuntu0.16.04.1
ProcVersionSignature: Ubuntu 4.4.0-22.39-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
ApportVersion: 2.20.1-0ubuntu2
Architecture: amd64
Date: Sat May 14 19:35:21 2016
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
SourcePackage: nginx
UpgradeStatus: No upgrade log present (probably fresh install)

Simon Déziel (sdeziel) wrote :
Simon Déziel (sdeziel) wrote :

It seems to be a race between systemd and nginx. As if systemd was expecting the PID file to be populated before nginx had the time to create it.

Workaround:

 printf "[Service]\nExecStartPost=/bin/sleep 0.1\n" > /etc/systemd/system/nginx.service.d/override.conf
 systemctl daemon-reload

Simon Déziel (sdeziel) wrote :

In the above workaround, I forgot the first step (mkdir) so here it is again:

Workaround:

 mkdir /etc/systemd/system/nginx.service.d
 printf "[Service]\nExecStartPost=/bin/sleep 0.1\n" > /etc/systemd/system/nginx.service.d/override.conf
 systemctl daemon-reload

Thomas Ward (teward) wrote :

Confirmed on Xenial, with 1-CPU systems, that there may be such a race condition.

Given that the process still starts, I'm marking this as Low.

Changed in nginx (Ubuntu):
assignee: nobody → Thomas Ward (teward)
importance: Undecided → Low
status: New → Confirmed
Hristian (hristian-carabulea) wrote :

Oct 16 12:11:48 xxx systemd[1]: Starting A high performance web server and a reverse proxy server...
Oct 16 12:11:48 xxx systemd[1]: nginx.service: Failed to read PID from file /run/nginx.pid: Invalid argument
Oct 16 12:11:48 xxx systemd[1]: Started A high performance web server and a reverse proxy server.

A proposed solution did not work:

sudo mkdir /etc/systemd/system/nginx.service.d

printf "[Service]\nExecStartPost=/bin/sleep 0.1\n" > /etc/systemd/system/
-bash: /etc/systemd/system/: Is a directory

nginx.service.d/override.conf
-bash: nginx.service.d/override.conf: No such file or directory

Simon Déziel (sdeziel) wrote :

Hi Hristian, the string after the ">" char should be on a single line so that printf's output is directed to the file /etc/systemd/system/nginx.service.d/override.conf.

Alex Badics (alex.badics.bb) wrote :

Hi,

We are also affected, and I've looked into the source. In main(), the ngx_create_pidfile() gets called right after the ngx_daemon() function. ngx_daemon of course forks, and the parent process exits immediately with exit(0). This is a pure race condition between nginx and systemd.

This is not a problem (other than the scary log), because the PIDFile option is not needed for determining the main PID of nginx (systemd can correctly "guess" the main pid), it's only for removing the file when the service exits.

As far as I can tell, this daemonization is not needed for the systemd service use-case. The service should be Type=simple, and 'daemon off'. The standard file handles get redirected by systemd anyway, and non-stop upgrade cannot be used in this case either. (See: http://nginx.org/en/docs/faq/daemon_master_process_off.html )

If this is too much of a change, another workaround is removing the PIDFile option, and adding the line:
ExecStopPost=/bin/rm -f /run/nginx.pid

Regards,
Alex

Thomas Ward (teward) wrote :

FYI: If we don't have this being a confirmed issue in Debian, and we don't have a bug report up in Debian, ***please do not add a Debian task for this***. I just checked the Debian bug tracker and this is not reported up in Debian.

Has anyone confirmed this is also an issue in Debian?

John Navratil (john-navratil) wrote :

I'm seeing this on a Raspberry Pi Model 3 running Raspbian

uname -a:Linux www 4.9.24+ #993 Wed Apr 26 17:56:54 BST 2017 armv6l GNU/Linux

John Navratil (john-navratil) wrote :

Edit to previous comment. It's an R-Pi 2 B+

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers