logrotate action "rotate" failed

[Robert Hooper]

Last night logrotate ran and came back with a error running the postrotate scripts:
error: error running shared postrotate script for '/var/log/nginx/*.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

The current postrotate script contains:
invoke-rc.d nginx rotate >/dev/null 2>&1

I ran this manually and got this error:
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service nginx rotate
The script you are attempting to invoke has been converted to an Upstart
job, but rotate is not supported for Upstart jobs.

It looks like rotate is no longer a command I can pass to nginx.

I have looked into possible alternatives but haven't tested them on my server:
[ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid`

Or using the following command from upstart:
reload JOB [KEY=VALUE]

I am running:
Ubuntu precise (12.04.5)
nginx-common 1.8.0-1+precise1

[Paul Pace (myfirstnameispaul)]

I have the same issue, but I have also observed that after logrotate fails, new access.log, access.log.1 and access.log.*.gz files are created. However, nginx will continue to write to the old acces.log file (i.e. access.log.1) until a configuration reload (service nginx reload, service nginx restart, system reboot, etc.).

This happens with both error and access log files.

[Robert Hooper (7ob)]

I have found the command `service nginx rotate` was running in the init.d script on another server:
start-stop-daemon --stop --signal USR1 --quiet --pidfile /run/nginx.pid --name nginx
Running it seems to fix my problem.

I have added this to the postrotate part of logrotate and commented out what was there before.
#invoke-rc.d nginx rotate >/dev/null 2>&1
start-stop-daemon --stop --signal USR1 --quiet --pidfile /run/nginx.pid --name nginx

[Paul Pace (myfirstnameispaul)]

Thanks, Robert, this has been working on my servers since I changed the logroate file the day you posted this.

[Ivan (ivanforward)]

For me works fine like this:
...
postrotate
  nginx -s reload
endscript
...

But I am confused why don't work the special nginx functions for this:
  nginx -s reopen
  /etc/init.d/nginx rotate

Is anybody who know?

[Paul Pace (myfirstnameispaul)]

I observed that with the same upgrade, Ubuntu reports differently on configuration reload.

Old reload dialogue:

# service nginx reload
 * Reloading nginx configuration nginx [ OK ]
#

Or:

# service nginx reload
 * Reloading nginx configuration nginx [ FAIL]
#

New reload dialog (pass or fail):

# service nginx reload
#

I don't know if this is related to any of the above problems, but it doesn't really seem to fall into the 'bug' category as the configuration does get reloaded, just without any confirmation dialog. I am already in the habit of using nginx -t for every configuration change. I have noticed that twice I have had to reboot Ubuntu to fully reload configurations, and it seems to be related to configuration changes with OpenSSL, but I haven't experimented enough to figure that aspect out.

[Michael Lustfield (michaellustfield)]

Would anyone be willing to test out the latest version of the init script to see if the issue has been resolved there?
http://anonscm.debian.org/cgit/collab-maint/nginx.git/plain/debian/nginx-common.nginx.init

[Andrew Grasso (andrew-grasso)]

I'm having the same issue on Ubuntu 12.04.5 running Nginx 1.8.0-1+precise1

I believe the issue is caused by by the change in Nginx 1.8.0-1+precise from SystemV init.d/ control to Upstart. The /etc/init.d/nginx script was changed to reflect this, but the /etc/logrotate.d/nginx script was not. The logrotate script still references the now removed "rotate" function in /etc/init.d/nginx.

@Ivan- "nginx -s reload" works because you are using the nginx binary directly to signal the reopen. "/etc/init.d/nginx rotate" does not work, because the "rotate" function was removed from the /etc/init.d/nginx script when switching to upstart.

I think the version of the init script that Michael Lustfield's references would resolve the issue, but it it doesn't seem in line with the direction of moving to Upstart. I think a better fix would be to modify the logrotate config deployed with this package to signal nginx to re-open the log files without depending on the init.d script at all.

--- nginx 2015-06-22 18:18:46.622051252 +0000
+++ nginx.new 2015-06-22 18:22:36.800340062 +0000
@@ -13,6 +13,6 @@
                fi \
        endscript
        postrotate
- invoke-rc.d nginx rotate >/dev/null 2>&1
+ [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
        endscript
 }

Like was done here:
https://launchpadlibrarian.net/209458833/nginx_1.4.6-1ubuntu3_1.8.0-1-ppa7.diff.gz

[Niels Böhm (blubberdiblub)]

Replacing "invoke-rc.d" by "service" and "rotate" by "reload" in the logrotate config file solves the problem for us. The logs are now rotated and afterwards nginx is successfully made to write to the new log files and we don't get error mails from cron/logrotate.

--- nginx.orig 2015-04-07 23:07:11.000000000 +0000
+++ nginx 2015-07-09 07:45:45.057176379 +0000
@@ -13,6 +13,6 @@
   fi \
  endscript
  postrotate
- invoke-rc.d nginx rotate >/dev/null 2>&1
+ service nginx reload >/dev/null 2>&1
  endscript
 }

[Alex Muntada (alex.muntada)]

I confirm that using service instead of invoke-rd.d works fine. Thanks.

[Lior Goikhburg (goikhburg)]

Related: https://bugs.launchpad.net/nginx/+bug/1476296

[Paul Tobias (tobias.pal)]

See this link why it's better to send the `USR1` signal instead of doing reload (which sends the `HUP` signal): http://article.gmane.org/gmane.comp.web.nginx.english/583

"If the rotator sends the -HUP signal, then them master does a reconfiguration and starts a new worker processes those write to the new log files, but the old shuting down worker processes still uses the old log files."

Also see https://www.nginx.com/resources/wiki/start/topics/examples/logrotation/

[Bob (v-bob)]

I didn't see the behaviour with the latest stable release 1.8.1-1+trusty0 on Ubuntu Trusty but after an upgrade to the mainline release 1.9.12-0+trusty0

/etc/cron.daily/logrotate:
error: error running shared postrotate script for '/var/log/nginx/*.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

invoke-rc.d nginx rotate

initctl: invalid command: rotate
Try `initctl --help' for more information.
invoke-rc.d: initscript nginx, action "rotate" failed.

[hackel (hackel)]

Um, I came here specifically because I just experienced this error with nginx 1.8.1-1+trusty0. How about we wait for confirmation before changing things?

[hackel (hackel)]

Nevermind, I didn't realise I even had the PPA enabled.

[aptalca (thedjinn2)]

I had this fixed a while back by modifying the nginx logrotate config to use "service nginx rotate" and everything was dandy.

When I updated to 1.10.0 on the stable ppa (trusty), it asked me to revert the logrotate config to the package version and I said yes. Lo and behold, I am getting the same errors as before.

Either this bug was reintroduced or it wasn't really fully fixed.

Thanks

[Daniel Holth (dholth-gmail)]

I had the same issue. New /etc/logrotate.d/nginx from nginx-common calls invoke-rc.d nginx rotate which does not work. /usr/sbin/nginx -s reopen is what works.

[Thomas Ward (teward)]

This is leftover from the sync of packages; this will be fixed as soon as possible, have some patience please, I don't dedicate all my time to the PPAs.

[aptalca (thedjinn2)]

:) Apologies if it came off as being inpatient. It was earlier reported to no longer affect nginx. I just wanted to report that it still does ;) it was just an fyi

Thanks so much for maintaining this ppa

[mdotk (mdotk)]

Fixed! Thank you

"Replacing "invoke-rc.d" by "service" and "rotate" by "reload" in the logrotate config file solves the problem for us."

[Otto Kekäläinen (otto)]

Hello Thomas!

I am a Debian Developer using your ppa:nginx/stable. I experienced this same bug and I fixed it for myself. I would now like to fix it as upstream as possible but I cannot figure out where the git repository of this packaging is. The control file does not contain any git:// lines and https://launchpad.net/~nginx/+archive/ubuntu/stable/+packages does not give any hints.

Please give me a git url and I will fix this plus possible other issues while at in regarding the Debian/Ubuntu packaging. (I really hope that you keep your packaging files in a source control system somewhere :)

[Thomas Ward (teward)]

Otto: I'm replying to you via email.

------

Everyone else: This was once fixed in the PPAs eons ago, but packaging changes in Debian which needed incorporated took precedent and the changes were lost.

I need to know, however, which versions of Ubuntu are affected by this issue. If it's a global issue, it should be sent up to Debian; if it's specific to only a few Ubuntu versions, then that has its own workflow for me to fix.

[JuhaU (juhau1)]

I'm having this issue on Ubuntu 14.04 with nginx-extras 1.10.1-0+trusty0.

[Boaz Ackerman (boazin)]

+1 for ubuntu 14.04
The official ppa using
add-apt-repository ppa:nginx/$nginx -y
produces the error

[Thomas Ward (teward)]

It looks to me like this is a Precise and Trusty problem. Now that I know this, have patience while I prepare a fix.

[Thomas Ward (teward)]

Hello to all affected.

For Precise and Trusty, I have made a package revision that needs further testing before I push it to the main repositories. Please add the following PPA and test the latest nginx versions in that PPA (nginx 1.10.1-0+trusty1 for Trusty, and nginx 1.10.1-0+precise1 for Precise):

sudo add-apt-repository ppa:teward/nginx-temporary
sudo apt-get update
sudo apt-get upgrade

If the issue is resolved with these packages, please let me know, so I can push the update to the main repository for Stable releases. (Mainline will get the updates at a later time)

[Otto Kekäläinen (otto)]

Here is a quick fix for anybody who wants to deploy it immediately:
 sudo rm /etc/init.d/nginx

This removes the old init script so that is will not conflict with the new upstart script shipped in this PPA.

[Claudio Kuenzler (napsty)]

Seems this issue still exists (or again) in Xenial.

$ dpkg -l|grep logrotate | awk '{print $1" "$2" "$3}'
ii logrotate 3.8.7-2ubuntu2

$ dpkg -l|grep nginx | awk '{print $1" "$2" "$3}'
ii nginx 1.10.0-0ubuntu0.16.04.2
ii nginx-common 1.10.0-0ubuntu0.16.04.2
ii nginx-core 1.10.0-0ubuntu0.16.04.2

$ cat /etc/logrotate.d/nginx
/var/log/nginx/*.log {
 daily
 missingok
 rotate 14
 compress
 delaycompress
 notifempty
 create 0640 www-data adm
 sharedscripts
 prerotate
  if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
   run-parts /etc/logrotate.d/httpd-prerotate; \
  fi \
 endscript
 postrotate
  invoke-rc.d nginx rotate >/dev/null 2>&1
 endscript
}

[Claudio Kuenzler (napsty)]

I can confirm that log rotation works when the postrotate command is changed to a command which actually works. I changed it to "service nginx reload" yesterday and this morning nginx logs were rotated and nginx logs into the correct new log file (see access.log):

# ls -ltr /var/log/nginx/
total 6292
-rw-r--r-- 1 root root 184 Jun 6 08:49 error.log.3.gz
-rw-r--r-- 1 root root 671461 Jun 6 09:03 access.log.7.gz
-rw-r----- 1 www-data adm 459 Jun 6 09:39 error.log.2.gz
-rw-r--r-- 1 root root 293 Jun 6 10:55 monitoring.error.log.4.gz
-rw-r----- 1 www-data adm 15756 Jun 8 10:02 access.log.6.gz
-rw-r----- 1 www-data adm 1115 Jul 8 12:00 monitoring.error.log.3.gz
-rw-r----- 1 www-data adm 1335195 Jul 8 12:00 access.log.5.gz
-rw-r----- 1 www-data adm 2734 Jul 8 12:46 monitoring.error.log.2.gz
-rw-r----- 1 www-data adm 782316 Jul 26 16:11 access.log.4.gz
-rw-r----- 1 www-data adm 318759 Aug 4 15:05 access.log.3.gz
-rw-r----- 1 www-data adm 65 Aug 4 15:06 error.log.1
-rw-r----- 1 www-data adm 0 Aug 5 06:32 error.log
-rw-r----- 1 www-data adm 0 Aug 16 06:51 monitoring.error.log
-rw-r----- 1 www-data adm 9964 Sep 8 16:14 monitoring.error.log.1
-rw-r----- 1 www-data adm 2299587 Oct 10 13:01 access.log.2.gz
-rw-r----- 1 www-data adm 919857 Oct 11 06:50 access.log.1
-rw-r----- 1 www-data adm 45764 Oct 11 07:45 access.log

Nevertheless the file /etc/logrotate.d/nginx should be fixed from the package it comes from (nginx-common).

[冯宇 (abcfy2)]

I can confirm this issue on Ubuntu 14.04. I installed nginx-full package from ppa:nginx/stable.

And as the previous reports, invoke-rc.d nginx rotate will raise an error:

initctl: invalid command: rotate
Try `initctl --help' for more information.
invoke-rc.d: initscript nginx, action "rotate" failed.

But change to invoke-rc.d nginx reload will work very well.

This because nginx-common package contains "/etc/init.d/nginx, /etc/init/nginx.conf, /lib/systemd/system/nginx.service". For Ubuntu 14.04, which is used upstart for service manager, will use "/etc/init/nginx.conf" rather than "/etc/init.d/nginx.conf", so "service nginx rotate" will be failed.

[Guillaume (e1msih)]

Same issue here on many ubuntu 16.04 server, invoke-rc.d commands should be replaced.

[Carlo Lobrano (c-lobrano)]

Tried with Ubuntu 16.04.02 (desktop), and both invoke-rc.d and logrotate seems to work fine

root@ubuntu:~# ll -r /var/log/ | grep syslog
-rw-r----- 1 syslog adm 41472 mag 14 18:17 syslog.2.gz
-rw-r----- 1 syslog adm 385 mag 14 18:25 syslog.1
-rw-r----- 1 syslog adm 0 mag 14 18:25 syslog
-rw-r----- 1 syslog adm 112912 mag 14 17:28 kern.log.1
-rw-r----- 1 syslog adm 0 mag 14 18:18 kern.log
-rw-r----- 1 syslog adm 11033 mag 14 18:17 auth.log.1
-rw-r----- 1 syslog adm 0 mag 14 18:18 auth.log
drwxrwxr-x 12 root syslog 4096 mag 14 18:25 ./

root@ubuntu:~# date
dom 14 mag 2017, 18.24.34, CEST
root@ubuntu:~# ll -r /var/log/ | grep syslog
-rw-r----- 1 syslog adm 191099 mag 14 18:17 syslog.1
-rw-r----- 1 syslog adm 294 mag 14 18:18 syslog
-rw-r----- 1 syslog adm 112912 mag 14 17:28 kern.log.1
-rw-r----- 1 syslog adm 0 mag 14 18:18 kern.log
-rw-r----- 1 syslog adm 11033 mag 14 18:17 auth.log.1
-rw-r----- 1 syslog adm 0 mag 14 18:18 auth.log
drwxrwxr-x 12 root syslog 4096 mag 14 18:18 ./

root@ubuntu:~# invoke-rc.d rsyslog rotate
 * Closing open files rsyslogd [ OK ]

root@ubuntu:~# logrotate --force /etc/logrotate.conf
root@ubuntu:~# ll -r /var/log/ | grep syslog
-rw-r----- 1 syslog adm 41472 mag 14 18:17 syslog.2.gz
-rw-r----- 1 syslog adm 385 mag 14 18:25 syslog.1
-rw-r----- 1 syslog adm 0 mag 14 18:25 syslog
-rw-r----- 1 syslog adm 112912 mag 14 17:28 kern.log.1
-rw-r----- 1 syslog adm 0 mag 14 18:18 kern.log
-rw-r----- 1 syslog adm 11033 mag 14 18:17 auth.log.1
-rw-r----- 1 syslog adm 0 mag 14 18:18 auth.log
drwxrwxr-x 12 root syslog 4096 mag 14 18:25 ./

[Radu Pantiru (radupantiru)]

Package 1.12.2-0+trusty0 is affected by the bug.

        postrotate
                invoke-rc.d nginx rotate >/dev/null 2>&1
        endscript

#invoke-rc.d nginx rotate
initctl: invalid command: rotate
Try `initctl --help' for more information.
invoke-rc.d: initscript nginx, action "rotate" failed.

[Sergei Mikhaltsov (metanovii)]

nginx 1.20.1-1~focal amd64

# cat /etc/logrotate.d/nginx
/var/log/nginx/*.log {
        daily
        size 512M
        missingok
        rotate 1
        compress
        notifempty
        create 0640 www-data adm
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
                invoke-rc.d nginx rotate >/dev/null 2>&1
        endscript
}

# invoke-rc.d nginx rotate
Usage: /etc/init.d/nginx {start|stop|status|restart|reload|force-reload|upgrade|configtest|check-reload}
invoke-rc.d: initscript nginx, action "rotate" failed.