rsyslog stops working after logrotate until restarted

[agent 8131]

This could otherwise be titled, rsyslog reload does not create log files; only restart does.

This is happening on a number of machines I work on. It's happening on 10.04 and 11.04. It might be similar to:

https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/407862

But in my case after the restart there is no /var/log/syslog being created, nor auth.log, kern.log, etc. The files are rotated, rsyslog is reloaded, and none of the log files are created and nothing is being logged. This has been plaguing my systems since moving from syslog-ng, which I may return to as it seems it was actually production ready.

Without manually restarting those files don't exist so here's what I did on an 11.04 system:

logrotate --force --verbose /etc/logrotate.conf

gives:

rotating pattern: /var/log/syslog
 forced from command line (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/syslog
  log /var/log/syslog does not exist -- skipping
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
 forced from command line (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/mail.info
  log /var/log/mail.info does not exist -- skipping
considering log /var/log/mail.warn
  log /var/log/mail.warn does not exist -- skipping
considering log /var/log/mail.err
  log does not need rotating
considering log /var/log/mail.log
  log does not need rotating
considering log /var/log/daemon.log
  log /var/log/daemon.log does not exist -- skipping
considering log /var/log/kern.log
  log /var/log/kern.log does not exist -- skipping
considering log /var/log/auth.log
  log /var/log/auth.log does not exist -- skipping
considering log /var/log/user.log
  log /var/log/user.log does not exist -- skipping
considering log /var/log/lpr.log
  log /var/log/lpr.log does not exist -- skipping
considering log /var/log/cron.log
  log /var/log/cron.log does not exist -- skipping
considering log /var/log/debug
  log /var/log/debug does not exist -- skipping
considering log /var/log/messages
  log /var/log/messages does not exist -- skipping
not running postrotate script, since no logs were rotated

Then
/sbin/reload rsyslog
logger -i testing

At this point there is no /var/log/syslog

Then:
/sbin/restart rsyslog

And voila there is a /var/log/syslog beginning with:

Feb 23 19:24:48 somehost kernel: imklog 4.6.4, log source = /proc/kmsg started.
Feb 23 19:24:48 somehost rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="2299" x-info="http://www.rsyslog.com"] (re)start
Feb 23 19:24:48 somehost rsyslogd: rsyslogd's groupid changed to 114
Feb 23 19:24:48 somehost rsyslogd: rsyslogd's userid changed to 108

Then to recreate:

logrotate --force --verbose /etc/logrotate.conf

rotating pattern: /var/log/syslog
 forced from command line (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/syslog
  log needs rotating
rotating log /var/log/syslog, log->rotateCount is 7
dateext suffix '-20120223'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/syslog to /var/log/syslog-20120223
running postrotate script
removing old log /var/log/syslog-20111219.gz

rotating pattern: /var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
 forced from command line (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/mail.info
  log /var/log/mail.info does not exist -- skipping
considering log /var/log/mail.warn
  log /var/log/mail.warn does not exist -- skipping
considering log /var/log/mail.err
  log does not need rotating
considering log /var/log/mail.log
  log does not need rotating
considering log /var/log/daemon.log
  log /var/log/daemon.log does not exist -- skipping
considering log /var/log/kern.log
  log needs rotating
considering log /var/log/auth.log
  log needs rotating
considering log /var/log/user.log
  log /var/log/user.log does not exist -- skipping
considering log /var/log/lpr.log
  log /var/log/lpr.log does not exist -- skipping
considering log /var/log/cron.log
  log /var/log/cron.log does not exist -- skipping
considering log /var/log/debug
  log /var/log/debug does not exist -- skipping
considering log /var/log/messages
  log /var/log/messages does not exist -- skipping
rotating log /var/log/kern.log, log->rotateCount is 4
dateext suffix '-20120223'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
rotating log /var/log/auth.log, log->rotateCount is 4
dateext suffix '-20120223'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/kern.log to /var/log/kern.log-20120223
renaming /var/log/auth.log to /var/log/auth.log-20120223
running postrotate script
removing old log /var/log/kern.log-20111218.gz
removing old log /var/log/auth.log-20111218.gz

And, what do you know, there is no more /var/log/syslog, auth.log, kern.log, etc.

Then /sbin/restart rsyslog and they're there again. I know from the other bug permissions were an issue but they seem not to be in this case:

-rw-r----- 1 syslog adm 0 2012-02-23 19:29 auth.log
-rw-r----- 1 syslog adm 79 2012-02-23 19:29 kern.log
-rw-r----- 1 syslog adm 350 2012-02-23 19:29 syslog

In any case, the solution seems to be updating /etc/logrotate.d/rsyslog

From:
        postrotate
                reload rsyslog >/dev/null 2>&1 || true
        endscript

To:
        postrotate
                /sbin/restart rsyslog >/dev/null 2>&1 || true
        endscript

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jim Tarvid (tarvid)]

after restart

root@helen:/var/log# service rsyslog restart
rsyslog stop/waiting
rsyslog start/running, process 3986
root@helen:/var/log# tail -f syslog
Jun 28 08:59:35 helen kernel: imklog 5.8.6, log source = /proc/kmsg started.
Jun 28 08:59:35 helen rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="3986" x-info="http://www.rsyslog.com"] start
Jun 28 08:59:35 helen rsyslogd: rsyslogd's groupid changed to 103
Jun 28 08:59:35 helen rsyslogd: rsyslogd's userid changed to 102
Jun 28 08:59:35 helen rsyslogd-2039: Could not open output pipe '/dev/xconsole' [try http://www.rsyslog.com/e/2039

why is rsyslogd trying to open '/dev/xconsole'?

[Jim Tarvid (tarvid)]

# Only check if jobs are disabled if the currently _running_ version of
# Upstart (which may be older than the latest _installed_ version)
# supports such a query.
#
# This check is necessary to handle the scenario when upgrading from a
# release without the 'show-config' command (introduced in
# Upstart for Ubuntu version 0.9.7) since without this check, all
# installed packages with associated Upstart jobs would be considered
# disabled.
#
# Once Upstart can maintain state on re-exec, this change can be
# dropped (since the currently running version of Upstart will always
# match the latest installed version).

UPSTART_VERSION_RUNNING=$(initctl version|awk '{print $3}'|tr -d ')')

if dpkg --compare-versions "$UPSTART_VERSION_RUNNING" ge 0.9.7
then
    initctl show-config -e '/dev/xconsole' "$JOB"|grep -q '^ start on' || DISABLE$
fi

[ghomem (gustavo)]

Happens on 12.04 VMs, only sometimes. The systems stops logging and logging only resumes with a restart of rsyslogd.

Rsyslog and logrotate are up to date.

[Aaron B. Russell (aaronr)]

I'm also seeing this on a 12.04 minimal install. After logrotate runs, when syslog stops being written to, here's what my /var/log directory looks like:

-rw-r--r-- 1 root root 3721 Oct 11 16:11 alternatives.log
drwxr-xr-x 2 root root 4096 Aug 21 22:02 apt
-rw-r--r-- 1 root root 6966 Oct 11 16:09 aptitude
drwxr-xr-x 4 asterisk asterisk 4096 Oct 11 16:11 asterisk
-rw-r----- 1 syslog adm 7700 Oct 12 09:41 auth.log
-rw-r----- 1 root adm 31 Aug 21 22:06 boot
-rw-r--r-- 1 root root 1136 Oct 11 16:07 boot.log
-rw-rw---- 1 root utmp 0 Aug 21 22:37 btmp
-rw-r--r-- 1 root root 0 Aug 21 22:37 debug
-rw-r--r-- 1 root adm 13399 Oct 11 16:07 dmesg
-rw-r--r-- 1 root adm 13399 Oct 11 16:06 dmesg.0
-rw-r--r-- 1 root root 28 Oct 11 16:06 dmesg.1.gz
-rw-r--r-- 1 root root 97979 Oct 11 16:11 dpkg.log
-rw-r--r-- 1 root root 24024 Oct 11 16:11 faillog
drwxr-xr-x 2 root root 4096 Aug 21 22:06 fsck
-rw-r----- 1 syslog adm 63451 Oct 11 16:08 kern.log
-rw-rw-r-- 1 root utmp 292292 Oct 12 09:41 lastlog
-rw-r--r-- 1 root root 0 Aug 21 22:37 mail.*
-rw-r----- 1 syslog adm 0 Oct 11 16:05 mail.err
-rw-r----- 1 syslog adm 0 Oct 11 16:05 mail.log
-rw-r--r-- 1 root root 0 Aug 21 22:37 messages
drwxr-xr-x 2 root root 4096 Oct 11 16:05 news
-rw-r--r-- 1 root root 0 Oct 12 07:35 syslog
-rw-r--r-- 1 root root 22423 Oct 11 16:07 syslog.1
-rw-r--r-- 1 root root 85907 Oct 11 16:07 udev
drwxr-xr-x 2 root root 4096 Oct 12 07:35 upstart
-rw-rw-r-- 1 root utmp 13056 Oct 12 09:41 wtmp

... and here's what it looks like after a reboot:

-rw-r--r-- 1 root root 3721 Oct 11 16:11 alternatives.log
drwxr-xr-x 2 root root 4096 Aug 21 22:02 apt
-rw-r--r-- 1 root root 6966 Oct 11 16:09 aptitude
drwxr-xr-x 4 asterisk asterisk 4096 Oct 11 16:11 asterisk
-rw-r----- 1 syslog adm 8206 Oct 12 09:51 auth.log
-rw-r----- 1 root adm 31 Aug 21 22:06 boot
-rw-r--r-- 1 root root 1073 Oct 12 09:48 boot.log
-rw-rw---- 1 root utmp 0 Aug 21 22:37 btmp
-rw-r--r-- 1 root root 0 Aug 21 22:37 debug
-rw-r--r-- 1 root adm 13399 Oct 12 09:48 dmesg
-rw-r--r-- 1 root adm 13399 Oct 11 16:07 dmesg.0
-rw-r--r-- 1 root adm 5228 Oct 11 16:06 dmesg.1.gz
-rw-r--r-- 1 root root 28 Oct 11 16:06 dmesg.2.gz
-rw-r--r-- 1 root root 97979 Oct 11 16:11 dpkg.log
-rw-r--r-- 1 root root 24024 Oct 11 16:11 faillog
drwxr-xr-x 2 root root 4096 Aug 21 22:06 fsck
-rw-r----- 1 syslog adm 85551 Oct 12 09:48 kern.log
-rw-rw-r-- 1 root utmp 292292 Oct 12 09:51 lastlog
-rw-r--r-- 1 root root 0 Aug 21 22:37 mail.*
-rw-r----- 1 syslog adm 0 Oct 11 16:05 mail.err
-rw-r----- 1 syslog adm 0 Oct 11 16:05 mail.log
-rw-r--r-- 1 root root 0 Aug 21 22:37 messages
drwxr-xr-x 2 root root 4096 Oct 11 16:05 news
-rw-r--r-- 1 root root 22260 Oct 12 09...

[Aaron B. Russell (aaronr)]

Workaround: open /etc/logrotate.d/rsyslog, and replace this line:

reload rsyslog >/dev/null 2>&1 || true

... with this line:

restart rsyslog >/dev/null 2>&1 || true

[RootDSP (leomcmanus)]

Noted that the permissions on the syslog and kern,log are changed from syslog:syslog to root:root. Upon changing the ownership back, the logs are updated correctly.

[RootDSP (leomcmanus)]

...without a restart of the syslog service

[Tom Mercelis (tom-mercelis)]

adding a line "create 640 syslog adm" in /etc/logrotate.d/rsyslog (in the two blocks) resolves the permission issue for me (Ubuntu 12.04)

[TJ (tj)]

This affects the current development track for Saucy.

I have a minimal server install of 64-bit 13.10 and since logrotate ran at 0630 there have been no updates to any log files. Restarting rsyslog hasn't improved things. some log files seem to be owned by "messagebus:adm", rsylog is user "syslog" and only a member of "syslog" group.

# date
Thu Aug 22 14:11:59 BST 2013

# ls -altr /var/log/
total 1700
drwxr-xr-x 2 root root 4096 May 24 2012 sysstat
drwxr-xr-x 2 ntp ntp 4096 Apr 4 23:25 ntpstats
drwxr-xr-x 2 root root 4096 Apr 25 19:07 fsck
drwxr-xr-x 2 root root 4096 Apr 25 19:07 apt
drwxr-xr-x 3 root root 4096 Apr 25 19:14 installer
drwxr-xr-x 2 root root 4096 Apr 25 19:14 news
drwxr-xr-x 2 landscape root 4096 Apr 25 19:14 landscape
drwxr-xr-x 2 root root 4096 Apr 25 19:14 ConsoleKit
drwxr-xr-x 2 root root 4096 Apr 25 19:16 unattended-upgrades
-rw-r----- 1 messagebus adm 0 Apr 26 17:14 ufw.log
-rw-r----- 1 messagebus adm 0 Apr 26 17:14 mail.log
-rw-r----- 1 messagebus adm 0 Apr 26 17:14 mail.err
-rw-rw---- 1 root utmp 0 Apr 26 17:14 btmp
-rw-r--r-- 1 root root 0 Apr 26 17:14 bootstrap.log
-rw-r--r-- 1 root root 0 Apr 26 17:14 boot.log
-rw-r----- 1 root adm 0 Apr 26 17:14 boot
-rw-r----- 1 messagebus adm 0 Apr 26 17:14 auth.log
-rw-r--r-- 1 root root 0 Apr 26 17:14 aptitude
-rw-r----- 1 root adm 4667 Aug 21 01:31 dmesg.4.gz
-rw-r----- 1 root adm 4744 Aug 21 02:57 dmesg.3.gz
-rw-r----- 1 root adm 4663 Aug 21 03:26 dmesg.2.gz
drwxr-xr-x 4 root root 4096 Aug 21 04:24 dist-upgrade
-rw-r----- 1 root adm 4704 Aug 21 04:27 dmesg.1.gz
drwxr-x--- 2 root adm 4096 Aug 21 04:37 apache2
-rw-r----- 1 root adm 11817 Aug 21 04:46 dmesg.0
-rw-r--r-- 1 root root 285844 Aug 21 04:56 udev
-rw-r----- 1 root adm 12150 Aug 21 04:57 dmesg
drwxr-xr-x 15 root root 4096 Aug 21 05:06 ..
-rw-r--r-- 1 root root 32064 Aug 21 06:52 faillog
drwxr-xr-x 2 root root 4096 Aug 21 07:17 dnssec-tools
-rw-r----- 1 messagebus adm 82150 Aug 21 18:25 kern.log
-rw-rw-r-- 1 root utmp 25344 Aug 22 04:47 wtmp
-rw-rw-r-- 1 root utmp 292584 Aug 22 04:47 lastlog
-rw-r--r-- 1 root root 417434 Aug 22 04:48 dpkg.log
-rw-r--r-- 1 root root 14003 Aug 22 04:48 alternatives.log
drwxr-xr-x 2 root root 4096 Aug 22 06:29 upstart
-rw-r----- 1 messagebus adm 741465 Aug 22 06:29 syslog.1
-rw-r----- 1 messagebus adm 0 Aug 22 06:29 syslog
drwxr-xr-x 15 root root 4096 Aug 22 06:29 .

# tail -1 /var/log/syslog.1
Aug 22 05:29:03 hush rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="2647" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

# ls -altr /proc/$(pidof rsyslogd)/fd/
total 0
dr-xr-xr-x 8 syslog syslog 0 Aug 22 14:04 ..
dr-x------ 2 root root 0 Aug 22 14:05 .
lr-x------ 1 root root 64 Aug 22 14:05 4 -> /proc/kmsg
lrwx------ 1 root root 64 Aug 22 14:05 3 -> socket:[59553]
lrwx------ 1 root root 64 Aug 22 14:05 0 -> socket:[59551]

[TJ (tj)]

This seems to be caused by the logrotate configuration for rsyslog using the system dbus-daemon. Here's the representative example for "/var/log/syslog":

/var/log/syslog
{
        rotate 7
        daily
        missingok
        notifempty
        delaycompress
        compress
        postrotate
                reload rsyslog >/dev/null 2>&1 || true
        endscript
}

The 'postrotate' script command "reload" is a symlink to "/usr/sbin/initctl"

initctl uses the system dbus to communicate:

# ps -C dbus-daemon -O user,group --noheader | awk '{cmd="getent passwd " $2; cmd | getline user; cmd="getent group " $3; cmd | getline group; printf "PID: %d USER: %s GROUP: %s CMD: %s\n", $1, user, group, $7}'

PID: 2617 USER: messagebus:x:102:105::/var/run/dbus:/bin/false GROUP: messagebus:x:105: CMD: dbus-daemon

# ps -C rsyslogd -O user,group
  PID USER GROUP S TTY TIME COMMAND
25970 syslog syslog S ? 00:00:00 rsyslogd -c5

I'm not quite sure how that causes ownership of the files to change since the 'rsyslogd' process is running as "syslog:syslog" and its configuration sets FileOwner:FileGroup to "syslog:adm" so the HUP should just not interfere with the ownership when it recreates the log file.

It seems FileGroup ("adm") survives but FileOwner is the dbus-daemon UID.

[TJ (tj)]

After reading "man initctl --reload" I wonder if this is caused if for some reason the 'rsyslogd' process has died/stopped during logrotate and therefore a new instance is started by dbus-daemon which (initially?) inherits the dbus-daemon owner credentials?

[TJ (tj)]

Looking back through the rotated "syslog" I noticed that the PID of rsyslogd hasn't changed when it was HUPed:

Aug 21 03:57:00 hush rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="2647" x-info="http://www.rsyslog.com"] start
Aug 21 03:57:00 hush rsyslogd: rsyslogd's groupid changed to 103
Aug 21 03:57:00 hush rsyslogd: rsyslogd's userid changed to 101
Aug 21 03:57:00 hush rsyslogd-2039: Could not open output pipe '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ]
Aug 22 05:29:03 hush rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="2647" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

But I guess it could have died whilst re-creating the files, which may lead to it being restarted with the dbus-daemon user "messagebus" user file-ownership credentials. Then, when it drops its privileges (I presume it expects it was started as root) it loses access to its own files.

[Dariusz Dwornikowski (dariusz-dwornikowski)]

I can confirm the same bug on precise server. After logrotate /var/log/syslog is root:root and I syslog does not log anymore. Multiple restarts fixes the issue.

On the other hand it works just fine on soucy server for me.

[Dariusz Dwornikowski (dariusz-dwornikowski)]

I can confirm adding "create 640 syslog adm" to logrotate fixes the problem.

[Gustavo L (gustavo-lapido)]

Should "create 640 syslog adm" be inserted right before the reload statement?

[Marcelo Fernandez (fernandezm)]

I also can confirm adding "create 640 syslog adm" to /etc/logrotate.d/rsyslog fixes the problem.

@Gustavo, you should change the /etc/logrotate.d/rsyslog file from this:

/var/log/syslog
{
        rotate 7
        daily
        missingok
        notifempty
        delaycompress
        compress
        postrotate
                reload rsyslog >/dev/null 2>&1 || true
        endscript
}

to this:

/var/log/syslog
{
        rotate 7
        daily
        missingok
        notifempty
        delaycompress
        compress
        create 640 syslog adm
        postrotate
                reload rsyslog >/dev/null 2>&1 || true
        endscript
}

and now you're ok.

And for the record, I also moved away from syslog-ng to rsyslog, so I think it may cause this bug (perhaps a package's install/uninstall script).

Regards

[Hendrik Mundt (hendrik-mundt)]

I can confirm the bug on 14.04.4 LTS using rsyslog and logrotate. /var/log/syslog and others are not empty and being filled ok, but other logfiles are empty, e.g. /var/log/samba/*, /var/log/mysql.*, /var/log/alternatives.log

[Matt Chesler (fu79)]

I'm also seeing this issue on 14.04.3 LTS and 14.04.4 LTS. In my case, "/sbin/reload rsyslog" does not cause rsyslog to reopen the file handles and it instead continues writing to the rotated files, e.g. /var/log/syslog.1 until explicitly restarted.

[robert mccallum (raversnet)]

I'm seeing the same issue as @Chesler #20.

On Ubuntu 16.04.1 LTS. Log files are created but logging keeps going in the rotated file. ie: Auth.log.1 until rsyslog is restarted.

[Claudio Kuenzler (napsty)]

Can confirm the same behavior on Ubuntu 16.04 LTS. Log files are rotated by logrotate but postrotate command doesn't seem to work.

ls -ltr /var/log | tail
-rw-r----- 1 syslog adm 0 Aug 19 06:42 kern.log
-rw-rw---- 1 root utmp 0 Sep 1 06:49 btmp.1
-rw-r--r-- 1 root root 252 Sep 7 17:02 memcached.log
-rw-rw-r-- 1 root utmp 15360 Sep 29 15:43 wtmp.1
-rw-rw---- 1 root utmp 0 Oct 1 06:41 btmp
-rw-r----- 1 syslog adm 605488 Oct 10 11:56 mail.log.1
-rw-rw-r-- 1 root utmp 4224 Oct 10 12:38 wtmp
-rw-rw-r-- 1 root utmp 292876 Oct 10 12:38 lastlog
-rw-r----- 1 syslog adm 17692156 Oct 10 13:53 auth.log.1
-rw-r----- 1 syslog adm 3045212 Oct 10 13:53 syslog.1

postrotate command seems wrong anyway:

# cat /etc/logrotate.d/rsyslog | grep -A 2 postrotate
 postrotate
  invoke-rc.d rsyslog rotate > /dev/null
 endscript
--
 postrotate
  invoke-rc.d rsyslog rotate > /dev/null
 endscript

invoke-rc.d rsyslog rotate doesn't work:

# invoke-rc.d rsyslog rotate
initctl: invalid command: rotate
Try `initctl --help' for more information.
invoke-rc.d: initscript rsyslog, action "rotate" failed.

But in general postrotate commands don't seem to be executed at all.

[Claudio Kuenzler (napsty)]

Correction: "But in general postrotate commands don't seem to be executed at all."
Yes it does work, phew. But the command "invoke-rc.d rsyslog rotate" is wrong as it doesn't work:

# invoke-rc.d rsyslog rotate
initctl: invalid command: rotate
Try `initctl --help' for more information.
invoke-rc.d: initscript rsyslog, action "rotate" failed.

https://bugs.launchpad.net/nginx/stable/+bug/1450770 is related. There it's also the problem with a non-working postrotate command.

[Robert (robert-ubuntu)]

Ubuntu 16.04.1 LTS
Can confirm that invoke-rc.d does not work.

root@server:~# invoke-rc.d rsyslog rotate
initctl: invalid command: rotate
Try `initctl --help' for more information.
invoke-rc.d: initscript rsyslog, action "rotate" failed.

However, service does work.

root@server:~# service rsyslog rotate
 * Closing open files rsyslogd
   ...done.

Appears that the problem is associated with invoke-rc.d

HTH

[e b (tehmoon)]

Hi,

I don't know if it can help but I've a similar bug which involves restarting Rsyslog.
I use Ubuntu 16.04 LTS 64bits with Rsyslog 8.16-0-1ubuntu3 on two kind of hardware:

  - docker-machine
  - AWS EC2

I noticed that on servers that don't use systemd -- in this case docker-machine -- using the sysV init script doesn't work.

The reason:

L 54 start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --exec $DAEMON

If you remove the --quiet:
 /etc/init.d/rsyslog stop
 * Stopping enhanced syslogd rsyslogd
No /usr/sbin/rsyslogd found running; none killed.
   ...done.

Then if you remove --exec $DAEMON it works.

I have the same behavior when I use the init script on a systemd based ubuntu. But the systemd job seems to work fine.

And of course, /etc/init.d/rsyslog rotate doesn't work either.

I'm currently debating with my self on how I'm going to handle that..

[jon bird (news-u)]

I'm seeing something similar with 14.04 since upgrading from 12.04 only in this case, I'm still getting kernel messages written to the logs however nothing else is until I force a restart of rsyslog. File permissions look ok though (syslog:adm).

[rahul (rahulshaw-2009)]

I see the same issue in Ubuntu 14.04. reload rsyslog doesn't write to new log file but the old one only.

[Guillaume (e1msih)]

Same issue here with rsyslog and logrotate on many ubuntu 16.04 servers.

[roy (roy123)]

A workaround.

Have/Had the same issue.
As the logrotate is executed by the daily cronjob, I modified /etc/cron.daily/logrotate to the following:
-------------------------------------------------
#!/bin/sh

/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
/bin/systemctl restart rsyslog
exit 0
----------------------------------------------------

And that resolved the issue.

[Vincenzo Do (vincenzo-do)]

Any updates on this bug? Is there a fix on the way? I'm experiencing this problem on 16.04 Ubuntu.

[Brian Murray (brian-murray)]

I've tested this on Ubuntu 16.04 with rsyslog version 8.16.0-1ubuntu3 on an install with systemd and I also tested it on the Bionic Beaver, which will become Ubuntu 18.04, with rsyslog version 8.16.0-1ubuntu10 on an install also with systemd and was unable to recreate the bug my test case on both releases follows.

1) Run 'sudo logrotate --force /etc/logrotate.conf /etc/logrotate.d/rsyslog'
2) Observe /var/log/syslog with recent timestamp
3) Run 'sudo service cups restart'
4) 'cat /var/log/syslog'
5) Observe messages regarding the restart of cups in the log file

In the event that anyone is still experiencing this issue please include detailed information about the version of rsyslog you have installed, whether or not using are using systemd, and the release of Ubuntu which you are running.

[Alexander Kallenbach (kallenbachalex)]

Has been hit by this bug on two of my servers apparently (both 18.04.)