[systemd] /etc/logrotate.d/rsyslog fails to instruct rsyslog to close & reopen fds

[Anders Kaseorg]

/etc/logrotate.d/rsyslog contains

        postrotate
                reload rsyslog >/dev/null 2>&1 || true
        endscript

which works on Upstart but not on systemd. Therefore, after logs are rotated, rsyslog continues to write to the old files, leaving the new files empty. It should be changed to ‘service rsyslog reload’ or ‘invoke-rc.d rsyslog reload’ or something.

ProblemType: Bug
DistroRelease: Ubuntu 14.10
Package: rsyslog 7.4.4-1ubuntu3
ProcVersionSignature: Ubuntu 3.15.0-6.11-generic 3.15.0
Uname: Linux 3.15.0-6-generic x86_64
NonfreeKernelModules: openafs
ApportVersion: 2.14.3-0ubuntu2
Architecture: amd64
CurrentDesktop: GNOME
Date: Wed Jun 18 22:20:53 2014
EcryptfsInUse: Yes
InstallationDate: Installed on 2010-12-05 (1291 days ago)
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Alpha amd64 (20101202)
SourcePackage: rsyslog
UpgradeStatus: Upgraded to utopic on 2013-10-31 (230 days ago)

[Launchpad Janitor (janitor)]

This bug was fixed in the package rsyslog - 7.4.4-1ubuntu5

---------------
rsyslog (7.4.4-1ubuntu5) utopic; urgency=medium

  * Use "service" command in rsyslog's postrotate, since naked "reload"
    fails under non-upstart init. (LP: #1331891)
 -- Dimitri John Ledkov <email address hidden> Tue, 08 Jul 2014 09:24:53 +0100

[Anders Kaseorg (andersk)]

This needs a further tweak.

# service rsyslog reload
Usage: /etc/init.d/rsyslog {start|stop|rotate|restart|force-reload|status}
# zcat /usr/share/doc/rsyslog/NEWS.Debian.gz
rsyslog (5.8.1-1) unstable; urgency=low

  The way rsyslog processes SIGHUP has changed. It no longer does a reload
  of its configuration, but simply closes all open files, which is a much more
  lightweight operation.
  To apply a changed configuration, rsyslogd needs to be restarted now.
  As a consequence, the reload action has been dropped from the init script.

  A new action called "rotate" was added to the init script, which signals
  rsyslogd to close all open files. This new action is used in the rsyslog
  logrotate configuration file.

  For more information, see:
  http://www.rsyslog.com/doc/v4compatibility.html
  http://www.rsyslog.com/doc/v5compatibility.html

 -- Michael Biebl <email address hidden> Mon, 30 May 2011 18:26:51 +0200
# service rsyslog rotate
 * Closing open files rsyslogd [ OK ]

See also https://bugs.debian.org/626365. The right action is ‘service rsyslog rotate’, not reload. Sorry for not catching that earlier.

[Anders Kaseorg (andersk)]

I tested that ‘service rsyslog rotate’ works on both Upstart and systemd.

The alternative ‘invoke-rc.d rsyslog rotate’, which the Debian package uses, fails on Upstart, so we still need an Ubuntu delta here (although maybe we should fix invoke-rc.d).

[Dimitri John Ledkov (xnox)]

Post-logrotate needs rsyslog to close and reopen file descriptors. It's not requesting for rsyslog to reload configuration, nor wants to. "reload" is a standard command, and under upstart init it sends SIGHUP to rsyslog, similarly under systemd with init.d script it generates systemd unit for which "reload" similarly sends SIGHUP. In debian unstable a native systemd unit is added for rsyslog, for which reload similarly just sends SIGHUP. Thus under all current and future configuration, the new postrotate results in correct actions taken - i.e. rsyslog closes and reopens files.

For, sysv init scripts "reload" command is specified as something else though in Debian Policy. However, no such restrictions are currently applied for upstart & systemd. If you believe that it's a bug in rsyslog systemd unit and upstart job, please open a separate bug about that in Debian BTS.

[Dimitri John Ledkov (xnox)]

Adjusted bug title, with the essential problem reported here.

[Anders Kaseorg (andersk)]

Please read my comment #3 again. The rsyslog systemd unit does *not* support reload, for the reason documented in /usr/share/doc/rsyslog/NEWS.Debian.gz. ‘service rsyslog reload’ fails with an error. We need to fix the command to ‘service rsyslog rotate’ to perform the action that you describe.

[Anders Kaseorg (andersk)]

Er, that was ambiguous. Let me try again: to perform the action that you describe, we need to fix the command to ‘service rsyslog rotate’.

[Anders Kaseorg (andersk)]

And this is not a bug in Debian, because Debian’s logrotate configuration uses ‘invoke-rc.d rsyslog rotate’.

[Michael Biebl (mbiebl)]

2014-07-08 14:01 GMT+02:00 Dimitri John Ledkov <email address hidden>:
> systemd unit for which "reload" similarly sends SIGHUP. In debian
> unstable a native systemd unit is added for rsyslog, for which reload
> similarly just sends SIGHUP.

That is not true. The rsyslog.service file in Debian's rsyslog package
does *not* have ExecReload set, at least not anymore.
The ExecReload setting was removed upstream because a "reload" means
that a daemon rereads its configuration, which it no longer does for
rsyslog.

Which means, you can't run "systemctl reload rsyslog.service".

# systemctl reload rsyslog.service
Failed to reload rsyslog.service: Job type reload is not applicable
for unit rsyslog.service.

The logrotate file for rsyslog in Debian currently uses
invoke-rc.d rsyslog rotate

This is an action only implemented by the SysV init script. That means
it currently actually relies on the SysV init script even when booted
with systemd.

In Debian, SysV init scripts are still mandatory, so this is sort-of
ok, even though I don't particularly like this situation.

A solution that would work with all init systems would be something like

kill -SIGHUP $(cat /var/run/rsyslogd.pid).

A systemd-only solution would look like this:

systemctl kill -s HUP rsyslog.service

--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Michael Biebl (mbiebl)]

2014-07-08 14:25 GMT+02:00 Michael Biebl <email address hidden>:
> This is an action only implemented by the SysV init script. That means
> it currently actually relies on the SysV init script even when booted
> with systemd.
>
> In Debian, SysV init scripts are still mandatory, so this is sort-of
> ok, even though I don't particularly like this situation.
>
> A solution that would work with all init systems would be something like
>
> kill -SIGHUP $(cat /var/run/rsyslogd.pid).
>
> A systemd-only solution would look like this:
>
> systemctl kill -s HUP rsyslog.service

Maybe shipping a small helper script named rsyslog-rotate in
/usr/sbin might be the best solution for this.

This script could then do the right thing, depending on the init system.

E.g. under systemd it runs
$ systemctl kill -s HUP rsyslog.service
under upstart it runs
$ reload rsyslog
and under sysvinit
$ start-stop-daemon --stop --signal HUP --quiet --pidfile
/var/run/rsyslogd.pid --exec /usr/sbin/rsyslogd

--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Michael Biebl (mbiebl)]

Also related

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=672218

--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Dimitri John Ledkov (xnox)]

On 8 July 2014 13:25, Michael Biebl <email address hidden> wrote:
> 2014-07-08 14:01 GMT+02:00 Dimitri John Ledkov <email address hidden>:
>> systemd unit for which "reload" similarly sends SIGHUP. In debian
>> unstable a native systemd unit is added for rsyslog, for which reload
>> similarly just sends SIGHUP.
>
> That is not true. The rsyslog.service file in Debian's rsyslog package
> does *not* have ExecReload set, at least not anymore.
> The ExecReload setting was removed upstream because a "reload" means
> that a daemon rereads its configuration, which it no longer does for
> rsyslog.
>

I see.

> Which means, you can't run "systemctl reload rsyslog.service".
>
> # systemctl reload rsyslog.service
> Failed to reload rsyslog.service: Job type reload is not applicable
> for unit rsyslog.service.
>
> The logrotate file for rsyslog in Debian currently uses
> invoke-rc.d rsyslog rotate
>
> This is an action only implemented by the SysV init script. That means
> it currently actually relies on the SysV init script even when booted
> with systemd.
>
> In Debian, SysV init scripts are still mandatory, so this is sort-of
> ok, even though I don't particularly like this situation.
>
> A solution that would work with all init systems would be something like
>
> kill -SIGHUP $(cat /var/run/rsyslogd.pid).
>
> A systemd-only solution would look like this:
>
> systemctl kill -s HUP rsyslog.service
>
>
<merging>
> Maybe shipping a small helper script named rsyslog-rotate in
> /usr/sbin might be the best solution for this.

postrotate should send HUP to rsyslog in a init neutral way.
reload is not a required command, and defined to do more than mere HUP
signal implementation in current rsyslogd.
I'm for "kill -SIGHUP $(cat /var/run/rsyslogd.pid)" or a helper
binary, without supporting "reload" command in neither systemd unit,
nor in initd script.
(since reload not supported upstream and rsyslogd instructs sysadmins
to restart daemon to reload configuration)
I'm undecided about, a helper binary vs "kill" line. Kill line is
documented in the manpage, thus is expected interface.

--
Regards,

Dimitri.

[Launchpad Janitor (janitor)]

This bug was fixed in the package rsyslog - 7.4.4-1ubuntu6

---------------
rsyslog (7.4.4-1ubuntu6) utopic; urgency=medium

  * debian/rsyslog.logrotate: Call "rotate" action for rotation instead of
    "reload". (LP: #1331891)
 -- Martin Pitt <email address hidden> Thu, 31 Jul 2014 11:06:52 +0200