rsyslog stopped too early during shutdown/reboot

Bug #1515379 reported by Simon Déziel on 2015-11-11
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
rsyslog (Ubuntu)
Undecided
Unassigned

Bug Description

The upstart job managing rsyslog includes a "stop on runlevel [06]" clause that makes it terminate very early during the shutdown/reboot sequence.

This prevents proper logging of the shutdown/reboot sequence. Since this logging can be crucial rsyslog should be kept running for as long as possible. As such, I believe that the "stop on" clause should be dropped from the upstart job definition.

Original description:

When rsyslog is managed by the init script, it symlinks its PID to /run/sendsigs.omit.d so that we can capture logs very late during the shutdown sequence:

sendsigs_omit() {
  OMITDIR=/run/sendsigs.omit.d
  mkdir -p $OMITDIR
  ln -sf $PIDFILE $OMITDIR/rsyslog
}

When rsyslog is managed by upstart this functionality is lost. I quickly looked in Xenial's package and it seems this functionality is also absent when run through systemd.

P.S.: I'm trying to investigate a reboot sequence taking ~forever but for which I have 0 log of the actual shutdown sequence:

Nov 10 22:00:46 node9 rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="1603" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 10 22:55:25 node9 rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="1935" x-info="http://www.rsyslog.com"] start

This 55 minutes gap caused of lot of stress as the machine is a big iron in a DC far away.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: rsyslog 7.4.4-1ubuntu2.6
ProcVersionSignature: Ubuntu 3.13.0-68.111-generic 3.13.11-ckt27
Uname: Linux 3.13.0-68-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.19
Architecture: amd64
CurrentDesktop: Unity
Date: Wed Nov 11 14:14:54 2015
InstallationDate: Installed on 2014-01-26 (653 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Alpha amd64 (20140124)
SourcePackage: rsyslog
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.logcheck.ignore.d.server.rsyslog: [deleted]

Simon Déziel (sdeziel) wrote :
Simon Déziel (sdeziel) wrote :

In the upstart case, simply removing the "stop on runlevel [06]" clause is enough to capture the log output of the shutdown sequence.

Simon Déziel (sdeziel) wrote :

A simplification of this workaround is:

  echo "stop on never-dies" >> /etc/init/rsyslog.override

Simon Déziel (sdeziel) on 2015-11-11
summary: - rsyslog not omitted by sendsigs when managed by upstart/systemd
+ rsyslog stopped too early during shutdown/reboot
description: updated
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers