rsyslog stopped too early during shutdown/reboot

Bug #1515379 reported by Simon Déziel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
rsyslog (Ubuntu)
New
Undecided
Unassigned

Bug Description

The upstart job managing rsyslog includes a "stop on runlevel [06]" clause that makes it terminate very early during the shutdown/reboot sequence.

This prevents proper logging of the shutdown/reboot sequence. Since this logging can be crucial rsyslog should be kept running for as long as possible. As such, I believe that the "stop on" clause should be dropped from the upstart job definition.

Original description:

When rsyslog is managed by the init script, it symlinks its PID to /run/sendsigs.omit.d so that we can capture logs very late during the shutdown sequence:

sendsigs_omit() {
  OMITDIR=/run/sendsigs.omit.d
  mkdir -p $OMITDIR
  ln -sf $PIDFILE $OMITDIR/rsyslog
}

When rsyslog is managed by upstart this functionality is lost. I quickly looked in Xenial's package and it seems this functionality is also absent when run through systemd.

P.S.: I'm trying to investigate a reboot sequence taking ~forever but for which I have 0 log of the actual shutdown sequence:

Nov 10 22:00:46 node9 rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="1603" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 10 22:55:25 node9 rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="1935" x-info="http://www.rsyslog.com"] start

This 55 minutes gap caused of lot of stress as the machine is a big iron in a DC far away.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: rsyslog 7.4.4-1ubuntu2.6
ProcVersionSignature: Ubuntu 3.13.0-68.111-generic 3.13.11-ckt27
Uname: Linux 3.13.0-68-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.19
Architecture: amd64
CurrentDesktop: Unity
Date: Wed Nov 11 14:14:54 2015
InstallationDate: Installed on 2014-01-26 (653 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Alpha amd64 (20140124)
SourcePackage: rsyslog
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.logcheck.ignore.d.server.rsyslog: [deleted]

Revision history for this message
Simon Déziel (sdeziel) wrote :
Revision history for this message
Simon Déziel (sdeziel) wrote :

In the upstart case, simply removing the "stop on runlevel [06]" clause is enough to capture the log output of the shutdown sequence.

Revision history for this message
Simon Déziel (sdeziel) wrote :

A simplification of this workaround is:

  echo "stop on never-dies" >> /etc/init/rsyslog.override

Simon Déziel (sdeziel)
summary: - rsyslog not omitted by sendsigs when managed by upstart/systemd
+ rsyslog stopped too early during shutdown/reboot
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.