rsyslogd crashes when the spool is corrupted

Bug #1233441 reported by Alec Warner on 2013-09-30
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
rsyslog (Ubuntu)
Undecided
Unassigned
Precise
Medium
Chris J Arges
Quantal
Medium
Chris J Arges

Bug Description

[Impact]

When daemonizing, the child process segfaults and the parent acts oddly.

[Test Case]

Edit /etc/rsyslog.conf and /etc/rsyslog.d/40-loghost-client.conf as shown below.

stop rsyslog
mv /var/spool/rsyslog /var/spool/rsyslog.bak
tar -xf var-spool-rsyslog.tar.gz -C /

gdb /usr/sbin/rsyslogd
set follow-fork-mode child
run -c5

(gdb) bt
#0 strmFlushInternal (pThis=0x0) at stream.c:1236
#1 0x00000000004291b9 in strmSerialize (pThis=0x0, pStrm=0x676150) at stream.c:1569
#2 0x000000000042ee02 in qqueuePersist (pThis=0x675a10, bIsCheckpoint=0) at queue.c:2015
#3 0x000000000042f026 in qqueueDestruct (ppThis=0x674d78) at queue.c:2153
#4 0x0000000000432660 in StartDA (pThis=0x674b50) at queue.c:353
#5 InitDA (pThis=0x674b50, bLockMutex=<optimized out>) at queue.c:402
#6 qqueueStart (pThis=0x674b50) at queue.c:1914
#7 0x0000000000437987 in actionConstructFinalize (pThis=0x674a60) at ../action.c:405
#8 0x000000000043977e in addAction (ppAction=0x7fffffff8ca8, pMod=<optimized out>, pModData=0x673910, pOMSR=0x6732f0, bSuspended=0)
    at ../action.c:1779
#9 0x0000000000416934 in cflineDoAction (ppAction=<synthetic pointer>, p=0x7fffffff8c90) at conf.c:1105
#10 cflineClassic (ppRule=0x7fffffff8d10, p=0x7fffffff8d5c "") at conf.c:1172
#11 cfline (line=<optimized out>, pfCurr=0x7fffffff8d10) at conf.c:1207
#12 0x00000000004171a5 in processConfFile (pConfFile=0x671af0 "/etc/rsyslog.d/40-loghost-client.conf") at conf.c:464
#13 0x0000000000417586 in doIncludeLine (pp=<optimized out>, pVal=<optimized out>) at conf.c:234
#14 0x0000000000435771 in cslchCallHdlr (ppConfLine=0x7fffffffa858, pThis=<optimized out>) at cfsysline.c:724
#15 processCfSysLineCommand (pCmdName=<optimized out>, p=0x7fffffffa898) at cfsysline.c:936
#16 0x00000000004166b1 in cfsysline (p=0x7fffffffa99f "/etc/rsyslog.d/*.conf") at conf.c:371
#17 0x0000000000416a29 in cfline (line=0x7fffffffa991 "IncludeConfig /etc/rsyslog.d/*.conf", pfCurr=0x7fffffffa980) at conf.c:1204
#18 0x00000000004171a5 in processConfFile (pConfFile=0x44204c "/etc/rsyslog.conf") at conf.c:464
#19 0x000000000040aa53 in init () at syslogd.c:1620
#20 0x000000000040e0ce in mainThread () at syslogd.c:2179
#21 realMain (argc=<optimized out>, argv=<optimized out>) at syslogd.c:2914
#22 0x00007ffff71fc76d in __libc_start_main (main=0x40af80 <main>, argc=2, ubp_av=0x7fffffffe508, init=<optimized out>,
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe4f8) at libc-start.c:226
#23 0x000000000040afc9 in _start ()

Now of course, you need our syslog config.

cat /etc/rsyslog.conf
$PreserveFQDN on

$ModLoad imuxsock
$ModLoad imklog
$ModLoad immark

$umask 0000

$CreateDirs on

$DirOwner root
$DirGroup root
$DirCreateMode 0755

$EscapeControlCharactersOnReceive off

$FileOwner root
$FileGroup root
$FileCreateMode 0644

$MarkMessagePeriod 600

$IncludeConfig /etc/rsyslog.d/*.conf

antarus@antarus-z620:/var/lib/chaps$ cat /etc/rsyslog.d/40-loghost-client.conf

$WorkDirectory /var/spool/rsyslog
$ActionQueueFileName loghost-client
$ActionQueueCheckpointInterval 100
  # A 100000µs delay effects a rate limit of 10 messages per second.
$ActionQueueDequeueSlowdown 100000
  $ActionQueueMaxDiskSpace 1g
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList
# Do not suspend the retry on machines assumed to be connected
# (I.e., stick to once per second.)
$ActionResumeRetryCount -1
$ActionResumeInterval 1

#*.* @@REDACTED_LOGHOST.:514;RSYSLOG_ForwardFormat

I will attach the busted spool files as well.

root@antarus-z620:/etc# lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04

root@antarus-z620:/etc# apt-cache policy rsyslog
rsyslog:
  Installed: 5.8.6-1ubuntu8.5
  Candidate: 5.8.6-1ubuntu8.5

I expect rsyslog to not crash, and instead detect bad spool files and tell me.

Instead it crashed.

Alec Warner (antarus) wrote :

This is the corrupted spool. I recommend using a VM, or otherwise backing up /var/spool/rsyslog.

-A

tags: added: precise
Chris J Arges (arges) wrote :

So far,

works: 5.8.11-2ubuntu2
segfault: 5.8.6-1ubuntu9

This means Q/P are affected.

Changed in rsyslog (Ubuntu Precise):
assignee: nobody → Chris J Arges (arges)
importance: Undecided → Medium
status: New → In Progress
Changed in rsyslog (Ubuntu):
status: New → Fix Released
Changed in rsyslog (Ubuntu Quantal):
assignee: nobody → Chris J Arges (arges)
importance: Undecided → Medium
status: New → In Progress
tags: added: quantal
Chris J Arges (arges) wrote :

Uploaded fixes for P/Q.

Hello Alec, or anyone else affected,

Accepted rsyslog into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/rsyslog/5.8.6-1ubuntu8.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in rsyslog (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in rsyslog (Ubuntu Quantal):
status: In Progress → Fix Committed
Stéphane Graber (stgraber) wrote :

Hello Alec, or anyone else affected,

Accepted rsyslog into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/rsyslog/5.8.6-1ubuntu9.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Chris J Arges (arges) on 2013-12-04
description: updated
Chris J Arges (arges) on 2013-12-04
description: updated
description: updated
Chris J Arges (arges) on 2013-12-04
tags: added: verification-done
removed: verification-needed
Chris J Arges (arges) wrote :

Verified bug for p/q.

description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rsyslog - 5.8.6-1ubuntu8.6

---------------
rsyslog (5.8.6-1ubuntu8.6) precise; urgency=low

  * debian/patches/203-bugfix-segfault-on-startup-if-actionqueuefilename-wa.patch:
    upstream fix for segfault when spool is corrupted. (LP: #1233441)
 -- Chris J Arges <email address hidden> Fri, 22 Nov 2013 11:28:58 -0600

Changed in rsyslog (Ubuntu Precise):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for rsyslog has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rsyslog - 5.8.6-1ubuntu9.3

---------------
rsyslog (5.8.6-1ubuntu9.3) quantal; urgency=low

  * debian/patches/203-bugfix-segfault-on-startup-if-actionqueuefilename-wa.patch:
    upstream fix for segfault when spool is corrupted. (LP: #1233441)
 -- Chris J Arges <email address hidden> Fri, 22 Nov 2013 11:47:09 -0600

Changed in rsyslog (Ubuntu Quantal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers