echo string to /dev/kmsg fails to appear on /var/log/syslog

[Chris J Arges]

[Impact]
In the precise 3.2 series kernel writing strings to /dev/kmsg appears and dmesg and /var/log/syslog, in 3.5 series kernel this does not happen properly even with the equivalent userspace packages.

This is because the default facility number in the kernel has changed from 0 to 1 for user-space generated kmsg entries. Because of this; rsyslog (or any affected sysloggers) need to accept non-kernel facility numbers.

This change was introduced by this commit (in the linux kernel):
7ff9554bb578ba02166071d2d487b7fc7d860d62

[Test Case]
Run the following script as root:

#!/bin/bash
STRING=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 32)
# > /var/log/syslog # this isn't necessary
echo $STRING > /dev/kmsg
sleep 5
grep $STRING /var/log/syslog
# This should return 0.

[Regression Potential]
Any scripts that rely on syslog output may finally see output produced by packages like init/systemd show up in syslog. This changes the default rsyslog.conf file settings but isn't a code change.

[Chris J Arges (arges)]

After doing a bisect, I found the first bad commit is:
7ff9554bb578ba02166071d2d487b7fc7d860d62

[Chris J Arges (arges)]

Before that patch if we echo something into /dev/kmsg we get:
<4>[ 35.084348] testing

If we do it on or after that patch we get:
<12>[ 71.091005] test

This change makes sense from the commit message "All userspace-injected messages enforce a
      facility value > 0 now, to be able to reliably distinguish them from
      the kernel-generated messages"

[Chris J Arges (arges)]

Ok so the <N> is both the priority and facility numbers combined. Where the lowest 3 bits are priority and any upper bits are facility. By default any userspace log is 1, and only kernel messages originating from the kernel can be 0.

http://www.mjmwired.net/kernel/Documentation/ABI/testing/dev-kmsg

So this sounds like something that needs to be patched in sysloggers and not the kernel.

[Chris J Arges (arges)]

Sent email to rsyslog about this:
http://lists.adiscon.net/pipermail/rsyslog/2014-June/037843.html

[Steve Langasek (vorlon)]

thanks for tracking this down, Chris!

[Chris J Arges (arges)]

Ok here is the workaround:
in /etc/rsyslog.conf we need to add:

$KLogPermitNonKernelFacility on

[Chris J Arges (arges)]

Attaching a patch here for utopic. I know there is a pending merge, so perhaps this can be applied on top of that merge.

[Chris J Arges (arges)]

Marking this as affecting trusty/precise because those series can run 3.5+ kernels.

[Chris J Arges (arges)]

Uploaded for Utopic; after it lands in -updates and seems to not break things, I'll look at SRU'ing into previous releases.

[Chris J Arges (arges)]

Re: #7, to clarify. it wasn't a pending merge it was an outstanding merge.

[Launchpad Janitor (janitor)]

This bug was fixed in the package rsyslog - 7.4.4-1ubuntu4

---------------
rsyslog (7.4.4-1ubuntu4) utopic; urgency=medium

  * Enable non-kernel facility klog messages. (LP: #1274444)
 -- Chris J Arges <email address hidden> Tue, 01 Jul 2014 14:59:40 -0500

[Chris J Arges (arges)]

Uploaded SRU fixes for Trusty/Precise.

[Brian Murray (brian-murray)]

Hello Chris, or anyone else affected,

Accepted rsyslog into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/rsyslog/7.4.4-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Chris, or anyone else affected,

Accepted rsyslog into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/rsyslog/5.8.6-1ubuntu8.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Simon Déziel (sdeziel)]

The verification on Precise was done using the linux-image-generic-lts-trusty kernel.

[Scott Kitterman (kitterman)]

The verification of the Stable Release Update for rsyslog has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package rsyslog - 5.8.6-1ubuntu8.7

---------------
rsyslog (5.8.6-1ubuntu8.7) precise; urgency=medium

  * Enable non-kernel facility klog messages. (LP: #1274444)
 -- Chris J Arges <email address hidden> Tue, 19 Aug 2014 15:00:53 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package rsyslog - 7.4.4-1ubuntu2.1

---------------
rsyslog (7.4.4-1ubuntu2.1) trusty; urgency=medium

  * Enable non-kernel facility klog messages. (LP: #1274444)
 -- Chris J Arges <email address hidden> Tue, 19 Aug 2014 14:51:26 -0500