journald takes over /dev/log in bionic, impacts usability of syslog querying

[Steve Langasek]

In bionic, /dev/log is now owned by systemd-journald.

$ sudo fuser -v /dev/log
                     USER PID ACCESS COMMAND
/run/systemd/journal/dev-log:
                     root 1 F.... systemd
                     root 628 F.... systemd-journal
$

This is ok; there are good reasons to want to do this. (Timestamp precision that's not dependent on the syslog protocol; logging available earlier in boot and later at shutdown; unified queriability of logs around services.)

The logfiles previously populated by rsyslog under /var/log are now empty (no longer being updated, and the previous files logrotated away):

$ ls -l /var/log/auth.log
-rw-r----- 1 syslog adm 0 Mar 25 00:06 /var/log/auth.log
$

This is also ok, we don't really want log data duplicated in two places on the system; so since we now have persistent journal under /var/log/journal, we don't want to also create these plaintext files by default.

However, it's not clear that this is by design, rather than by accident. rsyslog is still /configured/ to log to these files; it just isn't receiving any data because it no longer controls the socket.

$ sudo lsof -p 852|grep DGRAM
rsyslogd 852 syslog 3u unix 0xffff8e5680435000 0t0 351 /run/systemd/journal/syslog type=DGRAM
$

Dimitri and I have discussed that rsyslog should continue to function, so that users who have remote syslogging configured can still make use of this. It looks like currently this is not the case, because journald is not forwarding to rsyslog.

That is not ok.

What is also not ok is that, now that logs are only being written to the journal by default instead of to syslog files, querying these logs by syslog priority only works if you know the syslog facility by number (and, afaics, you can only filter by one syslog facility at a time). So whereas before, you could find mail logs by looking in /var/log/mail.log by default, you now have to know to run 'journalctl SYSLOG_FACILITY=2'; and if you want a view of what was previously in /var/log/syslog (i.e. filtering out non-syslog systemd logs, it seems the most compact way to express this is 'journalctl --system SYSLOG_FACILITY={0,1,2,3,5,6,7,8,9,11,12,13,14,15,16,17,18,19,20,21,22,23}'.

This is really not good. Admins have never needed to know the mapping of symbolic names to facility numbers to work with syslog; this throws away 30 years of convention with log handling.

If we are going to store the logs in the journal by default, I think there needs to be a way to query the logs using symbolic names consistent with syslog(3) and /etc/rsyslog.d/50-default.conf.

I don't think we can release with things in the current state.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dimitri John Ledkov (xnox)]

Looking at systemd-journald-dev-log.socket, it creates /run/systemd/journal/dev-log socket, and that unit specifies Symlink to take over /dev/log. And has been doing so since at least xenial (check in a LXD container).

In journald.conf it has set #ForwardToSyslog=yes, which forwards to this socket:

$ sudo fuser -v /run/systemd/journal/syslog
                     USER PID ACCESS COMMAND
/run/systemd/journal/syslog:
                     root 1 F.... systemd
                     syslog 2023 F.... rsyslogd

Which is owned by rsyslogd. Hence, e.g. my /var/log/syslog is populated.

I see that most things configured in /etc/rsyslog.d/50-default.conf, are not getting populated... which is not good:

/var/log/auth.log
/var/log/kern.log
/var/log/mail.log
/var/log/mail.err

All appear to be empty. I guess i need to compare this with Trusty, figure out what's wrong, add an autopkgtest which tests for this, and tests things.

All entries have PRIORITY, SYSLOG_FACILITY, SYSLOG_IDENTIFIER set... since journal must set that upon message arrival, in case it will forward these to the real syslog. To view thing that arrived via syslog protocol, one should filter on the _TRANSPORT field instead.

E.g. The below command shows the syslog stuff
$ journalctl _TRANSPORT=syslog

BTW. All options for _TRANSPORT field are shown below
$ journalctl -F _TRANSPORT
stdout
syslog
journal
driver
audit
kernel

One can filter on severity levels, using syslog keywords for them, e.g. 'crit' using the -p option.

$ journalctl -p crit

On my machine that prints a lot of stuff from consolekit.... no idea why i still have consolekit installed. Maybe we should be force purging it these days on upgrades?

It is sad that one cannot specify logical facility names, as listed in syslog(3) under Values for facility LOG_AUTH..LOG_UUCP.

I'm not sure if we are allowed to forward audit & kernel messages.

[Dimitri John Ledkov (xnox)]

Hmmm.... is rsyslog failing to write things to files?

Apr 07 15:15:01 sochi rsyslogd[2023]: action 'action 6' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.32.0 try http://www.rsyslog.co
Apr 07 15:15:01 sochi rsyslogd[2023]: action 'action 6' resumed (module 'builtin:omfile') [v8.32.0 try http://www.rsyslog.com/e/2359 ]

I see a lot of stuff like that from rsyslog.

syslog 2023 0.0 0.0 347096 5648 ? Ssl Apr06 0:08 /usr/sbin/rsyslogd -n

$ ls -latr auth.log
-rw-r----- 1 root adm 0 Jul 14 2014 auth.log

How can a syslog user write to a root owned file?

<insert mem gif of geometry confusion / does not compute>

$ sudo chown syslog:adm /var/log/auth.log; sudo systemctl restart rsyslog

Seems to fix things for me, and auth.log is getting entries... My guess is that /usr/lib/tmpfiles.d/00rsyslog.conf is incomplete? and that we need to override more permissions in it?

In that case, it sounds like a bug in rsyslog, for not having right permissions specified in tmpfiles.d?

[Dimitri John Ledkov (xnox)]

Possibly duplicate of https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/1748147

[Dimitri John Ledkov (xnox)]

Adding block-proposed to avoid automatic migration, without release team ack.

[Dimitri John Ledkov (xnox)]

Removing systemd task, as the fixes are only needed in the rsyslog package w.r.t. permissions used.

[Dimitri John Ledkov (xnox)]

The current rsyslog does not store logs properly, nor accept remote logs (well accepts, but they end up nowhere). This update solves all that.

[Launchpad Janitor (janitor)]

This bug was fixed in the package rsyslog - 8.32.0-1ubuntu3

---------------
rsyslog (8.32.0-1ubuntu3) bionic; urgency=medium

  * tmpfiles.d: Let var.conf to create /var/log with 'd' directive, and
    only adjust the permissions of /var/log with 'z' directive, thus
    avoiding warnings about duplicate lines for path /var/log. LP:
    #1730028

rsyslog (8.32.0-1ubuntu2) bionic; urgency=medium

  * Ensure correct permissions on files that rsyslog writes to. LP:
    #1761630
  * Drop upgrade scripts from pre-xenial.

 -- Dimitri John Ledkov <email address hidden> Mon, 09 Apr 2018 14:44:54 +0100