[FFe] /var/log/dmesg No Longer Being Updated

Bug #1450588 reported by Jared Fernandez on 2015-04-30
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
rsyslog (Ubuntu)
Undecided
Canonical Server Team
systemd (Ubuntu)
Undecided
Unassigned

Bug Description

[FFe]
 * (re-)introduce a feature to ensure the initial (boot time) kernel
   messages are preserved
 * This existed up to Trusty (upstart) but was lost afterwards as it had
   no systemd coutnerpart.
 * It is a "new" feature since we have lacked it for so many releases and
   worth - a hopefully simple - ack by the release Team

 * It is not a new version or any change to the actual rsyslog code.
   Instead it just adds a new service "dmesg" that will achieve what was
   lost post trusty. Therefore the potential regression to the existing
   function should be minimal, if anything the new service might hit
   issues on some unexpected environments but atm that seems unlikely.

 * It does not add/remove Dependencies nor modify build

 * I ahve made upgrade/install tests as well as Ahasenack doing the same
   on the MP that is linked.

---

After upgrading to Ubuntu 15.04 Vivid, /var/log/dmesg is no longer updated after boot.

It appears that this was previously done via /etc/init/dmesg.conf

Related branches

Martin Pitt (pitti) wrote :

Right, /var/log/dmesg was an upstart specific file. There is little point in writing yet another log file, given that all the information is already available in the systemd journal, /var/log/kern.log, and /var/log/syslog.

Changed in systemd (Ubuntu):
status: New → Won't Fix
Haw Loeung (hloeung) wrote :

Unfortunately, systemd journal, /var/log/kern.log, and /var/log/syslog are all log rotated so on long running systems, the initial boot logs are lost.

Any chance we can reconsider the decision to stop writing out /var/log/dmesg?

Changed in systemd (Ubuntu):
status: Won't Fix → Confirmed
Dimitri John Ledkov (xnox) wrote :

No.

Adjust your journal config for longer retention, if so desired.
http://manpages.ubuntu.com/manpages/xenial/man5/journald.conf.5.html

Changed in systemd (Ubuntu):
status: Confirmed → Won't Fix
James Troup (elmo) wrote :

Sorry but that argument is illogical. The only retention period that would ensure that you always have the boot time kernel messages is 'infinity' and that's not a reasonable option for obvious reasons (c.f. LP #1618188).

Changed in systemd (Ubuntu):
status: Won't Fix → New
Dimitri John Ledkov (xnox) wrote :

./debian/rsyslog.dmesg.upstart was shipped by rsyslog package in Trusty and/or other releases.

It was:
start on runlevel [2345]
task
script
    savelog -q -p -c 5 /var/log/dmesg
    dmesg -s 524288 > /var/log/dmesg
    chgrp adm /var/log/dmesg
end script

Thus if such functionality is desired to be preserved, imho rsyslog should be shipping a similar configuration or unit. Note that journald pushes all entries to rsyslog, thus i would have thought it could have generated /var/log/dmesg using normal rsyslog logging configuration facilities.

I'm not sure how to configure rsyslog correctly to retain these. But I can write an equivalent systemd unit:

$ systemctl cat dmesg
# /etc/systemd/system/dmesg.service
[Unit]
Description=Save kernel messages

[Service]
Type=idle
ExecStartPre=/usr/bin/savelog -q -p -c 5 /var/log/dmesg
ExecStart=/bin/sh -c '/bin/dmesg -s 524288 > /var/log/dmesg'
ExecStartPost=/bin/chgrp adm /var/log/dmesg

[Install]
WantedBy=multi-user.target

The dmesg command can be approximated with the following journalctl command, which may have more data than dmesg ring-buffer:

$ journalctl --boot 0 --dmesg --output short-monotonic --quiet --no-pager --no-hostname

however, that is slightly incompatible format due to the extra `kernel:` stanza after the timestamp

[1195009.811669] kernel: aufs au_plink_put:436:umount[15188]: pseudo-link is not flushed

--boot 0 -> means to show output from current boot only
--dmesg -> means to show dmesg logs only
--output short-monotonic -> means to prefix things with regular [monotonic] time in dmesg like format
--quiet -> disables the headers with when the logs/boot started finished
--no-pager -> dumps to stdout, instead of less
--no-hostname -> supresses the hostname entry which is injected between time and kernel: normally

E.g. on my system journalctl command above recovers things from 0.000000 time microcode updated, instead of from very late messages.

Imho journalctl should suppress the `kernel:` as that should be implied by --dmesg flag.

Depending on how much things happen during boot, i'm not sure if dmesg should be used, or journalctl to retrieve the boot log.

Changed in systemd (Ubuntu):
status: New → Invalid
Dimitri John Ledkov (xnox) wrote :

Marking as invalid for systemd package. As neither it, nor upstart shipped the functionality in question. It is a regression in the rsyslog package.

Or possibly makes sense to ship this in like util-linux which ships /bin/dmesg.

Changed in rsyslog (Ubuntu):
assignee: nobody → Canonical Server Team (canonical-server)
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in rsyslog (Ubuntu):
status: New → Confirmed

Thanks Dimitri for all the pre-work and the suggested systemd unit matching the old upstart job.
We really like that, especially as it avoids the concern of "just another ongoing log" due to being an initial one-shot task that does not continue to log the ongoing messages.
And yes - having especially the initial boot messages can be quite handy way later.

There also is not only the journal but also at /etc/rsyslog.d/50-default.conf:
 kern.* -/var/log/kern.log
But it is on logrotation (4) so it would as well over time loose the early boot messages.

In our discussion we even consider to SRU this as it is essentially a loss of functionality post-Trusty.

The journal style invocation is clearly better to not exceed ring buffers so fast on huge systems.
And especially on such it might be interesting later on what was going on.

Being lost post-trusty we might consider the more reliable way to get all messages (journalctl) more important than keeping the format absolutely the same.
There is nothing there people can rely upon yet, so the new format won't break them.

This isn't releas breaking, so as usual we'd have to wait until DD opens, then implement there and potentially SRU it back then.

tags: added: server-next
Changed in rsyslog (Ubuntu):
status: Confirmed → Triaged

As soon as something isn't urgent it falls off everyones lists :-/
I have found this old bug (again) in my notes, felt embarrassed and converted the suggestion of xnox into an MP that we can ack and push to early 19.10 then (hopefully not forgetting it again).

We probably will have to rebase then, but lets agree on an MP now by reviewing and testing functionality in Disco.

After some experiments I use systemd'd output redirection which will make it not rely on sh, nor internal redirection and nicely leaves stderr on the job/journal but the output we want in the file.

A PPA to try is at [1] and an MP to package that change is at [2].

The change actually isn't that bg even thou it is a feature. We might convert this to an FFe if review and tests are fine.

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1450588-rsyslog-to-get-dmesg-log-again/
[2]: https://code.launchpad.net/~paelzer/ubuntu/+source/rsyslog/+git/rsyslog/+merge/364133

Dimitri John Ledkov (xnox) wrote :

I'm not sure why you think an FFe is needed for this. It is a simple regression since move from upstart->systemd in the rsyslog package, and we must SRU this back to all the releases back.

@xnox - thanks for your ping, I agree if we would have worked on this in Vivid or at least Xenial.
But being so late I'd think to ask for an ack by the Release team is not wrong.
I thought here "better safe than sorry" somewhat applies.
OTOH fortunately the change itself isn't very invasive to an existing thing (as it is a new service), so it seems rather safe anyway.

But I can be convinced - let me ask in the Team if everyone else also thinks this wouldn't be requiring an FFe. If so we can just upload.

Robie Basak (racb) wrote :

It would be the readdition of a feature that we lost as a feature regression nine releases ago. I don't see why the release team would object, but I can't speak for the release team. It feels like a feature to me. In the general case, ignoring feature freeze for features being readded that were dropped multiple releases ago seems disingenuous to me.

Per former discussion, better safe than sorry - added a FFe request to the description and subscribed the release-team.

description: updated
Changed in rsyslog (Ubuntu):
status: Triaged → New
summary: - /var/log/dmesg No Longer Being Updated
+ [FFe] /var/log/dmesg No Longer Being Updated
Steve Langasek (vorlon) wrote :

From a feature freeze POV, this is ok with me. I don't know that I agree it makes sense to reintroduce this log file now after so long if the information is available elsewhere, but I don't think I need to block this w/ release team hat.

Thanks Steve,
as people have explained above the problem with the "if the information is available elsewhere" argument is, that very long living systems have rotated out the logs that contained the initial boot. This is the extra value this wants to give back to the users.

Changed in rsyslog (Ubuntu):
status: New → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rsyslog - 8.32.0-1ubuntu7

---------------
rsyslog (8.32.0-1ubuntu7) disco; urgency=medium

  * Install dmesg.service (LP: #1450588) to provide /var/log/dmesg.log
    as non log-rotated log for boot time kernel messages.
    - debian/dmesg.service: new service to write /var/log/dmesg
    - debian/rsyslog.install: install dmesg.service with rsyslog

 -- Christian Ehrhardt <email address hidden> Thu, 07 Mar 2019 14:00:30 +0100

Changed in rsyslog (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers