Multiple issues with pam_motd update-motd code

[Steve McIntyre]

Package: libpam-modules
Version: 1.1.1-2ubuntu5

Running Lucid on a Beagleboard, wondering why logins take so long. Horrified to find that the problem is pam_motd... Several issues:

 1. Why on earth is *every* login updating a global file (/etc/motd -> /var/run/motd)?
    a. Why spam every user?
    b. Why even write to the file every time? It's being generated on all logins, so there doesn't seem to be any useful caching go on.
 2. This is adding ~5 seconds of startup lag to every login, which is annoying
 3. I don't see any locking around the updates of /var/run/motd.new, so it seems that multiple parallel logins could easily case corruption
 4. No documentation of any of this in pam_motd(8)

If you must update motd automatically, then surely running something via cron would be better than this messy hack?

[Steve McIntyre (steve-mcintyre)]

4 months with no responses...

[Steve Langasek (vorlon)]

Hi Steve,

If anything, using cron for this would be worse, not better. The information from some of the data sources used here is dynamic enough that, when this was being done with cron previously, the job was set to a five minute interval - and since cron itself opens a PAM session for each job, that meant hammering the disk once every five minutes to write out security logs.

On a default system, only ssh and login will call pam_motd; and typically both of these are rather infrequent events. Of all the possible implementations (and we did try them all; see bug #399071), doing the update inline in pam_motd was determined to be the least wasteful despite the possible problem it introduces with login-time interactivity.

Can you help pin down why it takes 5 seconds to generate this file? The individual component scripts under /etc/update-motd.d now each do their own out-of-dateness checks in shell; there will still be some cost, but I wouldn't expect it to add up to a whole 5 seconds on every login. On the first login after installation, yes, it will take some time to do the initial generation; but afterwards all the information is cached in /var/lib (which should be /var/cache, heh) and only the concatenation to /var/run/motd is duplicated.

To answer your specific questions:

> 1. Why on earth is *every* login updating a global file (/etc/motd -> /var/run/motd)?

To avoid unnecessarily updating it when people *aren't* logging into a service that uses pam_motd. On typical desktop systems, console logins happen approximately never. We could add a 5 minute cache timeout or so to /var/run/motd itself. I'm not sure how much that would help, really.

> a. Why spam every user?

It's... the motd. That's what it's for. Now, it's certainly designed to be tunable *what* information gets included in the motd; perhaps Linaro would prefer not to use any dynamic motd for instance.

> b. Why even write to the file every time? It's being generated on all logins, so there doesn't seem to be any useful caching go on.

Well, it's cached in the sense that anything else which looks at the motd without using pam_motd will be able to get the updated information. But otherwise, yes, not being cached very effectively.

> 3. I don't see any locking around the updates of /var/run/motd.new, so it seems that multiple parallel logins could easily case corruption

Confirmed, that appears to be a bug. Patches welcome of course. It's not the highest priority for fixing, given that a corrupted motd isn't particularly fatal.

> 4. No documentation of any of this in pam_motd(8)

There is a link to update-motd(5) which documents this, but you're right that it should be called out more explicitly.

[Steve McIntyre (steve-mcintyre)]

Hmmm.

I don't think I saw a message to tell me there was an update to this bug, which is why I've not responded in ages. But I could be wrong...

I've just been bitten by this daft issue again, which prompted me to look at this bug for any responses.

Using a micro-SD card with an ARM lucid installation, just logged in via the serial console to rescue the real system (installed on a SATA hard drive but broken due to kernel testing).

Logging in as root, multiple seconds (maybe 20?) with no discernible activity. Then it eventually printed out the motd. I've just removed all of /etc/update-motd.d to save me the pain again on this machine. I'm guessing that the *huge* slowdown could well be due to network timeouts on this device, which is not on the network. Or maybe the very slow uSD is the problem.

Going back to the report:

If console logins are so rare, I don't see any value to update-motd anyway. Even on a system without the long delays I'm seeing, it looks like make-work. In my case, I'm only using the console to start with to try and fix a problem, so slowing down access to the console unnecessarily is very frustrating.

In terms of the bad coding with no locking, I'm afraid you'll have to look elsewhere for fixes. My preferred solution to this would be to simply remove the update-motd misfeature altogether.

[Marius Gedminas (mgedmin)]

> > 3. I don't see any locking around the updates of /var/run/motd.new, so it seems that multiple parallel logins could easily case corruption

> Confirmed, that appears to be a bug. Patches welcome of course. It's not the highest priority for fixing, given that a corrupted motd isn't particularly fatal.

Speaking of "patches welcome", I provided a bzr branch for a similar motd corruption caused by a different script in https://bugs.launchpad.net/ubuntu/+source/update-notifier/+bug/1146170 seven months ago. As of today, nobody responded.

[Marc Deslauriers (mdeslaur)]

@Marius: I've subscribed the ubuntu-sponsors group to the other bug so you can get your patch sponsored.