multiple running instances of update-motd cause errors

Bug #394556 reported by Stephen Sentoff
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
update-motd (Ubuntu)
Fix Released
High
Dustin Kirkland 

Bug Description

Binary package hint: update-motd

Description: Ubuntu 9.04
Release: 9.04
update-motd:
  Installed: 1.11.1

When update-motd is executed out of /etc/cron.d, the instances are all set to run at the same minute (if they run at all). This means that up to five instances of update-motd can be started simultaneously. This causes two problems--

1. The method used to lock out simultaneous executions is not very robust. The $NEW file, which is used as a synchronization lock, is tested early in the script, but not created until later. This leaves a window open for multiple scripts to get started, and then collide with each other, ultimately messing up the resulting message of the day. At least on my machine, this error happens about half the time on each hour. The code to "nice" the process seems to be the part that opens the window the widest. Using lockfile to create a semaphore for synchronization corrects this flaw. I've created a modified version of update-motd and included it as an attachment.

2. By design, only one instance of update-motd can run at a time, and others just exit. However, this means that some of the work scheduled for the same minute doesn't get done (assuming the synchronization works correctly). But this isn't really what you want--if both the 10-minute and the hourly script are scheduled on the hour, you want both to execute serially. I can see two possible solutions to this:
a) Alter the lockfile code in the attachment to allow retries. Changing -r0 to -r5 in the attached code would allow all the instances that could possibly be scheduled to take their turns to run. The downside of this solution is that if update-motd is run manually, it could take up to 40 seconds to fail, when the lock is in place. Even the 8 second delay for a single retry is likely to be confusing to the user.
b) Alter the update-motd crontab entry in /etc/cron.d so that the instances of update-motd don't coincide. For example, one job on the 10-minute boundaries, one 1 minute after the hour, one 2 minutes after midnight each day, one 3 minutes after midnight each week, one 4 minutes after midnight each month. This change would also reduce the likelihood of multiple running instances to the point that the current method of synchronization with the $NEW file would rarely cause a problem.

Revision history for this message
Stephen Sentoff (steve30401) wrote :
Changed in update-motd (Ubuntu):
assignee: nobody → Dustin Kirkland (kirkland)
importance: Undecided → High
milestone: none → ubuntu-9.04
status: New → Triaged
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Hi Stephen-

I'm working on this at the moment.

I'm curious what implementation of lockfile you're using in your script?

dpkg -S `which lockfile`?

The one from procmail, perhaps?

:-Dustin

Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Nevermind, I found flock(1). It's working like a champ ;-)

:-Dustin

Changed in update-motd (Ubuntu):
status: Triaged → In Progress
Revision history for this message
Stephen Sentoff (steve30401) wrote : RE: [Bug 394556] Re: multiple running instances of update-motd cause errors
Download full text (3.3 KiB)

Yes, I was using the lockfile from the procmail package.

But I agree flock should work just as well. I saw both were available and arbitrarily picked lockfile for my testing.

Thanks for your work.

--
Steve

> Date: Fri, 10 Jul 2009 20:14:13 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: [Bug 394556] Re: multiple running instances of update-motd cause errors
>
> Nevermind, I found flock(1). It's working like a champ ;-)
>
> :-Dustin
>
> ** Changed in: update-motd (Ubuntu)
> Status: Triaged => In Progress
>
> --
> multiple running instances of update-motd cause errors
> https://bugs.launchpad.net/bugs/394556
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in “update-motd” package in Ubuntu: In Progress
>
> Bug description:
> Binary package hint: update-motd
>
> Description: Ubuntu 9.04
> Release: 9.04
> update-motd:
> Installed: 1.11.1
>
> When update-motd is executed out of /etc/cron.d, the instances are all set to run at the same minute (if they run at all). This means that up to five instances of update-motd can be started simultaneously. This causes two problems--
>
> 1. The method used to lock out simultaneous executions is not very robust. The $NEW file, which is used as a synchronization lock, is tested early in the script, but not created until later. This leaves a window open for multiple scripts to get started, and then collide with each other, ultimately messing up the resulting message of the day. At least on my machine, this error happens about half the time on each hour. The code to "nice" the process seems to be the part that opens the window the widest. Using lockfile to create a semaphore for synchronization corrects this flaw. I've created a modified version of update-motd and included it as an attachment.
>
> 2. By design, only one instance of update-motd can run at a time, and others just exit. However, this means that some of the work scheduled for the same minute doesn't get done (assuming the synchronization works correctly). But this isn't really what you want--if both the 10-minute and the hourly script are scheduled on the hour, you want both to execute serially. I can see two possible solutions to this:
> a) Alter the lockfile code in the attachment to allow retries. Changing -r0 to -r5 in the attached code would allow all the instances that could possibly be scheduled to take their turns to run. The downside of this solution is that if update-motd is run manually, it could take up to 40 seconds to fail, when the lock is in place. Even the 8 second delay for a single retry is likely to be confusing to the user.
> b) Alter the update-motd crontab entry in /etc/cron.d so that the instances of update-motd don't coincide. For example, one job on the 10-minute boundaries, one 1 minute after the hour, one 2 minutes after midnight each day, one 3 minutes after midnight each week, one 4 minutes after midnight each month. This change would also reduce the likelihood of multiple running instances to the point that the current method of synchronization with the $NEW file would rarely cause a problem.

___...

Read more...

Revision history for this message
Ahmad Fauzi (vladislas) wrote :

that's same problem with
upgrade process
in "update-motd" error
in the log:

Errors were encountered while processing:
 update-motd
E: Sub-process /usr/bin/dpkg returned an error code (1)

I'm use the order
$ sudo /etc/init.d/update-motd stop

And continued upgrade it's no problem

Changed in update-motd (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.