Wrong permissions in /dev after Intrepid->Jaunty upgrade

Bug #317944 reported by Matt Zimmerman on 2009-01-16
4
Affects Status Importance Assigned to Milestone
udev (Ubuntu)
Undecided
Unassigned

Bug Description

Binary package hint: udev

I just upgraded a system from Intrepid to Jaunty, and ran into a lot of problems afterward due to incorrect permissions in /dev. The most obvious were /dev/null and /dev/random, both of which were 0660 root/root (matching my umask, FWIW).

A reboot (which was tricky due to the above) restored correct permissions.

stat shows that the inode was changed during the upgrade:

  File: `/dev/null'
  Size: 0 Blocks: 0 IO Block: 4096 character special file
Device: eh/14d Inode: 5862 Links: 1 Device type: 1,3
Access: (0660/crw-rw----) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2008-10-10 16:25:20.000000000 +0100
Modify: 2008-10-10 16:25:20.000000000 +0100
Change: 2009-01-16 15:08:00.892201634 +0000

I am not sure which package is at fault. I will attach a dpkg log and find -ls output to this bug.

ProblemType: Bug
Architecture: i386
DistroRelease: Ubuntu 9.04
Package: udev 136-4
ProcEnviron:
 LC_COLLATE=C
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/zsh
SourcePackage: udev
Uname: Linux 2.6.28-4-generic i686

Matt Zimmerman (mdz) wrote :
Matt Zimmerman (mdz) wrote :

Your ls does not show incorrect permissions:

  5862 0 crw-rw-rw- 1 root root Oct 10 16:25 /dev/null

The Jan 16 15:08 is the time you did the upgrade? That suggests something has run "udevtrigger" in its postinst. Could you grep for that?

Changed in udev:
status: New → Incomplete

On Fri, Jan 16, 2009 at 06:34:41PM -0000, Scott James Remnant wrote:
> Your ls does not show incorrect permissions:
>
> 5862 0 crw-rw-rw- 1 root root Oct 10 16:25
> /dev/null

That's after I corrected it. Refer to the stat(1) output I provided to see
the state of it before I fixed it. Things were broken enough (e.g. zsh)
with /dev/null permissions wrong that I couldn't even usefully report the
bug.

You can see in the find -ls output that /dev/zero and /dev/random were wrong
as well.

> The Jan 16 15:08 is the time you did the upgrade? That suggests something has run "udevtrigger" in its postinst. Could you grep for that?

That's approximately the upgrade time, yes.

mizar:[~] grep udevtrigger /var/lib/dpkg/info/*.{pre,post}*
zsh: exit 1 grep udevtrigger /var/lib/dpkg/info/*.{pre,post}*
mizar:[~] grep udevadm /var/lib/dpkg/info/*.{pre,post}*
zsh: exit 1 grep udevadm /var/lib/dpkg/info/*.{pre,post}*

--
 - mdz

Changed in udev:
status: Incomplete → New

Strange that every single device has lost its world ownership permission; the only ones I note with it left are:

23204068 0 srw-rw-rw- 1 root root 0 Jan 16 15:12 /dev/log
  5862 0 crw-rw-rw- 1 root root Oct 10 16:25 /dev/null
  1305 0 crw-rw-rw- 1 root root Oct 21 10:25 /dev/urandom

Do you think something might have actually done "find | chmod" and escaped and run across /dev for some reason?

Do you happen to have system logs from the upgrade time, there may be messages that give us a clue what happened.

Changed in udev:
status: New → Incomplete
Matt Zimmerman (mdz) wrote :

On Mon, Jan 19, 2009 at 08:01:29AM -0000, Scott James Remnant wrote:
> Strange that every single device has lost its world ownership
> permission; the only ones I note with it left are:
>
> 23204068 0 srw-rw-rw- 1 root root 0 Jan 16 15:12 /dev/log
> 5862 0 crw-rw-rw- 1 root root Oct 10 16:25 /dev/null
> 1305 0 crw-rw-rw- 1 root root Oct 21 10:25 /dev/urandom

/dev/null and /dev/urandom are the two that I manually corrected in order to
get basic functionality working again. /dev/log I didn't touch, but that
looks like it was recreated during the upgrade (syslogd was presumably
restarted).

> Do you think something might have actually done "find | chmod" and
> escaped and run across /dev for some reason?

I've looked around the filesystem a bit, and nothing looks out of the
ordinary outside of /dev.

You can see in the stat output that only the ctime changed, so it's the
original device node which was created by udev, but has been mangled by
something.

> Do you happen to have system logs from the upgrade time, there may be
> messages that give us a clue what happened.

I have the dist-upgrader logs which look fairly complete, and will attach
those. The timestamps in the log confirm that the ctime on /dev/null was
during the upgrade, and narrows down the possible packages (somewhere from
netbase to gsm-utils):

2009-01-16 13:33:36,223 INFO cache.commit()
2009-01-16 14:08:47,065 DEBUG got a conffile-prompt from dpkg for file: '/etc/services'
2009-01-16 15:00:50,532 WARNING no activity on terminal for 240 seconds (Configuring netbase)
2009-01-16 15:09:13,277 DEBUG got a conffile-prompt from dpkg for file: '/etc/init.d/gsm-utils'

I also searched for other similar reports, but didn't find anything which was
obviously a match:

https://bugs.edge.launchpad.net/ubuntu/intrepid/+source/linux/+bug/290153/comments/67 (intrepid)
http://ubuntuforums.org/showthread.php?t=229938 (dapper to edgy upgrade)
http://ubuntuforums.org/showthread.php?t=1033252 (August 2007)

--
 - mdz

Matt Zimmerman (mdz) wrote :
Matt Zimmerman (mdz) wrote :

This is the list of packages whose maintainer scripts were executed during the timeframe when the permissions appear to have changed.

Matt Zimmerman (mdz) wrote :

Here's the terminal output from that span of time

On Mon, 2009-01-19 at 08:51 +0000, Matt Zimmerman wrote:

> > Do you think something might have actually done "find | chmod" and
> > escaped and run across /dev for some reason?
>
> I've looked around the filesystem a bit, and nothing looks out of the
> ordinary outside of /dev.
>
> You can see in the stat output that only the ctime changed, so it's the
> original device node which was created by udev, but has been mangled by
> something.
>
The /dev/.udev/db have been touched as well, which heavily implies that
udev was triggered.

With the rules being moved, it is possible that the trigger would reset
the permissions to 0660.

This is a pretty bad side-effect, so we should probably at least stop
the udev daemon on upgrade.

But this doesn't solve what's done it for you - as that's another bug.
I'd rather leave the symptom there for now otherwise we're hiding
something else.

> > Do you happen to have system logs from the upgrade time, there may be
> > messages that give us a clue what happened.
>
> I have the dist-upgrader logs which look fairly complete, and will attach
> those. The timestamps in the log confirm that the ctime on /dev/null was
> during the upgrade, and narrows down the possible packages (somewhere from
> netbase to gsm-utils):
>
Notably udev isn't in this list.

Scott
--
Scott James Remnant
<email address hidden>

On Mon, Jan 19, 2009 at 12:01:23PM -0000, Scott James Remnant wrote:
> On Mon, 2009-01-19 at 08:51 +0000, Matt Zimmerman wrote:
>
> > > Do you think something might have actually done "find | chmod" and
> > > escaped and run across /dev for some reason?
> >
> > I've looked around the filesystem a bit, and nothing looks out of the
> > ordinary outside of /dev.
> >
> > You can see in the stat output that only the ctime changed, so it's the
> > original device node which was created by udev, but has been mangled by
> > something.
> >
> The /dev/.udev/db have been touched as well, which heavily implies that
> udev was triggered.
>
> With the rules being moved, it is possible that the trigger would reset
> the permissions to 0660.

Why would a trigger set the permissions to 0660 when the udev rule calls for
0666?

> > > Do you happen to have system logs from the upgrade time, there may be
> > > messages that give us a clue what happened.
> >
> > I have the dist-upgrader logs which look fairly complete, and will attach
> > those. The timestamps in the log confirm that the ctime on /dev/null was
> > during the upgrade, and narrows down the possible packages (somewhere from
> > netbase to gsm-utils):
> >
> Notably udev isn't in this list.

It definitely seems as if this happened when running maintainer scripts from
the Jaunty packages, so whatever did this is still lurking out there.

I'm currently grepping through the package contents for anything suspicious.
One thing which caught my eye was the new LSB init script dependency
comments (e.g. Required-Start: udev in pcmciautils). Could these have
caused the udev init script to be run somehow?

--
 - mdz

On Mon, 2009-01-19 at 12:38 +0000, Matt Zimmerman wrote:

> > The /dev/.udev/db have been touched as well, which heavily implies that
> > udev was triggered.
> >
> > With the rules being moved, it is possible that the trigger would reset
> > the permissions to 0660.
>
> Why would a trigger set the permissions to 0660 when the udev rule calls for
> 0666?
>
Because the running udev can't parse the new configuration files? (It
doesn't even look in the right place for them.)

If something forced udev to reload its configuration (which has gone
away), then it would end up with none, so fallback on defaults
(root:root/0660)

> I'm currently grepping through the package contents for anything suspicious.
> One thing which caught my eye was the new LSB init script dependency
> comments (e.g. Required-Start: udev in pcmciautils). Could these have
> caused the udev init script to be run somehow?
>
Not that I know of - that would have actually unmounted /dev which I
would expect to have a few more drastic consequences

Scott
--
Scott James Remnant
<email address hidden>

On Mon, 2009-01-19 at 12:59 +0000, Scott James Remnant wrote:

> > I'm currently grepping through the package contents for anything suspicious.
> > One thing which caught my eye was the new LSB init script dependency
> > comments (e.g. Required-Start: udev in pcmciautils). Could these have
> > caused the udev init script to be run somehow?
> >
> Not that I know of - that would have actually unmounted /dev which I
> would expect to have a few more drastic consequences
>
(Plus if it restarted the new udev daemon, things would have worked ;-)

Scott
--
Scott James Remnant
<email address hidden>

On Mon, Jan 19, 2009 at 12:59:39PM -0000, Scott James Remnant wrote:
> On Mon, 2009-01-19 at 12:38 +0000, Matt Zimmerman wrote:
>
> > > The /dev/.udev/db have been touched as well, which heavily implies that
> > > udev was triggered.
> > >
> > > With the rules being moved, it is possible that the trigger would reset
> > > the permissions to 0660.
> >
> > Why would a trigger set the permissions to 0660 when the udev rule calls for
> > 0666?
> >
> Because the running udev can't parse the new configuration files? (It
> doesn't even look in the right place for them.)
>
> If something forced udev to reload its configuration (which has gone
> away), then it would end up with none, so fallback on defaults
> (root:root/0660)

I haven't been able to identify any misbehaving packages. Is it feasible to
have it fail in a less horrible way?

--
 - mdz

Changed in udev:
status: Incomplete → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package udev - 137-1

---------------
udev (137-1) jaunty; urgency=low

  * New upstream release:
    - udevadm test no longer has force option.
    - udevd has --resolve-names=early|late|never option.
    - Group of IDE CD-ROM drives fixed. LP: #315997.
    - Group of DRI subsystem fixed. LP: #317430.
    - /etc/udev/rules.d not existing is not an error. LP: #315780.
    - Bug where device nodes would be replaced by symlinks on rename has been
      fixed. LP: #315979.

  * Use --resolve-names=never in the installer and initramfs, since we don't
    have a useful name service. LP: #319199.
  * Since we don't have to worry about group lookup, we may as well copy the
    default rules into the initramfs as well. This actually double-solves
    LP: #315979.
  * Make sure the root filesystem is writable before attempting to copy
    generated rules across. LP: #224870.
  * Remove /dev/MAKEDEV symlink; the FHS no longer requires it when /dev
    is automatically managed.

  * It is not permitted to call udevadm trigger or settle during an upgrade
    without depending on udev. Attempting this will fail.
  * Change /etc/init.d/udev restart to actually restart the daemon, with a
    bit of detection to print a warning if we missed events while the
    daemon was down.
  * Refreshing /dev is now /etc/init.d/udev refresh-devices
  * Restart udev daemon after upgrade. LP: #317944.

 -- Scott James Remnant <email address hidden> Fri, 23 Jan 2009 15:15:07 +0000

Changed in udev:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers