wrong permissions for /dev/null when /usr is a separate filesystem

[Timo Aaltonen]

/etc/init.d/udev calls usplash_write which fails because libx86.so is in /usr (not mounted at this point if /usr is a separate filesystem). /etc/init.d/udev's set -e setting then causes udevd not to run and /dev/null (and many other entries in /dev) end up being wrong.

Additionally, it is a bug that usplash_write links to libx86 since it is just the IPC client and shouldn't need it.

[Timo Aaltonen (tjaalton)]

uh, of course I could've also shown what the permissions are now:

#luthien 0:17 /var/log 16 # ls -l /dev/null
crw------- 1 root root 1, 3 Feb 7 22:21 /dev/null

[Mathieu Bérard (mathieu-berard)]

Hi,
get this behavior too, in fact lots of files in /dev have bad permission, some symlinks are missing too. A "/etc/init.d/udev restart" workaround the problem.

[Matt Zimmerman (mdz)]

Ian, do you know what's going on here?

[Áron Sisak (asisak)]

Confirmed, Feisty, Linux 2.6.19-7-386.

[Ian Jackson (ijackson)]

This is indeed rather worrying. The most obvious conclusion to jump
to is that my uploads yesterday broke it, but the symptoms seem to
correlate with udev not running, or still running from the initramfs.
(Needless to say it works just fine for me.)

Does this problem happen only on the first boot after the upgrade ?
Can someone with a broken system say
  ps -ef | grep udev
  dpkg -l '*udev*'
?

Do any of you have any out-of-the-ordinary block devices ? (lvm,
mdraid, cryptsetup, evms)

Ian.

[Mathieu Bérard (mathieu-berard)]

Hi,
The problem happen upon each boot not only the first one after upgrade.

I indeed have lvm block devices:
/dev/mapper/Base-Root / xfs defaults 0 1
/dev/mapper/Base-Home /home xfs defaults 0 2
/dev/mapper/Base-Usr /usr xfs defaults 0 2
/dev/mapper/Base-Var /var xfs defaults 0 2

Notice that I have root-on-lvm and /usr as a distinct partition.

When booting in recovery mode:

ps -ef :
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 13:32 ? 00:00:00 /sbin/init single
root 2 1 0 13:32 ? 00:00:00 [ksoftirqd/0]
root 3 1 0 13:32 ? 00:00:00 [events/0]
root 4 1 0 13:32 ? 00:00:00 [khelper]
root 5 1 0 13:32 ? 00:00:00 [kthread]
root 23 5 0 13:32 ? 00:00:00 [kblockd/0]
root 24 5 0 13:32 ? 00:00:00 [kacpid]
root 129 5 0 13:32 ? 00:00:00 [kseriod]
root 149 5 0 13:32 ? 00:00:00 [pdflush]
root 150 5 0 13:32 ? 00:00:00 [pdflush]
root 151 5 0 13:32 ? 00:00:00 [kswapd0]
root 152 5 0 13:32 ? 00:00:00 [aio/0]
root 617 5 0 13:32 ? 00:00:00 [ata/0]
root 618 5 0 13:32 ? 00:00:00 [ata_aux]
root 622 5 0 13:32 ? 00:00:00 [scsi_eh_0]
root 623 5 0 13:32 ? 00:00:00 [scsi_eh_1]
root 624 5 0 13:32 ? 00:00:00 [scsi_eh_2]
root 625 5 0 13:32 ? 00:00:00 [scsi_eh_3]
root 758 5 0 13:32 ? 00:00:00 [ksuspend_usbd]
root 759 5 0 13:32 ? 00:00:00 [khubd]
root 837 5 0 13:32 ? 00:00:00 [khpsbpkt]
root 1083 5 0 13:32 ? 00:00:00 [knodemgrd_0]
root 1084 5 0 13:32 ? 00:00:00 [scsi_eh_4]
root 1085 5 0 13:32 ? 00:00:00 [scsi_eh_5]
root 1261 5 0 13:32 ? 00:00:00 [xfslogd/0]
root 1262 5 0 13:32 ? 00:00:00 [xfsdatad/0]
root 1270 5 0 13:32 ? 00:00:00 [xfsbufd]
root 1297 5 0 13:32 ? 00:00:00 [xfssyncd]
root 1503 1 0 13:32 ? 00:00:00 /sbin/udevd --daemon
root 1552 5 0 13:32 ? 00:00:00 [kpsmoused]
root 1620 5 0 13:32 ? 00:00:00 [kmmcd]
root 1837 5 0 13:32 ? 00:00:00 [kjournald]
root 1850 5 0 13:32 ? 00:00:00 [xfsbufd]
root 1851 5 0 13:32 ? 00:00:00 [xfssyncd]
root 1853 5 0 13:32 ? 00:00:00 [xfsbufd]
root 1854 5 0 13:32 ? 00:00:00 [xfssyncd]
root 1856 5 0 13:32 ? 00:00:00 [xfsbufd]
root 1857 5 0 13:32 ? 00:00:00 [xfssyncd]
root 2209 1 0 13:32 tty1 00:00:00 /bin/sh -e -c ?runlevel --set S >/dev/null || true??/sbin/sulogin???if [ -r /etc/inittab ]; then?? RL="$(sed -n -e "/^id:[0-9]*:initdefault:/{s/^id://;s/:.*//;p}" /etc/inittab || true)"?? if [ -n "$RL" ]; then???telinit $RL?? else???telinit 2?? fi??else?? telinit 2??fi?
root 2211 2209 0 13:32 tty1 00:00:00 bash
root 2235 2211 0 13:32 tty1 00:00:00 ps -ef

ls -al /dev :
total 4
drwxr-xr-x 14 root root 3020 Feb...

[Mathieu Bérard (mathieu-berard)]

Sorry forgot the udev package requested info:

LANG=C dpkg -l '*udev*' :
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-installed
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name Version Description
+++-==============-==============-===============================================
ii udev 103-0ubuntu11 rule-based device node and kernel event manager

[Ian Jackson (ijackson)]

Mathieu Bérard writes ("[Bug 83878] Re: wrong permissions for /dev/null"):
> [stuff]

Thanks for the information. Could you edit /etc/init.d/udev and
replace this line
        if start-stop-daemon --start --quiet --exec /sbin/udevd -- --daemon; the
with
        if start-stop-daemon --start --quiet --exec /sbin/udevd -- --daemon --verbose >/var/run/udevd-output 2>&1; then

and then boot normally and send me a copy of udevd-output ? Note that
this file has no timestamps so I want a copy from before you do the
experiments below.

Also, some more questions:
 * does running udevtrigger fix a broken system ?
 * grep 'event queue' /var/log/*
 * check for last entry like this
    UDEV [1170948102.066608] add@/class/mem/null
    UDEV_LOG=6
    ACTION=add
    DEVPATH=/class/mem/null
    SUBSYSTEM=mem
    SEQNUM=1695
    MAJOR=1
    MINOR=3
    UDEVD_EVENT=1
    DEVNAME=/dev/null
   in /var/log/udev (and tell me the time_t at the last boot, eg
   with the help of date +%s).

Ian.

[Till Kamppeter (till-kamppeter)]

This causes also HPLIP's hpssd not starting:

https://launchpad.net/bugs/83878

[Till Kamppeter (till-kamppeter)]

Sorry, wrong link:

https://launchpad.net/bugs/83924

[Mathieu Bérard (mathieu-berard)]

Hi,
while doing the things you asked me, I've pinpointed the origin of the problem:

/etc/init.d/udev call usplash_write at line 51, but last usplash update introduce the use of the libx86 library:

ldd /sbin/usplash_write
        libx86.so.1 => /usr/lib/libx86.so.1 (0xb7f9d000)
        libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7e5c000)
        /lib/ld-linux.so.2 (0xb7fb2000)

As you can see libx86.so.1 is in /usr which in my config is a separate partition and is thus not yet mounted a this point of the boot process, as the udev init.d script is called with the '-e' switch set the script abort at this point, as udetrigger is called later in the script, it never get a change to run.

So the recently uploaded udev-lvm modifications are not the origin of the problem, usplash is.

As a side-note I have no /var/log/udev as this file is produced by udevmonitor which is also in /usr.

[Ian Jackson (ijackson)]

Mathieu Bérard writes ("[Bug 83878] Re: wrong permissions for /dev/null"):
> while doing the things you asked me, I've pinpointed the origin of
> the problem:
>
> /etc/init.d/udev call usplash_write at line 51, but last usplash update
> introduce the use of the libx86 library:
>
> ldd /sbin/usplash_write
> libx86.so.1 => /usr/lib/libx86.so.1 (0xb7f9d000)

[stuff]

Well done and thank you! I will fix this shortly.

Ian.

[Stéphane Graber (stgraber)]

I have the same issue (/dev/null and /dev/zero are 660 instead of 666)

[Stéphane Graber (stgraber)]

 * does running udevtrigger fix a broken system ?
yes, it does
 * grep 'event queue' /var/log/*
nothing
 * check for last entry like this
root@laptop:/home/stgraber# ls /var/log/udev*
ls: /var/log/udev*: No such file or directory

[Ian Jackson (ijackson)]

libx86_0.99-1ubuntu1 should fix the most-underlying cause but I think a change to udev is needed too (to make it less fragile).

Thanks very much to Mathieu Bérard for doing the hard lifting here.

[Colin Watson (cjwatson)]

libx86 0.99-1.2 also fixes this; I've resurrected the lost i386 upload that should have made it into the archive this morning but didn't.

[Ian Jackson (ijackson)]

udev_103-0ubuntu12 uses ||: appropriately.

[Colin Watson (cjwatson)]

usplash (0.4-38) feisty; urgency=low

  * Fix typo in usplash_write(8).
  * Only link libusplash.so against libx86, not usplash or usplash_write
    (LP: #83878).
  * Remove redundant debian/usplash.dirs and debian/usplash-dev.dirs.

 -- Colin Watson <email address hidden> Fri, 9 Feb 2007 10:21:56 +0000