When I start runlevel 2 I get 600 permissions, but when I switch to runlevel 1 /dev/null gets proper 666 permissions. So I restarted to get 600 again, and then started stopping one by one service to see when does the change happen: for file in `ls -1 /etc/rc1.d/*|grep -v README|grep -v S90single`; do echo $file >> /tmp/null.log; ls -l /dev/null >> /tmp/null.log; sh $file stop; ls -l /dev/null >> /tmp/null.log; echo "___________" >> /tmp/null.log ; done File /tmp/null.log got this content: K01gdm crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K01kdm crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K01usplash crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K09apache2 crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K11anacron crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K11atd crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K11cron crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K19cupsys crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K19hplip crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K19samba crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20acpi-support crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20apmd crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20apport crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20avahi-daemon crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20console-setup crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20dbus crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20hotkey-setup crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20kde-guidance crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20keyboard-setup crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20makedev crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20networking crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20nvidia-kernel crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20powernowd crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20procps.sh crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20rsync crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20ssh crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20udev crw------- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K20vsftpd crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K21acpid crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K21mysql crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K22mysql-ndb crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K23mysql-ndb-mgm crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K30killprocs crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K30squid crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K74bluetooth crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K79nfs-common crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K80nfs-kernel-server crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K81portmap crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K89klogd crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ K90sysklogd crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 2007-04-15 13:49 /dev/null ___________ Ok, then I restarted, and went to single-user mode, and started starting all of the services from runlevel 2: for file in /etc/rc2.d/S*; do echo $file >> /tmp/null-single.log; ls -l /dev/null >> /tmp/null-single.log; sh $file start; ls -l /dev/null >> /tmp/null-single.log; done File /tmp/null-single.log got: S20makedev crw-rw-rw- 1 root root 1, 3 Apr 15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 Apr 15 13:49 /dev/null S20procps.sh crw-rw-rw- 1 root root 1, 3 Apr 15 13:49 /dev/null crw-rw-rw- 1 root root 1, 3 Apr 15 13:49 /dev/null S20udev crw-rw-rw- 1 root root 1, 3 Apr 15 13:49 /dev/null crw------- 1 root root 1, 3 Apr 15 13:49 /dev/null S99rmnologin crw------- 1 root root 1, 3 Apr 15 13:49 /dev/null crw------- 1 root root 1, 3 Apr 15 13:49 /dev/null It's udev script again. Ok, then I tried debugging it: #!/bin/sh -e # init script for udev # Edited by turncoat for debugging # Check the package is still installed [ -x /sbin/udevd ] || exit 0 # Get LSB functions echo 1: `ls -l /dev/null` >> /tmp/null-edit.log; . /lib/lsb/init-functions echo 2: `ls -l /dev/null` >> /tmp/null-edit.log; . /etc/default/rcS case "$1" in start) # We need the uevent support introduced in 2.6.15, bail out if we # don't have it and fall back to a static /dev if [ ! -f /sys/class/mem/null/uevent ]; then if mountpoint -q /dev; then # uh-oh, initramfs made some kind of /dev, get rid of it umount -l /dev/.static/dev umount -l /dev fi exit 1 fi echo 3: `ls -l /dev/null` >> /tmp/null-edit.log; if ! mountpoint -q /dev; then # initramfs didn't mount /dev, so we'll need to do that mount -n --bind /dev /etc/udev mount -n -t tmpfs -o mode=0755 udev /dev mkdir -m 0700 -p /dev/.static/dev mount -n --move /etc/udev /dev/.static/dev fi echo 4: `ls -l /dev/null` >> /tmp/null-edit.log; # Copy over default device tree cp -a -f /lib/udev/devices/* /dev # It's all over netlink now if [ -e /proc/sys/kernel/hotplug ]; then echo "" > /proc/sys/kernel/hotplug fi echo 5: `ls -l /dev/null` >> /tmp/null-edit.log; # Start udevd log_begin_msg "Starting kernel event manager..." if start-stop-daemon --start --quiet --exec /sbin/udevd -- --daemon; then log_end_msg 0 else log_end_msg $? fi echo 6: `ls -l /dev/null` >> /tmp/null-edit.log; # This next bit can take a while if type usplash_write >/dev/null 2>&1; then usplash_write "TIMEOUT 360" ||: trap "usplash_write 'TIMEOUT 15' ||:" 0 fi echo 7: `ls -l /dev/null` >> /tmp/null-edit.log; # Log things that udevtrigger does (won't work on seb's machine) if [ -x /usr/sbin/udevmonitor ]; then /usr/sbin/udevmonitor -e >/dev/.udev.log & UDEV_MONITOR_PID=$! fi echo 8: `ls -l /dev/null` >> /tmp/null-edit.log; # Fix permissions and missing symlinks/programs for devices made in # initramfs, and catch up on everything we missed log_begin_msg "Loading hardware drivers..." /sbin/udevtrigger echo 8-1: `ls -l /dev/null` >> /tmp/null-edit.log; if /sbin/udevsettle; then log_end_msg 0 echo 8-1-if: `ls -l /dev/null` >> /tmp/null-edit.log; else log_end_msg $? echo 8-1-else: `ls -l /dev/null` >> /tmp/null-edit.log; fi echo 9: `ls -l /dev/null` >> /tmp/null-edit.log; # Kill the udevmonitor again if [ -n "$UDEV_MONITOR_PID" ]; then kill $UDEV_MONITOR_PID fi echo 10: `ls -l /dev/null` >> /tmp/null-edit.log; ;; stop) log_begin_msg "Stopping kernel event manager..." echo stop1: `ls -l /dev/null` >> /tmp/null-edit.log; if start-stop-daemon --stop --quiet --oknodo --exec /sbin/udevd; then echo stop-if: `ls -l /dev/null` >> /tmp/null-edit.log; log_end_msg 0 else echo stop-else: `ls -l /dev/null` >> /tmp/null-edit.log; log_end_msg $? fi echo stop-fi: `ls -l /dev/null` >> /tmp/null-edit.log; umount -l /dev/.static/dev echo stop-static: `ls -l /dev/null` >> /tmp/null-edit.log; umount -l /dev echo stop-dev: `ls -l /dev/null` >> /tmp/null-edit.log; ;; restart) cp -au /lib/udev/devices/* /dev log_begin_msg "Loading additional hardware drivers..." /sbin/udevtrigger if /sbin/udevsettle; then log_end_msg 0 else log_end_msg $? fi ;; reload|force-reload) log_begin_msg "Reloading kernel event manager..." if start-stop-daemon --stop --signal 1 --exec /sbin/udevd; then log_end_msg 0 else log_end_msg $? fi ;; *) echo "Usage: /etc/init.d/udev {start|stop|restart|reload|force-reload}" exit 1 ;; esac exit 0 After restarting and going to runlevel 2, file /tmp/null-edit.log had: 1: crw-rw---- 1 root root 1, 3 Jul 10 2007 /dev/null 2: crw-rw---- 1 root root 1, 3 Jul 10 2007 /dev/null 3: crw-rw---- 1 root root 1, 3 Jul 10 2007 /dev/null 4: crw-rw---- 1 root root 1, 3 Jul 10 2007 /dev/null 5: crw------- 1 root root 1, 3 Apr 15 13:49 /dev/null 6: crw------- 1 root root 1, 3 Apr 15 13:49 /dev/null 7: crw------- 1 root root 1, 3 Apr 15 13:49 /dev/null 8: crw------- 1 root root 1, 3 Apr 15 13:49 /dev/null 8-1: crw------- 1 root root 1, 3 Apr 15 13:49 /dev/null 8-1-if: crw-rw-rw- 1 root root 1, 3 Apr 15 13:49 /dev/null 9: crw-rw-rw- 1 root root 1, 3 Apr 15 13:49 /dev/null 10: crw-rw-rw- 1 root root 1, 3 Apr 15 13:49 /dev/null So, file /dev/null get created with 660 (?) permissions, then got changed to 600 after the line cp -a -f /lib/udev/devices/* /dev ( the file /lib/udev/devices/null has 600 permissions ) And then it got fixed to 666 by the /sbin/udevsettle program. The strangest thing happened - the /dev/null got right permissions. After I changed back udev script to it's original, I got 600 on /dev/null again! I suppose that it's some race condition problem with udev and udevsettle program, since it work right when I put the echo between running /sbin/udevtrigger and /sbin/udevsettle (or something similar). My workaround is to put 666 to /lib/udev/devices/null , so when it get's copied to /dev/null it has the right permissions, weather or not the udevsettle do it's job or not. But this is only workaround for /dev/null - who knows what other problems will occur if my suspicions about race condition are right. If you need any more information or testing from me, I will be happy to try something more. I'll keep this machine in this state for one more week, and then I will reinstall it. Cheers! Marko