rename(/dev//.udev-tmp, /dev//) failed: Invalid cross-device link

[Alvin]

Binary package hint: udev

While booting, a lot of errors from udevd-work are visible. This is after installation of Karmic beta with LVM (alternate cd)

$ grep udev /var/log/daemon.log
Oct 6 15:20:09 pc-alvin udevd-work[713]: rename(/dev//.udev-tmp, /dev//) failed: Invalid cross-device link

(Because of bug 328881 I can not give more information)

ProblemType: Bug
Architecture: amd64
Date: Tue Oct 6 15:40:59 2009
DistroRelease: Ubuntu 9.10
NonfreeKernelModules: nvidia
Package: udev 147~-5
ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.31-11-generic root=/dev/mapper/vg0-root ro quiet splash
ProcEnviron:
 LANGUAGE=
 PATH=(custom, user)
 LANG=C
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-11.38-generic
SourcePackage: udev
Uname: Linux 2.6.31-11-generic x86_64
dmi.bios.date: 04/21/2009
dmi.bios.vendor: Intel Corp.
dmi.bios.version: PRG3110H.86A.0065.2009.0421.1559
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: DG31PR
dmi.board.vendor: Intel Corporation
dmi.board.version: AAD97573-301
dmi.chassis.type: 3
dmi.modalias: dmi:bvnIntelCorp.:bvrPRG3110H.86A.0065.2009.0421.1559:bd04/21/2009:svntranstecAG:pn:pvr:rvnIntelCorporation:rnDG31PR:rvrAAD97573-301:cvn:ct3:cvr:
dmi.sys.vendor: transtec AG

[Daniel Hahler (blueyed)]

I can confirm this, this happens as of upgrades from yesterday.
I'll attach screenshots and will try with latest updates after installing them (via LiveCD).

[Daniel Hahler (blueyed)]

It appears to be fixed somehow for me, now I'm running into bug 442495 - while I have no invalid UUIDs, but cryptsetup appears to not setup them (bug 445894).

[Scott James Remnant (Canonical) (canonical-scott)]

Since this appears to have gone away all on its own, I'll mark the bug as Invalid

[Alvin (alvind)]

Reopening since the errors are still there.

[Daniel Hahler (blueyed)]

Alvin, have you tried it with the latest updates?
I guess the kernel updates might have fixed it (or actually made it worse). It might be that bug 442495 happens for me now, before I can run into this one.
I'll try with the problematic fstab entries removed later, and will stop to spam this report for now.

[Daniel Hahler (blueyed)]

I'm getting those still.. they have been on another VT8 before, and I've missed them.

I've finally managed to get my system booting now again, so these errors are not fatal (AFAICS).

[Christoph Langner (chrissss)]

I can confirm this issue too, i upgraded one of my systems to karmic, and the boot process stops with

> rename(/dev//.udev-tmp, /dev//) failed: Invalid cross-device link

too...

[Kees Cook (kees)]

This appears for me when manipulating LVM snapshots.

[Philipp Wendler (philw85)]

This bug affects me, too. I upgraded yesterday from Jaunty to Karmic with the update-manager. Since then, a lot of these udev errors appear while booting, but boot is successful. I'm using LVM for everything except /boot, but without snapshots (but with mirrored LVs). If you need more information, I'll be happy to supply logs etc, whatever you need.

I consider this bug important, because after this messages, there is a time in which nothing happens. This might lead to users thinking that Ubuntu does not work on their machine.

[Jean-Damien Bouvier (jdbouvier)]

Just like PhilW, I upgraded from Jaunty to Karmic with the update-manager and I get all those "Invalid cross-device link" error messages. I'm using LVM (with mirrored Lvs) for everything but the boot.

Around one out of two boot process is successful, although not critical I must say that this is quite irritating.

Let me know if you need some logs or other information.

[Philipp Wendler (philw85)]

I'm surprised that sometimes the boot process is not successful, because here it succeeds every time. Sometimes it takes a little bit longer, sometimes it shows the message that it is waiting for a few logical volumes to turn up, and sometimes a recovery console is started, because the root partition is missing. If i exit from the console, the computer reboots, but if I do nothing, it continues booting quite fine after a few seconds.

[Jean-Damien Bouvier (jdbouvier)]

Sometimes it is waiting for a few logical volumes to turn up but it seems that nothing happens and a few seconds later, I reboot the computer. Sometimes I get the recovery console ( a message about a script running and control D to escape ) but there again nothing happens. From what you tell me, I guess that I'm not patient enough. I'll try to wait one or two minutes to see if it's OK.

[Jean-Damien Bouvier (jdbouvier)]

So far, I guess that the boot really fails.
I get the following message :
Mount of root filesystem failed.
A maintenance shell will now be started
CONTROL-D will terminate this shell and reboot the system

I wait for 15 minutes but nothing happened.

Another time, I get "One or more of the mounts listed in /etc/fstab cannot yet be mounted" followed by the list of all my LVs and again nothing happens.

I'm going to make a new install of ubuntu.

[Alvin (alvind)]

PhilW and Jean-Damien Bouvier, I can confirm your problem of the 'sometimes missing root partition', but that is not this bug. (Point me in the right direction of you know what bug it is, because it is a pretty critical one.)

[Philipp Wendler (philw85)]

Well, I see and know nothing except the many udevd-work errors and the sometimes (not frequently) delayed availability of the logical volumes. But I'm willing to investigate further, and I already have a little bit of experience in working (and debugging) of initial ramdisks. So if you could tell me what to do, which information you need or in which direction I should investigate, I would try.

[Jürgen Kreileder (jk)]

(Coming from bug 483890)

I only see this happening when creating/removing snapshots for backup not while booting. In addition to "rename(/dev//.udev-tmp, /dev//) failed..." I also get "rmdir(/dev/) failed: Device or resource busy" and some error messages when creating the snapshots:

  Path /dev/DEVTYPE=_disk_ no longer valid for device(253,18)
  Path /dev/PWD=_/_ no longer valid for device(253,18)
  Path /dev/DM_STATE_ACTIVE=_1_ no longer valid for device(253,18)
  Path /dev/SUBSYSTEM=_block_ no longer valid for device(253,18)
  Path /dev/DM_TABLE_LIVE=_1_ no longer valid for device(253,18)
  Path /dev/IFS=_ no longer valid for device(253,18)
  Path /dev/DM_TARGET_COUNT=_1_ no longer valid for device(253,18)
  Path /dev/UDEV_LOG=_3_ no longer valid for device(253,18)
  Path /dev/PS1=_# no longer valid for device(253,18)
  Path /dev/_ no longer valid for device(253,18)
  Path /dev/PS2=__ no longer valid for device(253,18)
  Path /dev/DM_TARGET_TYPES=_linear_ no longer valid for device(253,18)
  Path /dev/DM_STATE=_ACTIVE_ no longer valid for device(253,18)
  Path /dev/PATH=_/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin_ no longer valid for device(253,18)
  Path /dev/PS4=_+ no longer valid for device(253,18)
  Path /dev/OPTIND=_1_ no longer valid for device(253,18)
  Path /dev/DM_LAST_EVENT_NR=_0_ no longer valid for device(253,18)
  Path /dev/DM_MAJOR=_253_ no longer valid for device(253,18)
  Path /dev/MAJOR=_253_ no longer valid for device(253,18)

I can get rid of those by pausing a few seconds between creating the individual snapshots. This just leaves the rename/rmdir entries in syslog then.
The snapshots seems to work fine but I'm not 100% sure if I really should trust the system.

[ppww (pete-launchpad)]

This occurs for me (many "rename(/dev//.udev-tmp, /dev//) failed: Invalid cross-device link" errors) at boot, if LVM2 snapshots exist. In my case, I often make snapshots of / and /var so that rollback of package updates is possible.

[Jürgen Kreileder (jk)]

While doing snapshots for backups I also this occasionally among the other errors:

upstart-udev-bridge[25372]: Disconnected from Upstart
upstart-udev-bridge main process (25372) terminated with status 1
upstart-udev-bridge main process ended, respawning

[Stephan Rütten (stephanrutten)]

I also see these (usually 4 lines of) messages during each boot on a Karmic (i386 alternate cd) install (no upgrade) using LVM2 with snapshots.

From /var/log/daemon.log:

Dec 2 06:53:28 laptop udevd-work[699]: rename(/dev//.udev-tmp, /dev//) failed: Invalid cross-device link

[Stephan Rütten (stephanrutten)]

After removing the snapshot volume I have not seen this error again.
Not during boot and not in the log files.

[Friedrich Schäuffelhut (fries)]

I can confirm this.
Before updating to 9.10 I took a snapshot of my root partition. From then on I kept seeing the 'failed: Invalid cross-device link' message and experienced delays of several seconds each time the message was printed during a boot.
After reading this thread I removed the snapshot and the problem went away.

[Alvin (alvind)]

Removing snapshots also removes these messages, but they come back after taking new snapshots.

[Sakari Maaranen (sam-iki)]

This bug also appears when using LVM mirrors at least when resyncing.

For testing purposes I have created six LVM partitions on two physical drives:

sam@karhuherra:~$ sudo lvs -a -o +devices
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices

  home karhu mwi-ao 37.25G 22.16 home_mimage_0(0),home_mimage_1(0)
  [home_mimage_0] karhu Iwi-ao 37.25G /dev/sdb1(5482)
  [home_mimage_1] karhu Iwi-ao 37.25G /dev/sda1(5482)

  opt karhu mwi-ao 2.79G 100.00 opt_mimage_0(0),opt_mimage_1(0)
  [opt_mimage_0] karhu iwi-ao 2.79G /dev/sdb1(4767)
  [opt_mimage_1] karhu iwi-ao 2.79G /dev/sda1(238)

  root karhu mwi-ao 952.00M 100.00 root_mimage_0(0),root_mimage_1(0)
  [root_mimage_0] karhu iwi-ao 952.00M /dev/sdb1(0)
  [root_mimage_1] karhu iwi-ao 952.00M /dev/sda1(0)

  swap karhu mwi-ao 9.31G 89.09 swap_mimage_0(0),swap_mimage_1(0)
  [swap_mimage_0] karhu Iwi-ao 9.31G /dev/sdb1(2383)
  [swap_mimage_1] karhu Iwi-ao 9.31G /dev/sda1(2383)

  usr karhu mwi-ao 5.59G 100.00 usr_mimage_0(0),usr_mimage_1(0)
  [usr_mimage_0] karhu iwi-ao 5.59G /dev/sdb1(238)
  [usr_mimage_1] karhu iwi-ao 5.59G /dev/sda1(953)

  var karhu mwi-ao 2.79G 100.00 var_mimage_0(0),var_mimage_1(0)
  [var_mimage_0] karhu iwi-ao 2.79G /dev/sdb1(1668)
  [var_mimage_1] karhu iwi-ao 2.79G /dev/sda1(4767)

These partitions will be resynced on every boot, because they have been created using lvcreate -m1 --corelog.

This resync produces several times the error message
udevd-work[***]: rename(/dev//.udev-tmp, /dev//) failed: Invalid cross-device link

Most often the boot is successful, but sometimes it fails in different ways, hangs, fails to bring up bridged networking, fails to autostart virtual guests. Since this is a test setup for a new server and I'm getting all these problems, I'm considering changing to some other distro especially if Ubuntu is giving these problems low priorities.

[Philipp Wendler (philw85)]

The bug also appears with LVM mirrors without resyncing. I have no snapshots, only two mirrors with their log on the disk.

But I must say, that since the last time I commented on this bug, all boots were successful for me. I always get the messages and there is a little time in which nothing happens, but there are no failed boots / rescue shells anymore for me. But I don't know why.

[Nikolai Bochev (n-bochev)]

I can confirm this is happening to me on a running machine, making it hang ( not responding to pings whatsoever ).
What i have is a script that creates a snapshot of a lvm volume, mounts it and starts a backup program, which then backups on another drive.

It's not any of the usual system mounts ( i.e. it's /srv/storage ).

Mar 7 03:00:02 blackwing udevd-work[32289]: rename(/dev//.udev-tmp, /dev//) failed: Invalid cross-device link
Mar 7 03:00:02 blackwing udevd-work[9338]: rename(/dev//.udev-tmp, /dev//) failed: Invalid cross-device link
Mar 7 03:00:02 blackwing udevd-work[22424]: rename(/dev//.udev-tmp, /dev//) failed: Invalid cross-device link
Mar 7 03:00:02 blackwing udevd-work[9338]: rename(/dev//.udev-tmp, /dev//) failed: Invalid cross-device link
Mar 7 03:00:02 blackwing udevd-work[32289]: rename(/dev//.udev-tmp, /dev//) failed: Invalid cross-device link
Mar 7 03:54:48 blackwing udevd-work[22429]: rename(/dev//.udev-tmp, /dev//) failed: Invalid cross-device link
Mar 7 03:54:48 blackwing udevd-work[22429]: rmdir(/dev/) failed: Device or resource busy
Mar 7 03:54:48 blackwing udevd-work[22429]: rmdir(/dev/) failed: Device or resource busy

The first events are when the backup starts ( 03:00:02 ). The second events are when the backup ends and the snapshot volume gets unmounted ( 03:54:48 ). After this point i loose connectivity with the machine.

The snapshot volume is never present at boot so i don't have any boot problems, but those hangs are getting very annoying since this is a live production server.

I don't understand why this is marked low, as it seems to be critical for the operation of the servers.

[frell (lee)]

Im seeing the same thing when creating and removing an lvm snapshots.

However it doesn't seem to effect the system or use of the lvm in my case.

[Jürgen Kreileder (jk)]

Has anyone tested whether this is fixed in Lucid?

It would be a shame to make an LTS release with this bug.
It's slightly frightening to see /var/log/syslog filled by more than 50% by these messages on servers that use snapshots for backups (I have that on some Karmic machines). It might be a harmless race condition but it should be fixed.

[K Richard Pixley (rich-noir)]

I'm seeing similar problems in a similar situation. I'm not doing backups but I am creating and removing a sequence of lv's and snapshots. Mine are on amd64 karmic. I don't believe I've seen these on lucid with the same code, (which only proves I haven't done enough testing :).

If this is a harmless race condition, it would be nice to have that validated and documented.

[K Richard Pixley (rich-noir)]

If this is the same bug as https://bugs.launchpad.net/bugs/483890, and I'm not sure that it is, then it is still in lucid. Lucid is apparently better, but not fixed.

I ran some tests over the last few days. Using the script I attached to 483890 I can reproduce the problem in seconds on karmic. Here's a cross section.

All tests on identical vmwares. Most tests run 7 - 14 min. Fedora
takes almost an hour.

ubuntu-10.04 beta1 server amd64 - 300 reps, 6 errors
ubuntu-10.04 beta1 server i386 - 300 reps, 7 errors

ubuntu-9.10 server amd64 - 300 reps, 1667 errors
ubuntu-9.10 server i386 - 300 reps, 878 errors

ubuntu-9.04 server amd64 - 300 reps, 0 errors
ubuntu-9.04 server i386 - 300 reps, 0 errors

debian-5.04 (lenny) amd64 - 300 reps, 0 errors
debian-5.04 (lenny) i386 - 300 reps, 0 errors

fedora-12 x86_64 - 300 reps, 0 errors
fedora-12 i386 - 300 reps, 0 errors

[Wessel Dankers (wsl)]

I think I got it. In /lib/udev/rules.d/65-dmsetup.rules there's this line:

ENV{DM_UUID}=="LVM-*", PROGRAM="/bin/sh -c 'set `lvm lvdisplay -C -o vg_name,lv_name --noheadings /dev/$name` ; echo $1/$2'", SYMLINK+="$result"

which is a particularly fragile piece of shell script that doesn't do any error checking. Replacing it with:

ENV{DM_UUID}=="LVM-*", PROGRAM="/bin/sh -ec 'link=$(lvm lvdisplay -C -o vg_name,lv_name --noheadings /dev/$name); set -- $link; echo $1/$2'", SYMLINK+="$result"

will fix the errors and warnings. No more empty or bogus symlinks.

[Wessel Dankers (wsl)]

And even that wasn't enough to stop all the warnings. Apparently lvdisplay doesn't set an error code if there were zero results. The following should catch that situation and stop the last of it:

ENV{DM_UUID}=="LVM-*", PROGRAM="/bin/sh -ec 'set -- $(lvm lvdisplay -C -o vg_name,lv_name --noheadings /dev/$name); echo ${2+$1/$2}'", SYMLINK+="$result"

[Rolf Leggewie (r0lf)]

karmic has seen the end of its life and is no longer receiving any updates. Marking the karmic task for this ticket as "Won't Fix".