Installer warns "Could not stat device /dev/mapper/hostname-root - No such file or directory", works after retry

[Sgifford]

[Impact]
Installs using LVM can fail due to a race with the message "Could not stat device /dev/mapper/$hostname-root - No such file or directory". This is very reproducible on some systems, but not others (presumably due to timing). I've had good luck reproducing w/ a kvm VM. During manual installs this is a fairly minor error - selecting "Retry" allows the install to continue. However, this is a fatal error on automated installs.
[Test Case]
Do an LVM install using d-i - manual or preseeded and look for the above error message.
[Regression Potential]
This introduced a new call into the udev stack, so a bug there could trip us up - but, it seems like if any such bug existed, we'd hit it already w/ one of the other settle calls.

[Sgifford (sgifford)]

The fix that worked for me was to add:

  sleep 1

after both of the "lv_create" commands in "/bin/perform_recipe_by_lvm".

With that fix, the error goes away, and everything seems to work properly.

[Sgifford (sgifford)]

I originally reported this as part of bug #922438, but investigation revealed it to be a separate issue.

[Sgifford (sgifford)]

Here is a short video demonstrating the problem and the fix.

[dann frazier (dannf)]

I think this is now fixed in Debian's git tree. Would you be able to verify? Let me know if you need any help with doing so.

commit 7d03d5e8fee63ff09716f9979ff5c9893d234396
Author: dann frazier <email address hidden>
Date: Mon May 7 14:07:44 2012 -0700

    Call update-dev --settle between creating lvs and accessing them

[Sgifford (sgifford)]

I tried testing out the final release of Precise Pangolin, and instead of this issue, I encountered another error with root on LVM, which I have reported as bug #1000770.

I am not sure of a straightforward way to test out the fix in Debian's git tree. I am familiar with using git and compiling software, but am not familiar enough with the installation process to know how to get the dpkg into the installer. Maybe I could pause the installer before partitioning, download an updated dpkg I build, apt-get install it into the running image, then continue with the installation? Any tips are appreciated.

Thanks!

[dann frazier (dannf)]

On Thu, May 17, 2012 at 02:21:56PM -0000, Sgifford wrote:
> I tried testing out the final release of Precise Pangolin, and instead
> of this issue, I encountered another error with root on LVM, which I
> have reported as bug #1000770.
>
> I am not sure of a straightforward way to test out the fix in Debian's
> git tree. I am familiar with using git and compiling software, but am
> not familiar enough with the installation process to know how to get the
> dpkg into the installer. Maybe I could pause the installer before
> partitioning, download an updated dpkg I build, apt-get install it into
> the running image, then continue with the installation? Any tips are
> appreciated.

Thank you for working with us to verify this fix!

The easiest way to verify this might be using the method you used in
the video would be sufficient. But, instead of adding sleeps, you
could add the 'update-dev --settle' line as shown here:

  http://anonscm.debian.org/gitweb/?p=d-i/partman-auto-lvm.git;a=commitdiff;h=7d03d5e8fee63ff09716f9979ff5c9893d234396

Or, to avoid having to dropping to a shell, you could build a custom
udeb and insert it into your initrd.gz file with these commands:

$ originitrd="/path/to/initrd.gz"
$ newinitrd="/path/to/newinitrd.gz"
$ sudo apt-get build-dep partman-auto-lvm
$ bzr branch lp:~dannf/partman-auto-lvm/add-settle partman-auto-lvm
$ cd partman-auto-lvm
$ dpkg-buildpackage -rfakeroot -uc -us
$ cd -
  (gunzip < $originitrd && \
   echo partman-auto-lvm_42ubuntu2+dannf1_all.udeb | \
   cpio -H newc -o) | gzip > $newinitrd

And finally, add this to your preseed file, so that this custom udeb
gets deployed:

d-i preseed/early_command string udpkg -i /*.udeb

You can verify this worked by dropping to a shell and running:
 # grep update-dev /bin/perform_recipe_by_lvm

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[dann frazier (dannf)]

fyi, just uploaded partman-auto-lvm_43 to sid, which should fix this issue.

[Launchpad Janitor (janitor)]

This bug was fixed in the package partman-auto-lvm - 44ubuntu1

---------------
partman-auto-lvm (44ubuntu1) quantal; urgency=low

  * Resynchronise with Debian (update-dev change in version 43 fixes
    LP: #964957). Remaining changes:
    - Accept autopartitioning automatically.
    - Change fallback VG name to Ubuntu.
    - Ask how much of the VG should be used for logical volumes, rather than
      unconditionally using it all.

partman-auto-lvm (44) unstable; urgency=low

  * Team upload
  * Replace XC-Package-Type by Package-Type

  [ Updated translations ]
  * Welsh (cy.po) by Dafydd Tomos
  * Lithuanian (lt.po) by Rimas Kudelis
  * Latvian (lv.po) by Rūdolfs Mazurs

partman-auto-lvm (43) unstable; urgency=low

  [ dann frazier ]
  * Call update-dev --settle between creating lvs and accessing them

  [ Updated translations ]
  * bo (bo.po) by Tennom
  * Macedonian (mk.po) by Arangel Angov
  * Punjabi (Gurmukhi) (pa.po) by A S Alam
  * Portuguese (Brazil) (pt_BR.po) by Felipe Augusto van de Wiel (faw)
 -- Colin Watson <email address hidden> Thu, 21 Jun 2012 11:49:06 +0100

[Dimitri John Ledkov (xnox)]

Thanks for reminding about this bug. I will look into sponsoring your debdiff into Precise SRU, looks good.

[Dimitri John Ledkov (xnox)]

Uploaded into precise-proposed, now pending on sru team to accept this package.

[Clint Byrum (clint-fewbar)]

Hello Sgifford, or anyone else affected,

Accepted partman-auto-lvm into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/partman-auto-lvm/42ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[dann frazier (dannf)]

I've verified that I can still reproduce this problem intermittently when not using apt-setup/proposed=true (which pulls in partman-auto-lvm 42ubuntu2.1), but that I can no longer reproduce this problem when using apt-setup/proposed=true (which pulls in partman-auto-lvm 42ubuntu2.2). Since I'm only observing the problem intermittently, I can't say without a doubt that the problem is no longer there, but I can say I've seen neither this bug nor any additional problems w/ 42ubuntu2.2.

[Colin Watson (cjwatson)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package partman-auto-lvm - 42ubuntu2.2

---------------
partman-auto-lvm (42ubuntu2.2) precise-proposed; urgency=low

  * Call update-dev --settle between creating lvs and accessing them
    (LP: #964957).
 -- Dann Frazier <email address hidden> Tue, 02 Oct 2012 10:09:12 -0600