Intermittent mount failure after growroot

Bug #765843 reported by Neil Wilson on 2011-04-19
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-initramfs-tools (Ubuntu)
High
Unassigned
Natty
High
Unassigned
initramfs-tools (Ubuntu)
Undecided
Unassigned
Natty
Undecided
Unassigned

Bug Description

Natty 11.04 Final beta

cloud-initramfs-growroot:
  Installed: 0.1ubuntu1

Getting intermittent mount failures after 'grow root' on an amd64 image. About 1 in 10 fail.

The 'wait-for-root' condition passes, yet the mount still fails with a '/dev/disk/by-uuid/xxx: No such file or directory' error.

It doesn't seem to happen at all on an i386 image.

These images are launched with kvm.

Neil Wilson (neil-aldur) wrote :
Neil Wilson (neil-aldur) wrote :

Running the mount by hand from the busybox prompt works.

Neil Wilson (neil-aldur) wrote :

The 'wait-for-root' command won't guard for the events that the BLKRRPART ioctl generates.

BLKRRPART generates the following:

ubuntu@srv-hgww8:~$ sudo udevadm --debug monitor --udev
run_command: calling: monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing

UDEV [1303300813.480716] remove /devices/pci0000:00/0000:00:06.0/virtio3/block/vdb/vdb1 (block)
UDEV [1303300813.538753] change /devices/pci0000:00/0000:00:06.0/virtio3/block/vdb (block)
UDEV [1303300813.570692] add /devices/pci0000:00/0000:00:06.0/virtio3/block/vdb/vdb1 (block)

The 'wait-for-root' command is written presuming that there is no device entry in the database and you're just waiting for it to arrive. If it runs before the kernel has queued or processed the 'remove'event, then it will succeed in any event.

The mount will then fail if it runs after the 'remove' event but before the 'add' event completes.

Scott Moser (smoser) wrote :

Neil,
  Thank you for reporting this issue (and fixing it).
  I've pulled your fix into the "upstream" location at https://code.launchpad.net/~cloud-init-dev/ubuntu/natty/cloud-initramfs-tools/natty . I just pushed that right now.

  I've also pushed a build to my ppa at https://launchpad.net/~smoser/+archive/ppa .

  Could you test that and report back?
  It also has a change to rescuevol that should not affect growroot at all.

Scott Moser (smoser) wrote :

@Neil,
  Could you please let me know if the suggested patches / debs fix the problem?

Yep,

Testing now. Been on hols since before Easter

On 3 May 2011 14:19, Scott Moser <email address hidden> wrote:
> @Neil,
>  Could you please let me know if the suggested patches / debs fix the problem?
--
Neil Wilson

Neil Wilson (neil-aldur) wrote :

I've tested the rebuilt package from your ppa as best I can with launchpad's PPA builder in virtual stasis :)

The fixes seem to do the job - thanks for incorporating them.

Changed in cloud-initramfs-tools (Ubuntu Natty):
status: New → Confirmed
Changed in cloud-initramfs-tools (Ubuntu):
status: New → Triaged
Changed in cloud-initramfs-tools (Ubuntu Natty):
status: Confirmed → Triaged
Changed in initramfs-tools (Ubuntu):
status: New → Invalid
Changed in initramfs-tools (Ubuntu Natty):
status: New → Invalid
Clint Byrum (clint-fewbar) wrote :

It looks like this has been merged into Scott's upstream branch and will be uploaded to Oneiric. I'm not sure this patch can be incorporated into an SRU as-is, but it looks like Scott has tweaked it a bit to use wait-for-root *and* udevadm settle, which seems safer.

Once this has landed in Oneiric, the bit that fixes this bug only should be extracted and prepared as an SRU to natty.

Changed in cloud-initramfs-tools (Ubuntu Natty):
importance: Undecided → High
Changed in cloud-initramfs-tools (Ubuntu):
importance: Undecided → High
status: Triaged → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-initramfs-tools - 0.2ubuntu1

---------------
cloud-initramfs-tools (0.2ubuntu1) oneiric; urgency=low

  * invoke blkid to search for the rescue volume rather than relying on
    /dev/disk/by-label/LABEL . The by-label entries would sometimes
    not exist while the device would.
  * improve failed boot message
  * update debian/README.source to point at upstream location

  [Neil Wilson]
  * fix race condition that resulted in intermittent mount failures
    after 'growroot'. (LP: #765843)
 -- Scott Moser <email address hidden> Wed, 08 Jun 2011 09:46:57 -0400

Changed in cloud-initramfs-tools (Ubuntu):
status: Fix Committed → Fix Released
Neil Wilson (neil-aldur) on 2012-03-31
Changed in cloud-initramfs-tools (Ubuntu Natty):
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers