parted calls 'udevadm settle' after every change causing it to hang for 180s if no udevd with matching magic key is running.

Bug #664115 reported by Al Stone on 2010-10-20
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OEM Priority Project
High
Al Stone
parted (Ubuntu)
Medium
Colin Watson
Lucid
Medium
Colin Watson
Maverick
Medium
Colin Watson

Bug Description

parted can take a very long time when run within a chroot.

The issue here is an interface mismatch between udevd and udevadm.

'udevadm settle' works by sending a netlink message to the udevd process. However, the magic key (UDEV_MONITOR_MAGIC) udevd uses to communicate w/ udevadm changed between lucid and maverick (presumably due to a non-backwards-compatible interface change). So, if you are running a maverick system and try to run lucid's udevadm in a chroot, there will be no compatible udevd to respond. The long delay is due to a hardcoded timeout default in udevadm - it will wait up to 180s for udevd to signal settle completion.

Development branch: Fixed in parted 2.3-4ubuntu2 by not calling 'udevadm settle' if we're chrooted.

TEST CASE: I verified this by debootstrapping both a lucid and a maverick chroot on a maverick host. In a pristine host config:

   chroot /chroot/lucid udevadm settle -> hang
   chroot /chroot/maverick udevadm settle -> no hang

If I modify the host's udev to listen to lucid's UDEV_MONITOR_MAGIC define, I get the inverse results.

So how does live-helper trigger this? parted has an ubuntu-specific patch that calls 'udevadm settle' after manipulating partitions to avoid races with other applications that might be triggered by partition changes.

Al Stone (ahs3) wrote :
summary: - choot loop devices stall for extremely long periods
+ chroot loop devices stall for extremely long periods

The issue here is as actually an interface mismatch between udevd and udevadm.

'udevadm settle' works by sending a netlink message to the udevd process. However, the magic key (UDEV_MONITOR_MAGIC) udevd uses to communicate w/ udevadm changed between lucid and maverick (presumably due to a non-backwards-compatible interface change). So, if you are running a maverick system and try to run lucid's udevadm in a chroot, there will be no compatible udevd to respond. The long delay is due to a hardcoded timeout default in udevadm - it will wait up to 180s for udevd to signal settle completion.

I verified this by debootstrapping both a lucid and a maverick chroot on a maverick host. In a pristine host config:

   chroot /chroot/lucid udevadm settle -> hang
   chroot /chroot/lucid udevadm settle -> no hang

If I modify the host's udev to listen to lucid's UDEV_MONITOR_MAGIC define, I get the inverse results.

So how does live-helper trigger this? parted has an ubuntu-specific patch that calls 'udevadm settle' after manipulating partitions to avoid races with other applications that might be triggered by partition changes.

In my opinion, the right place to fix this is in live-helper, since it could use the host's parted (and therefore the host's udevadm) to tweak devices in the chroot.

Al Stone (ahs3) wrote :

Re-assigning to live-build

affects: coreutils (Ubuntu) → live-build (Ubuntu)
Al Stone (ahs3) wrote :

Attaching a patch that changes which parted is used by lh, thus avoiding the udev problems previously mentioned in this bug.

The bug isn't in live-build, even if we can work around this issue with a hack, and there is no way upstream will accept this patch. Reassigning to udev.

affects: live-build (Ubuntu) → udev (Ubuntu)
tags: added: patch
tags: added: oem-services
description: updated
Steve Magoun (smagoun) wrote :

Marking oem-priority; this dramatically increases (order of magnitude) the time needed to build a Maverick or Natty OS image in our infrastructure.

Changed in oem-priority:
status: New → Confirmed
importance: Undecided → High
Pedro Villavicencio (pedro) wrote :

Thanks for the report and the patch, Could someone on the foundations team have a look into it? Thanks a lot.

Changed in udev (Ubuntu):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
importance: Undecided → Medium
status: New → Triaged

Deleting patch from bug since its already been rejected.

tags: added: natty
removed: amd64 apport-bug patch
Changed in udev (Ubuntu):
assignee: Canonical Foundations Team (canonical-foundations) → Colin Watson (cjwatson)
Colin Watson (cjwatson) wrote :

Sorry about the delay on this. My preference, I think, is simply to change parted to avoid calling 'udevadm settle' when in a chroot. The fact that it calls 'udevadm settle' is an Ubuntu patch for the benefit of our installer in the first place, and we never rely on calling it in a chroot. I'd be more comfortable with doing this than with going round chasing up everything that might call parted, or with trying to persuade udev upstream to maintain compatibility (which I sort of doubt they're going to care much about).

I'll get to this as soon as possible.

affects: udev (Ubuntu) → parted (Ubuntu)
Changed in parted (Ubuntu Maverick):
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → Colin Watson (cjwatson)
Colin Watson (cjwatson) wrote :

Incidentally, I did look at live-helper and it seems to me that it isn't going to run across the races that the 'udevadm settle' call was originally introduced to prevent, so I believe that my fix is safe from that point of view.

Colin Watson (cjwatson) on 2010-12-06
description: updated
description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package parted - 2.3-4ubuntu2

---------------
parted (2.3-4ubuntu2) natty; urgency=low

  * Don't call 'udevadm settle' when chrooted (LP: #664115).
 -- Colin Watson <email address hidden> Mon, 06 Dec 2010 16:29:47 +0000

Changed in parted (Ubuntu):
status: Triaged → Fix Released

Accepted parted into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in parted (Ubuntu Maverick):
status: Triaged → Fix Committed
tags: added: verification-needed

I tested this update and can confirm that the hanging behaviour described in this bug no longer occurs with the proposed update installed.

summary: - chroot loop devices stall for extremely long periods
+ parted calls 'udevadm settle' after every change causing it to hang for
+ 180s if no udevd with matching magic key is running.
Martin Pitt (pitti) on 2011-01-15
tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package parted - 2.3-2ubuntu2

---------------
parted (2.3-2ubuntu2) maverick-proposed; urgency=low

  * Don't call 'udevadm settle' when chrooted (LP: #664115).
 -- Colin Watson <email address hidden> Mon, 06 Dec 2010 16:47:27 +0000

Changed in parted (Ubuntu Maverick):
status: Fix Committed → Fix Released
Colin Watson (cjwatson) on 2011-01-20
Changed in parted (Ubuntu Lucid):
assignee: nobody → Colin Watson (cjwatson)
importance: Undecided → Medium
milestone: none → ubuntu-10.04.3
status: New → Triaged
Chris Van Hoof (vanhoof) on 2011-06-20
Changed in oem-priority:
status: Confirmed → In Progress
Colin Watson (cjwatson) on 2011-06-30
Changed in parted (Ubuntu Lucid):
status: Triaged → In Progress
Chris Halse Rogers (raof) wrote :

SRU team ack. Please accept into lucid-proposed.

Martin Pitt (pitti) wrote :

Accepted parted into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in parted (Ubuntu Lucid):
status: In Progress → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Chris Van Hoof (vanhoof) wrote :

Hi Al -- Mind giving this one a test on Lucid now that an updated parted is in -proposed?

Cheers,
Chris

Changed in oem-priority:
status: In Progress → Incomplete
assignee: nobody → Al Stone (ahs3)
Al Stone (ahs3) wrote :

I can no longer get the original behavior to re-occur. Looks like we're good with this one, Chris.

Martin Pitt (pitti) on 2011-07-19
tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package parted - 2.2-5ubuntu5.2

---------------
parted (2.2-5ubuntu5.2) lucid-proposed; urgency=low

  * Don't call 'udevadm settle' when chrooted (LP: #664115).
 -- Colin Watson <email address hidden> Thu, 30 Jun 2011 09:11:45 +0100

Changed in parted (Ubuntu Lucid):
status: Fix Committed → Fix Released
Changed in oem-priority:
status: Incomplete → Fix Released
tags: added: testcase
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers