| 2015-06-04 22:38:42 |
Mike Pontillo |
bug |
|
|
added bug |
| 2015-06-04 22:38:42 |
Mike Pontillo |
attachment added |
|
Scott Moser's proposed fix + troubleshooting https://bugs.launchpad.net/bugs/1462139/+attachment/4410105/+files/curtin.patch |
|
| 2015-06-05 00:00:45 |
Launchpad Janitor |
branch linked |
|
lp:~mpontillo/curtin/fix-1462139 |
|
| 2015-06-05 13:48:56 |
Launchpad Janitor |
branch linked |
|
lp:curtin |
|
| 2015-06-05 13:49:48 |
Scott Moser |
curtin: status |
New |
Fix Committed |
|
| 2015-06-21 12:43:41 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/curtin |
|
| 2015-06-22 14:39:56 |
Launchpad Janitor |
branch linked |
|
lp:~smoser/ubuntu/trusty/curtin/trusty-sru |
|
| 2015-06-22 14:42:54 |
Scott Moser |
bug task added |
|
ubuntu (Ubuntu) |
|
| 2015-06-22 14:43:09 |
Scott Moser |
affects |
ubuntu (Ubuntu) |
curtin (Ubuntu) |
|
| 2015-06-22 14:43:09 |
Scott Moser |
curtin (Ubuntu): importance |
Undecided |
Medium |
|
| 2015-06-22 14:43:09 |
Scott Moser |
curtin (Ubuntu): status |
New |
Confirmed |
|
| 2015-06-22 14:43:29 |
Scott Moser |
nominated for series |
|
Ubuntu Trusty |
|
| 2015-06-22 14:43:29 |
Scott Moser |
bug task added |
|
curtin (Ubuntu Trusty) |
|
| 2015-06-22 14:43:29 |
Scott Moser |
nominated for series |
|
Ubuntu Vivid |
|
| 2015-06-22 14:43:29 |
Scott Moser |
bug task added |
|
curtin (Ubuntu Vivid) |
|
| 2015-06-22 14:43:39 |
Scott Moser |
curtin (Ubuntu Trusty): status |
New |
Confirmed |
|
| 2015-06-22 14:43:42 |
Scott Moser |
curtin (Ubuntu Vivid): status |
New |
Confirmed |
|
| 2015-06-22 14:43:44 |
Scott Moser |
curtin (Ubuntu Trusty): importance |
Undecided |
Medium |
|
| 2015-06-22 14:43:47 |
Scott Moser |
curtin (Ubuntu Vivid): importance |
Undecided |
Medium |
|
| 2015-06-22 16:12:18 |
Scott Moser |
description |
Here's the relevant part of the curtin output:
Installing for i386-pc platform.
Installation finished. No error reported.
umount: /tmp/tmpM4R1dI/target/dev: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
Unexpected error while running command.
Command: ['umount', '/tmp/tmpM4R1dI/target/dev']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'curthooks']
I spoke with Scott Moser on IRC about this, and he suggested the attached patch, which fixed the problem 100%.
I checked the installation output of a few of my MAAS nodes, and didn't see any lsof output. So I assume the "udevadm settle" command is the fix.
(Thanks Scott! I hope we can land this in time for MAAS 1.8.) |
=== Begin SRU Template ===
[Description]
A race condition can occur when invoking grub-install inside the target environment.
The failure case shows logs like this:
| Installing for i386-pc platform.
| Installation finished. No error reported.
| umount: /tmp/tmpM4R1dI/target/dev: device is busy.
| (In some cases useful info about processes that use
| the device is found by lsof(8) or fuser(1))
| Unexpected error while running command.
| Command: ['umount', '/tmp/tmpM4R1dI/target/dev']
| Exit code: 1
| Reason: -
| Stdout: ''
| Stderr: ''
| Installation failed with exception: Unexpected error while running command.
| Command: ['curtin', 'curthooks']
This is believed to be because some process (likely spawned by udev) has open filehandles on /dev when curtin went to clean up the target mounts.
The solution is to run 'udevadm settle' before unmounting '/dev/' from the target.
[Impact]
The impact is transient failure to install. This race condition is very rarely seen on hardware, but was somewhat easily reproduced in a heavily loaded vmware environment.
[Test Case]
In the original bug-opener's environment it fails fairly reliably under heavy host load using vmware. He would do a deploy to several guests on the same host at the same time and this would reproduce. Unfortunately I was unable to come up with a test case in a less complex environment.
[Regression Potential]
Regression potential should be very low here. The most likely fallout is just additional time for the install as a result of running 'udevadm settle'. A system that did not exhibit this bug will install a small fraction of a second slower.
$ sudo bash -c 'time for x in "$@"; do udevadm settle; done' -- $(seq 1 100)
real 0m0.214s
user 0m0.012s
sys 0m0.008s
As shown above, that is likely to be on the order of 1/100th of a second.
=== End SRU Template ===
Here's the relevant part of the curtin output:
Installing for i386-pc platform.
Installation finished. No error reported.
umount: /tmp/tmpM4R1dI/target/dev: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
Unexpected error while running command.
Command: ['umount', '/tmp/tmpM4R1dI/target/dev']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'curthooks']
I spoke with Scott Moser on IRC about this, and he suggested the attached patch, which fixed the problem 100%.
I checked the installation output of a few of my MAAS nodes, and didn't see any lsof output. So I assume the "udevadm settle" command is the fix.
(Thanks Scott! I hope we can land this in time for MAAS 1.8.) |
|
| 2015-06-22 17:02:18 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2015-06-24 18:37:04 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/wily-proposed/curtin |
|
| 2015-06-24 20:19:39 |
Chris J Arges |
curtin (Ubuntu Trusty): status |
Confirmed |
Fix Committed |
|
| 2015-06-24 20:19:42 |
Chris J Arges |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2015-06-24 20:19:43 |
Chris J Arges |
bug |
|
|
added subscriber SRU Verification |
| 2015-06-24 20:19:47 |
Chris J Arges |
tags |
|
verification-needed |
|
| 2015-06-24 20:22:59 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/trusty-proposed/curtin |
|
| 2015-06-24 20:27:41 |
Chris J Arges |
curtin (Ubuntu Vivid): status |
Confirmed |
Fix Committed |
|
| 2015-06-24 20:32:55 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/vivid-proposed/curtin |
|
| 2015-07-09 20:57:21 |
Scott Moser |
curtin (Ubuntu): status |
Confirmed |
Fix Released |
|
| 2015-07-09 20:57:33 |
Scott Moser |
tags |
verification-needed |
verification-done |
|
| 2015-07-15 18:47:45 |
Launchpad Janitor |
curtin (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2015-07-15 18:48:18 |
Chris J Arges |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2015-07-15 18:48:34 |
Launchpad Janitor |
curtin (Ubuntu Vivid): status |
Fix Committed |
Fix Released |
|
| 2015-12-09 18:37:12 |
Scott Moser |
curtin: importance |
Undecided |
Medium |
|
| 2017-12-15 22:09:20 |
Scott Moser |
curtin: status |
Fix Committed |
Fix Released |
|
