UTAH

UTAH failed to unmount ISO images

Bug #1123899 reported by Didier Roche-Tolomelli
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
UTAH
Fix Released
High
Javier Collado

Bug Description

As explained in #1, the problem was that there were not loop devices available
when trying to mount the image. This is a problem in UTAH's cleanup functions
that should take care of umounting everything when shutting down even if a
failure happened.

Example of jobs on 13/02/2012:

http://10.97.0.1:8080/job/ps-unity-autopilot-release-testing/label=autopilot-nvidia/83/console
http://10.97.0.1:8080/job/ps-unity-autopilot-release-testing/label=autopilot-ati/83/console
http://10.97.0.1:8080/job/ps-unity-autopilot-release-testing/label=autopilot-intel/83/console

http://10.97.0.1:8080/job/ps-indicators-autopilot-release-testing/label=autopilot-intel/122/console
http://10.97.0.1:8080/job/ps-indicators-autopilot-release-testing/label=autopilot-nvidia/122/console

We saw it as well twice on 12/02/2012.

The system reboots in busybox and so, everything is stalled.

See original description
Tags: ue-desktop

Related branches

lp:~nuclearbob/utah/loop-exception
Joe Talbott (community): Approve
Max Brustkern (community): Needs Resubmitting
Javier Collado (community): Approve
Diff: 47 lines (+12/-7)
1 file modified
utah/provisioning/baremetal/cobbler.py (+12/-7)
lp:~javier.collado/utah/install_sigterm_handler
Max Brustkern (community): Approve
Javier Collado (community): Needs Resubmitting
Diff: 70 lines (+25/-2)
3 files modified
examples/run_test_vm.py (+5/-1)
examples/run_utah_tests.py (+1/-1)
utah/run.py (+19/-0)
Didier Roche-Tolomelli (didrocks)
tags: added: ue-desktop
Revision history for this message
Javier Collado (javier.collado) wrote :

After some investigation it was found that the "Return code: 2" was because of
a failure while mounting images like this:
sudo mount -o loop /data/iso/ubuntu/daily-live/raring-desktop-i386.iso /tmp/dx-autopilot-nvidia_Lqu3iq/iso.d

The reason was that there were not enough available loop devices.

Looking at mount output, there were a few directories there that were left
mounted by utah:
/data/ISOs/raring/raring-desktop-i386.iso (deleted) on /tmp/ps-radeon-hd7850-he_NJzOa_/iso.d type iso9660 (ro)
/data/ISOs/raring/raring-desktop-i386.iso (deleted) on /tmp/ps-intel-2500-le_JDNopn/iso.d type iso9660 (ro)
/data/ISOs/raring/raring-desktop-i386.iso (deleted) on /tmp/ps-radeon-hd6870-he_45QNBz/iso.d type iso9660 (ro)
/data/ISOs/raring/raring-desktop-i386.iso (deleted) on /tmp/ps-intel-4000-he_tNutl6/iso.d type iso9660 (ro)
/data/ISOs/raring/raring-desktop-i386.iso (deleted) on /tmp/ps-radeon-hd7450-le_BV5KlQ/iso.d type iso9660 (ro)

After manually unmounting those directories, the loop devices were freed and
the job succeeded in mounting the images where it failed before.

summary: - UTAH is writing WARNING: Return code: 2 on the console, then stalling
- for a while until it timeouts (rebooted in busybox)
+ UTAH failed to unmount ISO images
description: updated
Revision history for this message
Javier Collado (javier.collado) wrote :

According to Max, the images should have been unmounted and the loop devices
freed in a normal run. However it might have happened that the jenkins job was
cancelled and the utah process was abruptly terminated.

Hence, some investigation is needed to make sure about how jenkins terminates
processes when a job is cancelled and how to handle that in utah.

Javier Collado (javier.collado)
Changed in utah:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Javier Collado (javier.collado)
status: In Progress → Fix Committed
Javier Collado (javier.collado)
Changed in utah:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.