partition2disk error starting up an instance

[Thierry Carrez]

Testing 20091009 / eucalyptus-1.6~bzr919-0ubuntu1

Starting up a 2G instance failed.

nc.log:
[EUCADEBUG ] walrus_request(): wrote 2147483648 bytes in 164952 writes
[EUCAINFO ] walrus_request(): saved image in /var/lib/eucalyptus/instances/admin/i-36FC0627/disk
[EUCAINFO ] vrun(): [//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/partition2disk /var/lib/eucalyptus/instances/admin/i-36FC0627/disk 0 0]
...
[EUCAERROR ] system(//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/partition2disk /var/lib/eucalyptus/instances/admin/i-36FC0627/disk 0 0) failed with 3328
[EUCAERROR ] error: partition-to-disk image conversion command failed
[EUCAERROR ] error: failed to download file from Walrus into /var/lib/eucalyptus/instances/admin/i-36FC0627/disk
[EUCAFATAL ] Failed to prepare images for instance i-36FC0627 (error=1)

httpd-nc_error_log:
/bin/dd if=/var/lib/eucalyptus/instances/admin/i-36FC0627/disk of=/dev/loop1 bs=512k
/bin/sync
/sbin/losetup -d /dev/loop0
loop: can't delete device /dev/loop1: Device or resource busy
ERROR while executing: /sbin/losetup -d /dev/loop1
loop: can't delete device /dev/loop1: Device or resource busy

Subsequent tries worked, so it might be another "you tried to start an instance that was not fully ready to be started" error.

[Thierry Carrez (ttx)]

I remember hitting that one on a previous test, so it's not really a regression, just a hard-to-reproduce issue.

[Dustin Kirkland  (kirkland)]

Marking incomplete. This needs to be re-tested against Lucid. Dan says they've made some changes upstream that serializes intensive IO operations that might help alleviate this issue.

Please mark confirmed if you have experienced this problem in Lucid.

[Dustin Kirkland  (kirkland)]

Marking fix-released, as we think this is fixed in Lucid. Please reopen if this can be reproduced in Lucid.

[Alex Wiechert (wiechert)]

This Problem also exists in Lucid.

Tue May 18 11:41:19 2010][009236][EUCAINFO ] walrus_request(): downloading /var/lib/eucalyptus/instances//admin/i-41EC07D4/disk-digest
[Tue May 18 11:41:19 2010][009236][EUCAINFO ] from http://192.168.150.105:8773/services/Walrus/image-store-1274110214/image.manifest.xml
[Tue May 18 11:41:19 2010][009236][EUCADEBUG ] walrus_request(): writing GET output to /var/lib/eucalyptus/instances//admin/i-41EC07D4/disk-digest
[Tue May 18 11:41:19 2010][009236][EUCADEBUG ] walrus_request(): wrote 5623 bytes in 0 writes
[Tue May 18 11:41:19 2010][009236][EUCAINFO ] walrus_request(): saved image in /var/lib/eucalyptus/instances//admin/i-41EC07D4/disk-digest
[Tue May 18 11:41:19 2010][009236][EUCAFATAL ] error: insufficient disk capacity remaining (2043MB) in VM Type of instance i-41EC07D4 for component disk
[Tue May 18 11:41:19 2010][009236][EUCAERROR ] get_cached_file() failed (errno=0)
[Tue May 18 11:41:19 2010][009236][EUCAFATAL ] Failed to prepare images for instance i-41EC07D4 (error=1)
[Tue May 18 11:41:19 2010][009236][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-41EC07D4' (code=42)
[Tue May 18 11:41:19 2010][009236][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances//admin/i-41EC07D4/]
[Tue May 18 11:41:19 2010][009236][EUCAINFO ] stopping the network (vlan=10)
[Tue May 18 11:41:30 2010][009236][EUCAINFO ] doTerminateInstance() invoked (id=i-41EC07D4)
[Tue May 18 11:41:30 2010][009236][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-41EC07D4' (code=42)
[Tue May 18 11:41:30 2010][009236][EUCAWARN ] warning: domain i-41EC07D4 to be terminated not running on hypervisor
[Tue May 18 11:41:34 2010][009236][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-41EC07D4' (code=42)
[Tue May 18 11:41:34 2010][009236][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances//admin/i-41EC07D4/]
[Tue May 18 11:41:34 2010][009236][EUCAINFO ] stopping the network (vlan=10)

[Alex Wiechert (wiechert)]

[Tue May 18 11:41:19 2010][009236][EUCAFATAL ] error: insufficient disk capacity remaining (2043MB) in VM Type of instance i-41EC07D4 for component disk

After increse size of disk in Front-End from 2 to 3 GB this failure isn't present anymore. But now instance hangs "pending".

[Neil Soman (neilsoman)]

Hi Alex, there is not enough information in your report to figure out what is wrong with your installation. At the very minimum, we need nc.log.

Please see open.eucalyptus.com/forum for help with this problem. The bug tracker is only for filing bugs.

hope that helps
neil