| 2009-03-05 17:53:19 |
Chris Jones |
bug |
|
|
added bug |
| 2009-03-05 17:53:31 |
Chris Jones |
bug |
|
|
added subscriber The Canonical Sysadmins |
| 2009-03-12 13:41:03 |
Chris Jones |
title |
Node controller doesn't create nodes |
Node controller doesn't report failed image downloads |
|
| 2009-03-12 13:43:16 |
Chris Jones |
description |
This is the log from the node controller. It clearly sees that it is supposed to be spawning an instance, and at least starts grabbing the images, but almost immediately just rm -rf's them. What additional information would be useful to debug this?
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] doRunInstance() invoked (id=i-3ECC080E cores=1 disk=1 memory=128
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] image=emi-94A3140C at http://curium.cloud:8773/services/Walrus/hardy-i386-rootfs/hardy-i386-rootfs.manifest.xml
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] krnel=eki-C8501449 at http://curium.cloud:8773/services/Walrus/hardy-i386-kernel/vmlinuz-2.6.24-23-xen.manifest.xml
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] rmdsk=eri-09BA1527 at http://curium.cloud:8773/services/Walrus/hardy-i386-initrd/initrd.img-2.6.24-23-xen.manifest.xml
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] vlan=10 priMAC=d0:0d:3E:CC:08:0E pubMAC=d0:0d:3E:CC:08:0E
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] network started for instance i-3ECC080E
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] retrieving images for instance i-3ECC080E...
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] walrus_request(): downloading /opt/instances/admin/i-3ECC080E/root-digest
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] from http://curium.cloud:8773/services/Walrus/hardy-i386-rootfs/hardy-i386-rootfs.manifest.xml
[Thu Mar 5 14:49:10 2009][006011][EUCADEBUG ] walrus_request(): writing GET output to /opt/instances/admin/i-3ECC080E/root-digest
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] NC is looking for configuration in //etc/eucalyptus/eucalyptus.conf
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] NC is looking for configuration in //etc/eucalyptus/eucalyptus.conf
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] sem_alloc(): cleaning up old semaphore eucalyptus-nc-xen-semaphore
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] sem_alloc(): cleaning up old semaphore eucalyptus-nc-inst-semaphore
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] sem_alloc(): cleaning up old semaphore eucalyptus-storage-semaphore
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] SC is looking for configuration in //etc/eucalyptus/eucalyptus.conf
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] euca_init_cert(): using file //var/lib/eucalyptus/keys/node-cert.pem
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] euca_init_cert(): using file //var/lib/eucalyptus/keys/node-pk.pem
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] looking for existing Xen domains
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] checking the integrity of instances directory (/opt/instances)
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] vrun(): [rm -rf /opt/instances/admin/i-3ECC080E] |
This is the log from the node controller. It clearly sees that it is supposed to be spawning an instance, and at least starts grabbing the images, but almost immediately just rm -rf's them.
It transpires that this is due to the images being incomplete in Walrus (ran out of disk space while uploading them). This suggests that some internal verification is taking place and rejecting the images. The NC should really log this fact, and if possible send a message back to the CC so it knows the instance setup failed (atm the instance shows as pending until some timeout is reached, when it is shown as terminated).
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] doRunInstance() invoked (id=i-3ECC080E cores=1 disk=1 memory=128
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] image=emi-94A3140C at http://curium.cloud:8773/services/Walrus/hardy-i386-rootfs/hardy-i386-rootfs.manifest.xml
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] krnel=eki-C8501449 at http://curium.cloud:8773/services/Walrus/hardy-i386-kernel/vmlinuz-2.6.24-23-xen.manifest.xml
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] rmdsk=eri-09BA1527 at http://curium.cloud:8773/services/Walrus/hardy-i386-initrd/initrd.img-2.6.24-23-xen.manifest.xml
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] vlan=10 priMAC=d0:0d:3E:CC:08:0E pubMAC=d0:0d:3E:CC:08:0E
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] network started for instance i-3ECC080E
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] retrieving images for instance i-3ECC080E...
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] walrus_request(): downloading /opt/instances/admin/i-3ECC080E/root-digest
[Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] from http://curium.cloud:8773/services/Walrus/hardy-i386-rootfs/hardy-i386-rootfs.manifest.xml
[Thu Mar 5 14:49:10 2009][006011][EUCADEBUG ] walrus_request(): writing GET output to /opt/instances/admin/i-3ECC080E/root-digest
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] NC is looking for configuration in //etc/eucalyptus/eucalyptus.conf
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] NC is looking for configuration in //etc/eucalyptus/eucalyptus.conf
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] sem_alloc(): cleaning up old semaphore eucalyptus-nc-xen-semaphore
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] sem_alloc(): cleaning up old semaphore eucalyptus-nc-inst-semaphore
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] sem_alloc(): cleaning up old semaphore eucalyptus-storage-semaphore
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] SC is looking for configuration in //etc/eucalyptus/eucalyptus.conf
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] euca_init_cert(): using file //var/lib/eucalyptus/keys/node-cert.pem
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] euca_init_cert(): using file //var/lib/eucalyptus/keys/node-pk.pem
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] looking for existing Xen domains
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] checking the integrity of instances directory (/opt/instances)
[Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] vrun(): [rm -rf /opt/instances/admin/i-3ECC080E] |
|
| 2009-03-12 13:44:05 |
Chris Jones |
title |
Node controller doesn't report failed image downloads |
Node controller doesn't report invalid image downloads |
|
| 2009-03-16 09:07:24 |
Soren Hansen |
marked as duplicate |
|
341714 |
|
| 2009-03-18 09:03:44 |
Soren Hansen |
removed duplicate marker |
341714 |
|
|
| 2009-03-18 09:04:22 |
Soren Hansen |
title |
Node controller doesn't report invalid image downloads |
Node controller sometimes crashes |
|
| 2009-03-24 16:57:09 |
Chris Jones |
summary |
Node controller sometimes crashes |
Node controller apache2 dependency satisfied by an incompatible mpm |
|
| 2009-09-25 21:37:10 |
Colin Watson |
eucalyptus (Ubuntu): status |
New |
Triaged |
|
| 2009-09-25 21:37:13 |
Colin Watson |
eucalyptus (Ubuntu): importance |
Undecided |
Low |
|
| 2009-09-25 21:37:15 |
Colin Watson |
eucalyptus (Ubuntu): assignee |
|
Colin Watson (cjwatson) |
|
| 2009-09-25 21:37:44 |
Colin Watson |
eucalyptus (Ubuntu): status |
Triaged |
Fix Committed |
|
| 2009-09-25 21:37:50 |
Colin Watson |
eucalyptus (Ubuntu): importance |
Low |
Medium |
|
| 2009-09-25 21:39:16 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-core-dev/eucalyptus/ubuntu |
|
| 2009-09-25 21:50:32 |
Dustin Kirkland |
eucalyptus (Ubuntu): status |
Fix Committed |
Triaged |
|
| 2009-09-25 21:50:39 |
Dustin Kirkland |
eucalyptus (Ubuntu): assignee |
Colin Watson (cjwatson) |
Dustin Kirkland (kirkland) |
|
| 2009-09-25 21:52:57 |
Dustin Kirkland |
eucalyptus (Ubuntu): status |
Triaged |
Fix Committed |
|
| 2009-09-26 02:04:15 |
Launchpad Janitor |
eucalyptus (Ubuntu): status |
Fix Committed |
Fix Released |
|
| 2009-10-09 22:26:32 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/eucalyptus |
|
