Activity log for bug #338344

Date Who What changed Old value New value Message
2009-03-05 17:53:19 Chris Jones bug added bug
2009-03-05 17:53:31 Chris Jones bug added subscriber The Canonical Sysadmins
2009-03-12 13:41:03 Chris Jones title Node controller doesn't create nodes Node controller doesn't report failed image downloads
2009-03-12 13:43:16 Chris Jones description This is the log from the node controller. It clearly sees that it is supposed to be spawning an instance, and at least starts grabbing the images, but almost immediately just rm -rf's them. What additional information would be useful to debug this? [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] doRunInstance() invoked (id=i-3ECC080E cores=1 disk=1 memory=128 [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] image=emi-94A3140C at http://curium.cloud:8773/services/Walrus/hardy-i386-rootfs/hardy-i386-rootfs.manifest.xml [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] krnel=eki-C8501449 at http://curium.cloud:8773/services/Walrus/hardy-i386-kernel/vmlinuz-2.6.24-23-xen.manifest.xml [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] rmdsk=eri-09BA1527 at http://curium.cloud:8773/services/Walrus/hardy-i386-initrd/initrd.img-2.6.24-23-xen.manifest.xml [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] vlan=10 priMAC=d0:0d:3E:CC:08:0E pubMAC=d0:0d:3E:CC:08:0E [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] network started for instance i-3ECC080E [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] retrieving images for instance i-3ECC080E... [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] walrus_request(): downloading /opt/instances/admin/i-3ECC080E/root-digest [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] from http://curium.cloud:8773/services/Walrus/hardy-i386-rootfs/hardy-i386-rootfs.manifest.xml [Thu Mar 5 14:49:10 2009][006011][EUCADEBUG ] walrus_request(): writing GET output to /opt/instances/admin/i-3ECC080E/root-digest [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] NC is looking for configuration in //etc/eucalyptus/eucalyptus.conf [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] NC is looking for configuration in //etc/eucalyptus/eucalyptus.conf [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] sem_alloc(): cleaning up old semaphore eucalyptus-nc-xen-semaphore [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] sem_alloc(): cleaning up old semaphore eucalyptus-nc-inst-semaphore [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] sem_alloc(): cleaning up old semaphore eucalyptus-storage-semaphore [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] SC is looking for configuration in //etc/eucalyptus/eucalyptus.conf [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] euca_init_cert(): using file //var/lib/eucalyptus/keys/node-cert.pem [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] euca_init_cert(): using file //var/lib/eucalyptus/keys/node-pk.pem [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] looking for existing Xen domains [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] checking the integrity of instances directory (/opt/instances) [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] vrun(): [rm -rf /opt/instances/admin/i-3ECC080E] This is the log from the node controller. It clearly sees that it is supposed to be spawning an instance, and at least starts grabbing the images, but almost immediately just rm -rf's them. It transpires that this is due to the images being incomplete in Walrus (ran out of disk space while uploading them). This suggests that some internal verification is taking place and rejecting the images. The NC should really log this fact, and if possible send a message back to the CC so it knows the instance setup failed (atm the instance shows as pending until some timeout is reached, when it is shown as terminated). [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] doRunInstance() invoked (id=i-3ECC080E cores=1 disk=1 memory=128 [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] image=emi-94A3140C at http://curium.cloud:8773/services/Walrus/hardy-i386-rootfs/hardy-i386-rootfs.manifest.xml [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] krnel=eki-C8501449 at http://curium.cloud:8773/services/Walrus/hardy-i386-kernel/vmlinuz-2.6.24-23-xen.manifest.xml [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] rmdsk=eri-09BA1527 at http://curium.cloud:8773/services/Walrus/hardy-i386-initrd/initrd.img-2.6.24-23-xen.manifest.xml [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] vlan=10 priMAC=d0:0d:3E:CC:08:0E pubMAC=d0:0d:3E:CC:08:0E [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] network started for instance i-3ECC080E [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] retrieving images for instance i-3ECC080E... [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] walrus_request(): downloading /opt/instances/admin/i-3ECC080E/root-digest [Thu Mar 5 14:49:10 2009][006011][EUCAINFO ] from http://curium.cloud:8773/services/Walrus/hardy-i386-rootfs/hardy-i386-rootfs.manifest.xml [Thu Mar 5 14:49:10 2009][006011][EUCADEBUG ] walrus_request(): writing GET output to /opt/instances/admin/i-3ECC080E/root-digest [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] NC is looking for configuration in //etc/eucalyptus/eucalyptus.conf [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] NC is looking for configuration in //etc/eucalyptus/eucalyptus.conf [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] sem_alloc(): cleaning up old semaphore eucalyptus-nc-xen-semaphore [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] sem_alloc(): cleaning up old semaphore eucalyptus-nc-inst-semaphore [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] sem_alloc(): cleaning up old semaphore eucalyptus-storage-semaphore [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] SC is looking for configuration in //etc/eucalyptus/eucalyptus.conf [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] euca_init_cert(): using file //var/lib/eucalyptus/keys/node-cert.pem [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] euca_init_cert(): using file //var/lib/eucalyptus/keys/node-pk.pem [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] looking for existing Xen domains [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] checking the integrity of instances directory (/opt/instances) [Thu Mar 5 14:49:10 2009][006037][EUCAINFO ] vrun(): [rm -rf /opt/instances/admin/i-3ECC080E]
2009-03-12 13:44:05 Chris Jones title Node controller doesn't report failed image downloads Node controller doesn't report invalid image downloads
2009-03-16 09:07:24 Soren Hansen marked as duplicate 341714
2009-03-18 09:03:44 Soren Hansen removed duplicate marker 341714
2009-03-18 09:04:22 Soren Hansen title Node controller doesn't report invalid image downloads Node controller sometimes crashes
2009-03-24 16:57:09 Chris Jones summary Node controller sometimes crashes Node controller apache2 dependency satisfied by an incompatible mpm
2009-09-25 21:37:10 Colin Watson eucalyptus (Ubuntu): status New Triaged
2009-09-25 21:37:13 Colin Watson eucalyptus (Ubuntu): importance Undecided Low
2009-09-25 21:37:15 Colin Watson eucalyptus (Ubuntu): assignee Colin Watson (cjwatson)
2009-09-25 21:37:44 Colin Watson eucalyptus (Ubuntu): status Triaged Fix Committed
2009-09-25 21:37:50 Colin Watson eucalyptus (Ubuntu): importance Low Medium
2009-09-25 21:39:16 Launchpad Janitor branch linked lp:~ubuntu-core-dev/eucalyptus/ubuntu
2009-09-25 21:50:32 Dustin Kirkland  eucalyptus (Ubuntu): status Fix Committed Triaged
2009-09-25 21:50:39 Dustin Kirkland  eucalyptus (Ubuntu): assignee Colin Watson (cjwatson) Dustin Kirkland (kirkland)
2009-09-25 21:52:57 Dustin Kirkland  eucalyptus (Ubuntu): status Triaged Fix Committed
2009-09-26 02:04:15 Launchpad Janitor eucalyptus (Ubuntu): status Fix Committed Fix Released
2009-10-09 22:26:32 Launchpad Janitor branch linked lp:ubuntu/eucalyptus