metadata server cannot be contacted when deploying new instances on EUC

[Eric Williams]

## Issue

New instances deployed on EUC cannot contact the metadata server, failing with Connection refused.

## Environment

Distributor ID: Ubuntu
Description: Ubuntu Natty (development branch)
Release: 11.04
Codename: natty

ii eucalyptus-admin-tools 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Admin tools
ii eucalyptus-cc 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Cluster controller
ii eucalyptus-cloud 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Cloud controller
ii eucalyptus-common 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Common files
ii eucalyptus-gl 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Logging service
ii eucalyptus-java-common 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Common Java package
ii eucalyptus-sc 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Storage controller
ii eucalyptus-walrus 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Walrus (S3)
ii libeucalyptus-commons-ext-java 0.5.0-0ubuntu2 Eucalyptus commons external Java library

## Resolution

- reinstalling usually fixes the issue, but this is suboptimal

## Diagnostic Steps

euca-get-console-output shows the following:

lxcmount stop/pre-start, process 47
init: udev-fallback-graphics main process (256) terminated with status 1
init: plymouth main process (49) killed by SEGV signal
init: plymouth-splash main process (259) terminated with status 2
cloud-init start-local running: Mon, 18 Apr 2011 22:05:50 +0000. up 7.14 seconds
no instance data found in start-local
init: cloud-init-local main process (255) terminated with status 1
cloud-init start running: Mon, 18 Apr 2011 22:05:50 +0000. up 7.70 seconds
2011-04-18 22:05:50,859 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id

2011-04-18 22:05:50,861 - DataSourceEc2.py[WARNING]: 22:05:50 [ 1/30]: url error [[Errno 111] Connection refused]

2011-04-18 22:05:51,864 - DataSourceEc2.py[WARNING]: 22:05:51 [ 2/30]: url error [[Errno 111] Connection refused]

2011-04-18 22:05:52,868 - DataSourceEc2.py[WARNING]: 22:05:52 [ 3/30]: url error [[Errno 111] Connection refused]

...

2011-04-18 19:55:09,908 - DataSourceEc2.py[WARNING]: 19:55:09 [26/30]: url error [[Errno 111] Connection refused]

2011-04-18 19:55:15,917 - DataSourceEc2.py[WARNING]: 19:55:15 [27/30]: url error [[Errno 111] Connection refused]

found data source: DataSourceEc2
mountall: Event failed

Opening http://169.254.169.254/2009-04-04/meta-data/instance-id from the cluster controller gives a 404 error, and not a "111 / Connection refused".

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: eucalyptus-cloud 2.0.1+bzr1256-0ubuntu4
ProcVersionSignature: Ubuntu 2.6.38-8.42-server 2.6.38.2
Uname: Linux 2.6.38-8-server x86_64
.etc.eucalyptus.eucalyptus.cc.conf: CC_NAME="disaster-cluster1"
Architecture: amd64
Date: Mon Apr 18 18:24:52 2011
ProcEnviron:
 LANGUAGE=en_US:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: eucalyptus
UpgradeStatus: No upgrade log present (probably fresh install)

[Boris Devouge (bdevouge)]

Reinstalling fixes the issue on Maverick, but not fully tested on 11.04, which could be a different issue.

[Torsten Spindler (tspindler)]

We tried the following to overcome the problem, nothing was successful:

1) Restart eucalyptus-cc
2) Reboot front-end and node-controlle
3) Purge and reinstall eucalyptus-cc

[Eric Williams (eric-canonical)]

Attaching a tcpdump of the following scenario:

- new instance is started
- it loops through the attempts to get the metadata via http
- the wget fails with connection refused

thanks,
eric