VMs fail to get metadata from time to time

[Chintana Wilamuna]

I'm able to start VMs successfully but couldn't login as a result of VMs not being able to access the metadata service. Following can be seen in the console output.

Begin: Running /scripts/init-bottom ...
Done.
[ 3.698102] e1000: 0000:00:02.0: e1000_probe: (PCI:33MHz:32-bit) d0:0d:32:8b:06:2f
[ 3.699876] FDC 0 is a S82078B
[ 3.746299] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
consuming user data failed!
Traceback (most recent call last):
  File "/usr/bin/cloud-init", line 90, in <module>
    main()
  File "/usr/bin/cloud-init", line 47, in main
    cloud.consume_userdata,[],False)
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 215, in sem_and_run
    if self.sem_has_run(semname,freq): return
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 173, in sem_has_run
    semfile = self.sem_getpath(name,freq)
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 167, in sem_getpath
    freqtok = self.datasource.get_instance_id()
  File "/usr/lib/python2.6/dist-packages/cloudinit/DataSourceEc2.py", line 64, in get_instance_id
    return(self.metadata['instance-id'])
KeyError: 'instance-id'
init: cloud-init main process (560) terminated with status 1
mountall: Event failed
init: ureadahead-other main process (579) terminated with status 4
 * Starting AppArmor profiles
[80G Traceback (most recent call last):
  File "/usr/bin/cloud-init-cfg", line 46, in main
    cc.handle(name,run_args)
  File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 105, in handle
Traceback (most recent call last):
  File "/usr/bin/cloud-init-cfg", line 46, in main
    self.cloud.sem_and_run(name, freq, handler, [ name, args ])
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 215, in sem_and_run
    if self.sem_has_run(semname,freq): return
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 173, in sem_has_run
    semfile = self.sem_getpath(name,freq)
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 167, in sem_getpath
    freqtok = self.datasource.get_instance_id()
  File "/usr/lib/python2.6/dist-packages/cloudinit/DataSourceEc2.py", line 64, in get_instance_id
    return(self.metadata['instance-id'])
KeyError: 'instance-id'
config handling of config-mounts failed
    cc.handle(name,run_args)
  File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 105, in handle
    self.cloud.sem_and_run(name, freq, handler, [ name, args ])
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 215, in sem_and_run
    if self.sem_has_run(semname,freq): return
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 173, in sem_has_run
    semfile = self.sem_getpath(name,freq)
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 167, in sem_getpath
    freqtok = self.datasource.get_instance_id()
  File "/usr/lib/python2.6/dist-packages/cloudinit/DataSourceEc2.py", line 64, in get_instance_id
    return(self.metadata['instance-id'])
KeyError: 'instance-id'
config handling of config-ssh failed
init: cloud-config-ssh main process (591) terminated with status 1
init: cloud-config-mounts main process (598) terminated with status 1
Traceback (most recent call last):
  File "/usr/bin/cloud-init-cfg", line 46, in main
    cc.handle(name,run_args)
  File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 105, in handle
    self.cloud.sem_and_run(name, freq, handler, [ name, args ])
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 215, in sem_and_run
    if self.sem_has_run(semname,freq): return
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 173, in sem_has_run
    semfile = self.sem_getpath(name,freq)
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 167, in sem_getpath
    freqtok = self.datasource.get_instance_id()
  File "/usr/lib/python2.6/dist-packages/cloudinit/DataSourceEc2.py", line 64, in get_instance_id
    return(self.metadata['instance-id'])
KeyError: 'instance-id'
config handling of apt-update-upgrade failed
[74G[ OK ]
Traceback (most recent call last):
  File "/usr/bin/cloud-init-cfg", line 46, in main
    cc.handle(name,run_args)
  File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 105, in handle
    self.cloud.sem_and_run(name, freq, handler, [ name, args ])
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 215, in sem_and_run
    if self.sem_has_run(semname,freq): return
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 173, in sem_has_run
    semfile = self.sem_getpath(name,freq)
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 167, in sem_getpath
    freqtok = self.datasource.get_instance_id()
  File "/usr/lib/python2.6/dist-packages/cloudinit/DataSourceEc2.py", line 64, in get_instance_id
    return(self.metadata['instance-id'])
KeyError: 'instance-id'
config handling of config-misc failed
init: cloud-apt-update-upgrade main process (602) terminated with status 1
init: cloud-config-misc main process (603) terminated with status 1
init: apport pre-start process (670) terminated with status 1
Traceback (most recent call last):
  File "/usr/bin/cloud-init-cfg", line 46, in main
    cc.handle(name,run_args)
  File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 105, in handle
    self.cloud.sem_and_run(name, freq, handler, [ name, args ])
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 215, in sem_and_run
    if self.sem_has_run(semname,freq): return
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 173, in sem_has_run
    semfile = self.sem_getpath(name,freq)
  File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 167, in sem_getpath
    freqtok = self.datasource.get_instance_id()
  File "/usr/lib/python2.6/dist-packages/cloudinit/DataSourceEc2.py", line 64, in get_instance_id
    return(self.metadata['instance-id'])
KeyError: 'instance-id'
config handling of config-puppet failed
Traceback (most recent call last):
  File "/usr/bin/landscape-is-cloud-managed", line 12, in <module>
    sys.exit(not is_cloud_managed())
  File "/usr/lib/python2.6/dist-packages/landscape/broker/registration.py", line 397, in is_cloud_managed
    raw_user_data, int(launch_index))
ValueError: invalid literal for int() with base 10: ''

Any pointers as to how I can get around this? Is this related to https://bugs.launchpad.net/bugs/571271 ? The fix however not seems to be working for me.

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: eucalyptus-cloud 2.0+bzr1241-0ubuntu4
ProcVersionSignature: Ubuntu 2.6.35-22.33-server 2.6.35.4
Uname: Linux 2.6.35-22-server x86_64
.etc.eucalyptus.eucalyptus.cc.conf: CC_NAME="wso2"
Architecture: amd64
Date: Mon Feb 7 21:54:18 2011
InstallationMedia: Ubuntu-Server 10.10 "Maverick Meerkat" - Release amd64 (20101007)
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: eucalyptus

[Scott Moser (smoser)]

Are you able to reliably reproduce this? We've seen this bug before, but it is hard to reproduce. Any information you have on what scenarios it pops up in would be very useful.

[Chintana Wilamuna (chintana)]

I have seen this bug in two different Eucalyptus installations. I did not check whether it's reproducible because the set of servers that we run our private cloud is being used heavily and generally not used as a test bed for UEC. Installation was done using UEC CD install.