| 2011-03-28 16:18:47 |
Vish Ishaya |
description |
Instance fails to access metadata server at launch resulting it fails to complete initial booting process including sshd startup.
Logs from tty linux.
=========================================
Lease of 10.0.0.5 obtained, lease time 120^M
starting DHCP forEthernet interface eth0 [ ^[[1;32mOK^[[0;39m ]^M
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id^M
cloud-setup: failed 1/30: up 7.97. iid had 1.0^M
cloud-setup: failed 2/30: up 9.18. iid had 1.0^M
cloud-setup: failed 3/30: up 10.35. iid had 1.0^M
cloud-setup: failed 4/30: up 11.52. iid had 1.0^M
cloud-setup: failed 5/30: up 12.70. iid had 1.0^M
cloud-setup: failed 6/30: up 13.88. iid had 1.0^M
cloud-setup: failed 7/30: up 15.06. iid had 1.0^M
cloud-setup: failed 8/30: up 16.24. iid had 1.0^M
cloud-setup: failed 9/30: up 17.43. iid had 1.0^M
cloud-setup: failed 10/30: up 18.62. iid had 1.0^M
cloud-setup: failed 11/30: up 19.81. iid had 1.0^M
cloud-setup: failed 12/30: up 21.00. iid had 1.0^M
cloud-setup: failed 13/30: up 22.20. iid had 1.0^M
cloud-setup: failed 14/30: up 23.40. iid had 1.0^M
cloud-setup: failed 15/30: up 24.60. iid had 1.0^M
cloud-setup: failed 16/30: up 25.80. iid had 1.0^M
cloud-setup: failed 17/30: up 27.01. iid had 1.0^M
cloud-setup: failed 18/30: up 28.22. iid had 1.0^M
cloud-setup: failed 19/30: up 29.43. iid had 1.0^M
cloud-setup: failed 20/30: up 30.65. iid had 1.0^M
cloud-setup: failed 21/30: up 31.86. iid had 1.0^M
cloud-setup: failed 22/30: up 33.08. iid had 1.0^M
cloud-setup: failed 23/30: up 34.30. iid had 1.0^M
cloud-setup: failed 24/30: up 35.60. iid had 1.0^M
cloud-setup: failed 25/30: up 36.89. iid had 1.0^M
cloud-setup: failed 26/30: up 38.11. iid had 1.0^M
cloud-setup: failed 27/30: up 39.34. iid had 1.0^M
cloud-setup: failed 28/30: up 40.56. iid had 1.0^M
cloud-setup: failed 29/30: up 41.82. iid had 1.0^M
cloud-setup: failed 30/30: up 43.05. iid had 1.0^M
cloud-setup: after 30 fails, debugging^M
cloud-setup: running debug (30 tries reached)^M
############ debug start ##############^M
### /etc/rc.d/init.d/sshd start^M
stty: /dev/console^M
generating DSS host key [^[[1;33mWATING^[[0;39m]^[[-11G^[[1;34m..^[[0;39m [ ^[[1;32mOK^[[0;39m ]^M
generating RSA host key [^[[1;33mWATING^[[0;39m]^[[-11G^[[1;34m..^[[0;39m [ ^[[1;32mOK^[[0;39m ]^M
startup dropbear [ ^[[1;32mOK^[[0;39m ]^M
### ifconfig -a^M
eth0 Link encap:Ethernet HWaddr 02:16:3E:57:D3:B5 ^M
=========================================
Logs from UEC image.
=========================================
init: plymouth-splash main process (263) terminated with status 2
init: plymouth main process (48) killed by SEGV signal
cloud-init running: Tue, 15 Feb 2011 09:55:54 +0000. up 30.11 seconds
consuming user data failed!
Traceback (most recent call last):
File "/usr/bin/cloud-init", line 103, in <module>
main()
File "/usr/bin/cloud-init", line 60, in main
cloud.consume_userdata,[],False)
File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 215, in sem_and_run
if self.sem_has_run(semname,freq): return
File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 173, in sem_has_run
semfile = self.sem_getpath(name,freq)
File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 167, in sem_getpath
freqtok = self.datasource.get_instance_id()
File "/usr/lib/python2.6/dist-packages/cloudinit/DataSourceEc2.py", line 65, in get_instance_id
return(self.metadata['instance-id'])
KeyError: 'instance-id'
init: cloud-init main process (334) terminated with status 1
mountall: Event failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Disconnected from Plymouth
init: plymouth-log main process (364) terminated with status 1
* Starting AppArmor profiles [ OK ]
Traceback (most recent call last):
File "/usr/bin/cloud-init-cfg", line 56, in <module>
Traceback (most recent call last):
File "/usr/bin/cloud-init-cfg", line 56, in <module>
main()
main()
File "/usr/bin/cloud-init-cfg", line 43, in main
cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 42, in __init__
File "/usr/bin/cloud-init-cfg", line 43, in main
cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 42, in __init__
self.cfg = self.get_config_obj(cfgfile)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 53, in get_config_obj
f=file(cfgfile)
Traceback (most recent call last):
File "/usr/bin/cloud-init-cfg", line 56, in <module>
self.cfg = self.get_config_obj(cfgfile)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 53, in get_config_obj
f=file(cfgfile)
IOError: IOError: [Errno 2] No such file or directory: '/var/lib/cloud/data/cloud-config.txt'
main()
File "/usr/bin/cloud-init-cfg", line 43, in main
cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 42, in __init__
self.cfg = self.get_config_obj(cfgfile)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 53, in get_config_obj
[Errno 2] No such file or directory: '/var/lib/cloud/data/cloud-config.txt'
f=file(cfgfile)
IOError: [Errno 2] No such file or directory: '/var/lib/cloud/data/cloud-config.txt'
Traceback (most recent call last):
File "/usr/bin/cloud-init-cfg", line 56, in <module>
main()
File "/usr/bin/cloud-init-cfg", line 43, in main
cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 42, in __init__
self.cfg = self.get_config_obj(cfgfile)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 53, in get_config_obj
f=file(cfgfile)
IOError: [Errno 2] No such file or directory: '/var/lib/cloud/data/cloud-config.txt'
Traceback (most recent call last):
File "/usr/bin/cloud-init-cfg", line 56, in <module>
main()
File "/usr/bin/cloud-init-cfg", line 43, in main
cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 42, in __init__
self.cfg = self.get_config_obj(cfgfile)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 53, in get_config_obj
f=file(cfgfile)
IOError: [Errno 2] No such file or directory: '/var/lib/cloud/data/cloud-config.txt'
landscape-client is not configured, please run landscape-config.
Traceback (most recent call last):
File "/usr/bin/cloud-init-cfg", line 56, in <module>
main()
File "/usr/bin/cloud-init-cfg", line 43, in main
cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 42, in __init__
self.cfg = self.get_config_obj(cfgfile)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 53, in get_config_obj
f=file(cfgfile)
IOError: [Errno 2] No such file or directory: '/var/lib/cloud/data/cloud-config.txt' |
Instances that have a local route (due to installing Network Manager in ubuntu for example) cannot contact the metadata server. This is because they send out an ARP (who-has) for 169.254.169.254 and never get a response. This issue also affects windows vms. The issue can be solved by giving the ip address to the host that is running nova-network, something along the lines of:
ip addr add 169.254.169.254/32 scope link dev eth1
This causes the network host to respond to the arp and the metadata request succeeds. Nova should automatically add this address to avoid this failure.
Example Error messages are below:
Instance fails to access metadata server at launch resulting it fails to complete initial booting process including sshd startup.
Logs from tty linux.
=========================================
Lease of 10.0.0.5 obtained, lease time 120^M
starting DHCP forEthernet interface eth0 [ ^[[1;32mOK^[[0;39m ]^M
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id^M
cloud-setup: failed 1/30: up 7.97. iid had 1.0^M
cloud-setup: failed 2/30: up 9.18. iid had 1.0^M
cloud-setup: failed 3/30: up 10.35. iid had 1.0^M
cloud-setup: failed 4/30: up 11.52. iid had 1.0^M
cloud-setup: failed 5/30: up 12.70. iid had 1.0^M
cloud-setup: failed 6/30: up 13.88. iid had 1.0^M
cloud-setup: failed 7/30: up 15.06. iid had 1.0^M
cloud-setup: failed 8/30: up 16.24. iid had 1.0^M
cloud-setup: failed 9/30: up 17.43. iid had 1.0^M
cloud-setup: failed 10/30: up 18.62. iid had 1.0^M
cloud-setup: failed 11/30: up 19.81. iid had 1.0^M
cloud-setup: failed 12/30: up 21.00. iid had 1.0^M
cloud-setup: failed 13/30: up 22.20. iid had 1.0^M
cloud-setup: failed 14/30: up 23.40. iid had 1.0^M
cloud-setup: failed 15/30: up 24.60. iid had 1.0^M
cloud-setup: failed 16/30: up 25.80. iid had 1.0^M
cloud-setup: failed 17/30: up 27.01. iid had 1.0^M
cloud-setup: failed 18/30: up 28.22. iid had 1.0^M
cloud-setup: failed 19/30: up 29.43. iid had 1.0^M
cloud-setup: failed 20/30: up 30.65. iid had 1.0^M
cloud-setup: failed 21/30: up 31.86. iid had 1.0^M
cloud-setup: failed 22/30: up 33.08. iid had 1.0^M
cloud-setup: failed 23/30: up 34.30. iid had 1.0^M
cloud-setup: failed 24/30: up 35.60. iid had 1.0^M
cloud-setup: failed 25/30: up 36.89. iid had 1.0^M
cloud-setup: failed 26/30: up 38.11. iid had 1.0^M
cloud-setup: failed 27/30: up 39.34. iid had 1.0^M
cloud-setup: failed 28/30: up 40.56. iid had 1.0^M
cloud-setup: failed 29/30: up 41.82. iid had 1.0^M
cloud-setup: failed 30/30: up 43.05. iid had 1.0^M
cloud-setup: after 30 fails, debugging^M
cloud-setup: running debug (30 tries reached)^M
############ debug start ##############^M
### /etc/rc.d/init.d/sshd start^M
stty: /dev/console^M
generating DSS host key [^[[1;33mWATING^[[0;39m]^[[-11G^[[1;34m..^[[0;39m [ ^[[1;32mOK^[[0;39m ]^M
generating RSA host key [^[[1;33mWATING^[[0;39m]^[[-11G^[[1;34m..^[[0;39m [ ^[[1;32mOK^[[0;39m ]^M
startup dropbear [ ^[[1;32mOK^[[0;39m ]^M
### ifconfig -a^M
eth0 Link encap:Ethernet HWaddr 02:16:3E:57:D3:B5 ^M
=========================================
Logs from UEC image.
=========================================
init: plymouth-splash main process (263) terminated with status 2
init: plymouth main process (48) killed by SEGV signal
cloud-init running: Tue, 15 Feb 2011 09:55:54 +0000. up 30.11 seconds
consuming user data failed!
Traceback (most recent call last):
File "/usr/bin/cloud-init", line 103, in <module>
main()
File "/usr/bin/cloud-init", line 60, in main
cloud.consume_userdata,[],False)
File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 215, in sem_and_run
if self.sem_has_run(semname,freq): return
File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 173, in sem_has_run
semfile = self.sem_getpath(name,freq)
File "/usr/lib/python2.6/dist-packages/cloudinit/__init__.py", line 167, in sem_getpath
freqtok = self.datasource.get_instance_id()
File "/usr/lib/python2.6/dist-packages/cloudinit/DataSourceEc2.py", line 65, in get_instance_id
return(self.metadata['instance-id'])
KeyError: 'instance-id'
init: cloud-init main process (334) terminated with status 1
mountall: Event failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Plymouth command failed
mountall: Disconnected from Plymouth
init: plymouth-log main process (364) terminated with status 1
* Starting AppArmor profiles [ OK ]
Traceback (most recent call last):
File "/usr/bin/cloud-init-cfg", line 56, in <module>
Traceback (most recent call last):
File "/usr/bin/cloud-init-cfg", line 56, in <module>
main()
main()
File "/usr/bin/cloud-init-cfg", line 43, in main
cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 42, in __init__
File "/usr/bin/cloud-init-cfg", line 43, in main
cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 42, in __init__
self.cfg = self.get_config_obj(cfgfile)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 53, in get_config_obj
f=file(cfgfile)
Traceback (most recent call last):
File "/usr/bin/cloud-init-cfg", line 56, in <module>
self.cfg = self.get_config_obj(cfgfile)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 53, in get_config_obj
f=file(cfgfile)
IOError: IOError: [Errno 2] No such file or directory: '/var/lib/cloud/data/cloud-config.txt'
main()
File "/usr/bin/cloud-init-cfg", line 43, in main
cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 42, in __init__
self.cfg = self.get_config_obj(cfgfile)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 53, in get_config_obj
[Errno 2] No such file or directory: '/var/lib/cloud/data/cloud-config.txt'
f=file(cfgfile)
IOError: [Errno 2] No such file or directory: '/var/lib/cloud/data/cloud-config.txt'
Traceback (most recent call last):
File "/usr/bin/cloud-init-cfg", line 56, in <module>
main()
File "/usr/bin/cloud-init-cfg", line 43, in main
cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 42, in __init__
self.cfg = self.get_config_obj(cfgfile)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 53, in get_config_obj
f=file(cfgfile)
IOError: [Errno 2] No such file or directory: '/var/lib/cloud/data/cloud-config.txt'
Traceback (most recent call last):
File "/usr/bin/cloud-init-cfg", line 56, in <module>
main()
File "/usr/bin/cloud-init-cfg", line 43, in main
cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 42, in __init__
self.cfg = self.get_config_obj(cfgfile)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 53, in get_config_obj
f=file(cfgfile)
IOError: [Errno 2] No such file or directory: '/var/lib/cloud/data/cloud-config.txt'
landscape-client is not configured, please run landscape-config.
Traceback (most recent call last):
File "/usr/bin/cloud-init-cfg", line 56, in <module>
main()
File "/usr/bin/cloud-init-cfg", line 43, in main
cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 42, in __init__
self.cfg = self.get_config_obj(cfgfile)
File "/usr/lib/python2.6/dist-packages/cloudinit/CloudConfig.py", line 53, in get_config_obj
f=file(cfgfile)
IOError: [Errno 2] No such file or directory: '/var/lib/cloud/data/cloud-config.txt' |
|
