heat broken with cloud-init 0.7.1

[Steven Hardy]

Still investigating the root cause, but it would appear that heat is broken with F18 guests (and probably any other distro which provides cloud-init 0.7.1, e.g RHEL 6.4)

The problem is that we fail to retrieve the instance metadata, despite the metadata service being accessible on 169.254.169.254:

You see the following in /var/log/cloud-init.log:

    Mar 22 13:22:11 localhost [CLOUDINIT] url_helper.py[DEBUG]: Attempting to open 'http://169.254.169.254/2009-04-04/meta-data/instance-id' with 1 attempts (0 retries, timeout=50) to be performed
    Mar 22 13:22:12 localhost [CLOUDINIT] url_helper.py[DEBUG]: Read from http://169.254.169.254/2009-04-04/meta-data/instance-id (200, 10b) after 1 attempts
    Mar 22 13:22:12 localhost [CLOUDINIT] DataSourceEc2.py[DEBUG]: Using metadata source: 'http://169.254.169.254'
    Mar 22 13:22:12 localhost [CLOUDINIT] util.py[WARNING]: Failed reading from metadata address http://169.254.169.254

So we can see we read the instance-id from the metadata service OK, but fail to retrieve the instance metadata - debugging indicates an exception is being thrown in the boto ec2.get_instance_metadata() call (in DataSourceEc2.py)

The next step is for cloud-init to fall back to the DataSourceCloudStack source, which results in a 2 minute timeout trying to read data from the default gateway (which is visible on the instance primary console)

I'm using the same boto version (2.5.2) on both F17 (cloud-init 0.6.3-0.5.bzr532) and F18 (cloud-init 0.7.1-2), so it would appear this is a cloud-init issue but I'm not yet certain what is going wrong.

[Steve Baker (steve-stevebaker)]

Does cloud-init work with the packaged python-boto-2.6.0-2.fc18?

[Jeff Sloyer (jsloyer)]

I can consistently reproduce this issue. It looks to be related to Quantum, when an instance does not receive an IPv4 address from DHCP the above problem happens.

[Steven Hardy (shardy)]

The issue mentioned in #2 is a different problem - I'm not using quantum and the instance definitely does get an IP since I can SSH to it.

[Steven Hardy (shardy)]

> Does cloud-init work with the packaged python-boto-2.6.0-2.fc18?

Yes, it does seem to solve the specific problem retrieving the metadata in DataSourceEc2 mentioned above.

However there still seems to be a problem with the ec2-user ssh key not getting installed, and of course the fact that 2.6.0 won't work with the heat cfntools due to bug #1122472, gah - yet more boto support matrix pain!!

[Steven Hardy (shardy)]

Ok, finally (after much investigation), it turns out this is a dup of #1100920 - this is a cloud-init problem, and should be fixed when 0.7.2 gets released and into Fedora.

I can't find any viable workaround, so perhaps we need to look at getting the patch into a new revision of the F18 package (unless 0.7.2 is imminent and the F18 package will be rebased, I'll aske the cloud-init folks)

https://bugs.launchpad.net/cloud-init/+bug/1100920

fix merged into lp:cloud-init at revision 764

I've created a test RPM package for F18 from the latest trunk (bzr rev 809) and it definitely works and fixes the ssh key issues, provided we stick with our existing "old style" cloud config user: definition, see https://bugs.launchpad.net/cloud-init/+bug/1168431