Cloud-init 0.7.5 Failed to get cloudstack user-data because of trailing slash

[Loic Lambiel]

=== Begin SRU Information ===
[Impact]
CloudStack provides an EC2-style metadata server which cloud-init consumes. Unfortunately, the CloudStack metadata server does not exactly match the behaviour of the EC2 metadata server. When an instance using cloud-init is booted in a CloudStack environment, it will fail to retrieve metadata from the metadata server. This leads to cloud-init bailing out and making no changes to the instance, generally rendering it inaccessible and unusable.

This patch changes all access to EC2-style metadata servers. Instead of accessing http://.../meta-data to determine the available metadata, cloud-init will access http://.../meta-data/ (i.e. with an added trailing slash).

[Test Case]
Booting an Ubuntu image configured to use the CloudStack data source within a CloudStack cloud environment. If (for example) the instance name is set correctly, then the fix has worked.

[Regression Potential]

Regression potential is low; this patch does affect more than just the CloudStack data source, but it is backported from the version of cloud-init in utopic which has been observed working.

=== End SRU Information ===

Hi,

Cloud-init 0.7.5 fails to get cloudstack user-data because of trailing slash (ok in previous versions).

As a temp workaround I use the following patch in url_helper.py line 239 (0.7.5-0ubuntu1):

           if e.response.status_code == 404:
                if url.endswith("/"):
                    url = url[:-1]
                    req_args["url"] = url
                else:
                    url = url + "/"
                    req_args["url"] = url

I don't know how other products (EC2, openstack & co) deal with trailing slash

Hope that helps

Thanks

[Scott Moser (smoser)]

hi. can you give some more information on what is failing here?
perhaps a cloud-init.log of a fail?

[Loic Lambiel (loic-lambiel)]

Hi,

Here you are (should have been done before):

root@diskperf:/var/log# cat cloud-init.log
2014-08-22 09:40:03,606 - util.py[DEBUG]: Cloud-init v. 0.7.5 running 'init-local' at Fri, 22 Aug 2014 07:40:03 +0000. Up 3.45 seconds.
2014-08-22 09:40:03,628 - util.py[DEBUG]: Writing to /var/log/cloud-init.log - ab: [420] 0 bytes
2014-08-22 09:40:03,628 - util.py[DEBUG]: Changing the ownership of /var/log/cloud-init.log to 101:4
2014-08-22 09:40:03,629 - util.py[DEBUG]: Attempting to remove /var/lib/cloud/instance/boot-finished
2014-08-22 09:40:03,629 - util.py[DEBUG]: Attempting to remove /var/lib/cloud/instance
2014-08-22 09:40:03,629 - util.py[DEBUG]: Attempting to remove /var/lib/cloud/data/no-net
2014-08-22 09:40:03,629 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False)
2014-08-22 09:40:03,630 - importer.py[DEBUG]: Looking for modules ['ubuntu', 'cloudinit.distros.ubuntu'] that have attributes ['Distro']
2014-08-22 09:40:03,630 - importer.py[DEBUG]: Failed at attempted import of 'ubuntu' due to: No module named ubuntu
2014-08-22 09:40:03,645 - importer.py[DEBUG]: Found ubuntu with attributes ['Distro'] in ['cloudinit.distros.ubuntu']
2014-08-22 09:40:03,645 - stages.py[DEBUG]: Using distro class <class 'cloudinit.distros.ubuntu.Distro'>
2014-08-22 09:40:03,645 - __init__.py[DEBUG]: Looking for for data source in: ['CloudStack'], via packages ['', 'cloudinit.sources'] that matches dependencies ['FILESYSTEM']
2014-08-22 09:40:03,645 - importer.py[DEBUG]: Looking for modules ['DataSourceCloudStack', 'cloudinit.sources.DataSourceCloudStack'] that have attributes ['get_datasource_list']
2014-08-22 09:40:03,648 - importer.py[DEBUG]: Failed at attempted import of 'DataSourceCloudStack' due to: No module named DataSourceCloudStack
2014-08-22 09:40:03,653 - importer.py[DEBUG]: Found DataSourceCloudStack with attributes ['get_datasource_list'] in ['cloudinit.sources.DataSourceCloudStack']
2014-08-22 09:40:03,653 - __init__.py[DEBUG]: Searching for data source in: []
2014-08-22 09:40:03,656 - cloud-init[DEBUG]: No local datasource found
2014-08-22 09:40:03,660 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
2014-08-22 09:40:03,663 - util.py[DEBUG]: Read 10 bytes from /proc/uptime
2014-08-22 09:40:03,663 - util.py[DEBUG]: cloud-init mode 'init' took 0.311 seconds (0.31)
2014-08-22 09:40:04,098 - util.py[DEBUG]: Cloud-init v. 0.7.5 running 'init' at Fri, 22 Aug 2014 07:40:04 +0000. Up 4.15 seconds.
2014-08-22 09:40:04,105 - util.py[DEBUG]: Writing to /var/log/cloud-init.log - ab: [420] 0 bytes
2014-08-22 09:40:04,106 - util.py[DEBUG]: Changing the ownership of /var/log/cloud-init.log to 101:4
2014-08-22 09:40:04,106 - util.py[DEBUG]: Running command ['ifconfig', '-a'] with allowed return codes [0] (shell=False, capture=True)
2014-08-22 09:40:04,114 - util.py[DEBUG]: Running command ['netstat', '-rn'] with allowed return codes [0] (shell=False, capture=True)
2014-08-22 09:40:04,133 - cloud-init[DEBUG]: Checking to see if files that we need already exist from a previous run that would allow us to stop early.
2014-08-22 09:40:04,133 - util.py[DEBUG]: Reading from /var/lib/cloud/data/no-net (quiet=False)
2014-08-22 09:40:04...

[Loic Lambiel (loic-lambiel)]

cloud-init.log

[Scott Moser (smoser)]

I filed this upstream as https://issues.apache.org/jira/browse/CLOUDSTACK-7405
as this is actually a bug in cloudstack as its incompatible in some way with ec2 metadata service.

[Scott Moser (smoser)]

fix-committed to trunk in revno 999.

[Vincent Bernat (vbernat)]

Hi Scott! Would it be possible to push the fix in Trusty? This would help users of CloudStack using verbatim Ubuntu images. Thanks.

[Scott Moser (smoser)]

This is fix-released in ubuntu at 0.7.7~bzr1055-0ubuntu1

[Brian Murray (brian-murray)]

Hello Loic, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dan Watkins (oddbloke)]

I've tested this version on Interoute (which is a CloudStack cloud). I'll smoke test on another cloud, to ensure that we haven't regressed anything, and then mark this as verification-done.

[Dan Watkins (oddbloke)]

Metadata retrieval still works on trusty, marking as verification-done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.5-0ubuntu1.5

---------------
cloud-init (0.7.5-0ubuntu1.5) trusty; urgency=medium

  * Backport support for fetching passwords in CloudStack (LP: #1422388).
  * Fix CloudStack metadata retrieval (LP: #1356855).
 -- Daniel Watkins <email address hidden> Wed, 11 Mar 2015 10:57:10 -0600

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Kevin (kbockspam)]

Could this also please be ported to precise?

[Dan Watkins (oddbloke)]

Hi Kevin,

If you're seeing problems on precise, could you file a separate bug detailing your issues? The code path which this bug fixes doesn't exist in precise, so you must be seeing another issue. :)

Thanks,

Dan

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2476