CentOS 6 cloud image with cloud-init-0.7.5 can not retrieve OpenStack metadata

[Denis Egorenko]

{"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}

Steps to reproduce:
1) HA;
2) CentOS;
3) Neutron GRE;
4) Ceph (cinder, glance, radosgw);

3 Controllers + 1 Compute

During boot CentOS image with cloud-init 0.7.5 has some errors:
2014-12-29 09:45:43,127 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceOpenStack.DataSourceOpenStack'> failed
2014-12-29 09:45:48,344 - util.py[WARNING]: Failed fetching userdata from url http://169.254.169.254/2009-04-04/user-data
2014-12-29 09:45:53,357 - util.py[WARNING]: Failed fetching metadata from url http://169.254.169.254/2009-04-04/meta-data
Starting cloud-init: Cloud-init v. 0.7.5 running 'modules:config' at Mon, 29 Dec 2014 14:45:54 +0000. Up 73.92 seconds.
Starting cloud-init: Cloud-init v. 0.7.5 running 'modules:final' at Mon, 29 Dec 2014 14:45:55 +0000. Up 74.72 seconds.
2014-12-29 09:45:55,393 - util.py[WARNING]: Running ssh-authkey-fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python2.6/site-packages/cloudinit/config/cc_ssh_authkey_fingerprints.pyc'>) failed

Also expecting datasource DataSourceOpenstack or DataSourceEc2 instead:
Cloud-init v. 0.7.5 finished at Mon, 29 Dec 2014 14:45:55 +0000. Datasource DataSourceNone. Up 75.09 seconds

As a result cloud-init doesn't work. This bug is reproduced on 2 from 3 VMs.

[Oleksandr Liemieshko (oliemieshko)]

I came across this bug in
Fuel 6.0: HA (3 controller+1 compute + 1 cinder), Ubuntu, Neutron GRE, Cinder(default), Glance(default)
But this error is not persistent. After reboot everything works fine some time

[Dave Johnston (dave-johnston)]

I have observed this on Fuel 6.0 HA setup 3 Controller + 5 Compute.
CentOS + Neutron GRE

Any potential work around ?

[Dave Johnston (dave-johnston)]

A;exander,

When you say reboot - reboot of what? In my setup, rebooting doesn't seem to change anything.

Thanks

[Oleksandr Liemieshko (oliemieshko)]

Dave Johnston,
When I rebooted all cluster(controller + compute), some time everything works fine but it's not work around

[Miroslav Anashkin (manashkin)]

Added MOS Neutron team.
Looks like this bug is in Neutron.

[Miroslav Anashkin (manashkin)]

Attached log with error

[Bogdan Dobrelya (bogdando)]

I'm setting this issue as invalid for Fuel as it related to Neutron

[Sergey Kolekonov (skolekonov)]

The problem is connected with cloud-init 0.7.5 package from CentOS. It contains patches and differs from the vanilla cloud-init and Ubuntu version, for example. I've found patches inside the source package which cause such problems with metadata. To be more specific, they are "cloud-init-centos-opennebula.patch", "cloud-init-centos-opennebula-requiretty.patch" and "cloud-init-centos-cloudstack-urlhandling.patch" After I rebuilt the package without them and made a new CentOS image, cloud-init works perfectly. So I think it should be fixed in CentOS packages.

[Alexander Bozhenko (alexbozhenko)]

Hello, Sergey.

What version of cloud-init is usually used in centos? From EPEL?
Here http://dl.fedoraproject.org/pub/epel/6/x86_64/repoview/cloud-init.html
I can see only this version: cloud-init-0.7.4-2.el6.noarch

But in epel7 repo
http://dl.fedoraproject.org/pub/epel/7/x86_64/repoview/cloud-init.html
cloud-init-0.7.5-6.el7.x86_64 is available

On fuel 6.0 changelog
rpm -qp --changelog /var/www/nailgun/2014.2-6.0/centos/x86_64/Packages/cloud-init-0.7.4-2.mira1.noarch.rpm
and changelog of that package downloaded from epel are the same
rpm -qp --changelog cloud-init-0.7.4-2.el6.noarch

Even though size of packages are different:
[root@fuel mira]# ll cloud-init-0.7.4-2.mira1.noarch.rpm
-rwxr-xr-x 1 root root 498732 Jan 4 02:02 cloud-init-0.7.4-2.mira1.noarch.rpm
[root@fuel epel]# ll cloud-init-0.7.4-2.el6.noarch.rpm
-rw-r--r-- 1 root root 498952 Feb 9 02:37 cloud-init-0.7.4-2.el6.noarch.rpm

I extracted packages with
rpm2cpio cloud-init-0.7.4-2.mira1.noarch.rpm | cpio -idmv
rpm2cpio cloud-init-0.7.4-2.el6.noarch.rpm | cpio -idmv

And found that only difference is cc_keys_to_console.pyc and cc_keys_to_console.pyo files. Even though source code cc_keys_to_console.py is the same. All other files have same md5sum too.

[root@fuel config_mira]# md5sum cc_keys_to_console.py
1eb40f754817fa1e468d4af72b368d55 cc_keys_to_console.py

[root@fuel config_epel]# md5sum cc_keys_to_console.py
1eb40f754817fa1e468d4af72b368d55 cc_keys_to_console.py

[root@fuel config_mira]# md5sum cc_keys_to_console.pyc
2f3d3e8f4911d588e8a95484aa12f671 cc_keys_to_console.pyc

[root@fuel config_epel]# md5sum cc_keys_to_console.pyc
77a5a33a70d762d44a9c934056fde1ca cc_keys_to_console.pyc

[root@fuel config_mira]# md5sum cc_keys_to_console.pyo
2f3d3e8f4911d588e8a95484aa12f671 cc_keys_to_console.pyo

[root@fuel config_epel]# md5sum cc_keys_to_console.pyo
77a5a33a70d762d44a9c934056fde1ca cc_keys_to_console.pyo

So, should we file a bug in centos|rhel bugtracker?
What our customers should do, before it is fixed in upstream? Probably, you can share you rebuilded version of package?

UPD:
I found cloud-init-0.7.5-10.el6.centos.2.src.rpm in centos extras repo for centos 6.6
http://vault.centos.org/6.6/extras/Source/SPackages/
And spec there contains packages, that you mentioned
Patch0001: 0001-update-paths-for-rhel.patch
Patch0002: 0002-fix-rsyslog-programname-match.patch
Patch0003: 0003-Stopped-calling-udevadm-settle-with-quiet-since-syst.patch
Patch0004: 0004-fix-typo-in-settings.py-preventing-metadata-being-fe.patch

Patch1001: cloud-init-centos-opennebula.patch
Patch1002: cloud-init-centos-hostnamefix.patch
Patch1003: cloud-init-centos-opennebula-requiretty.patch
Patch1004: cloud-init-centos-cloudstack-urlhandling.patch

But source rpmcloud-init-0.7.4-2.el6.src.rpm from here https://dl.fedoraproject.org/pub/epel/6/SRPMS/
doesn't contain patches, that you mentioned.
Version: 0.7.4
Release: 2%{?dist}
Summary: Cloud instance init scripts

Group: System Environment/Base
License: GPLv3
URL: http://launchpad.net/cloud-init
Source0: https://launchpad.net/cloud-init/trunk/%{version}/+download/cloud-init-%{...

[Sergey Kolekonov (skolekonov)]

Hello Alexander,

I think it makes sense to file a bug in centos bugtracker.
Speaking about customers, it's worth a try to downgrade cloud-init in cloud images to 0.7.4 version using the package from epel, for example.

[Dmitry Borodaenko (angdraug)]

Reopened: please provide a text that can be included in our documentation (release notes and ops guide) to warn customers about this issue and provide a workaround.

[Alexander Bozhenko (alexbozhenko)]

Sergey, could you please help me to file that bug in centos bugtracker correctly? You are more on the topic.
What exactly these packages "cloud-init-centos-opennebula.patch", "cloud-init-centos-opennebula-requiretty.patch" and "cloud-init-centos-cloudstack-urlhandling.patch" breaked?

[Alexander Bozhenko (alexbozhenko)]

Latest centos cloud image CentOS-6-x86_64-GenericCloud.qcow2, available here: http://cloud.centos.org/centos/6/images/
Contains this version of cloud-init: cloud-init-0.7.5-10.el6.centos.2.x86_64

Also I tried previous version CentOS-6-x86_64-GenericCloud-20140929_01.qcow2, available here: http://cloud.centos.org/centos/6.5/images/
but that version doesn't contain cloud-init at all.

I also checked rhel-guest-image-6.5-20140930.0.x86_64.qcow2, available here if you have rhel subscription https://rhn.redhat.com/rhn/software/channel/downloads/Download.do?cid=16952
and it contains cloud-init-0.7.5-1.el6.x86_64, and looks like that version without mentioned centos patches: "cloud-init-centos-opennebula.patch", "cloud-init-centos-opennebula-requiretty.patch" and "cloud-init-centos-cloudstack-urlhandling.patch"

So, one can use rhel images, or build their own image using 0.7.4 version from epel:
http://docs.openstack.org/image-guide/content/
https://openstack.redhat.com/Creating_CentOS_and_Fedora_images_ready_for_Openstack
Should we build our own centos image for our customers?

Also, cloud-init-0.7.5-10.el7.centos.1.src.rpm from centos7 contains those pathces too...

[Sergey Kolekonov (skolekonov)]

Alexander, unfortunately, I don't have much information. I think that it would be better if the root cause of this problem will be determined by CentOS engineers. This bug description should be enough.
I don't have access to rhel images. On CentOS 7 image I still see cloud-init warnings so the possibility of failure also persists. It's safer to rebuild images with cloud-init 0.7.4 or try to use rhel package if it doesn't contain opennebula patches.

Dmitry, I suggest the following note, please answer if it's ok.

Instances metadata on CentOS 6.6 cloud images with cloud-init 0.7.5 may be unset in Neutron environments
When an environment is deployed with one of the Neutron topologies and CentOS 6.6 cloud images are used, instances can be unreachable via ssh due to cloud-init failure because users' keypairs and the rest of configuration data may not be set correctly.

[Alexander Bozhenko (alexbozhenko)]

RHEL's cloud-init-0.7.5-1.el6.src.rpm without centos patchces available here:
http://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/RH-COMMON/SRPMS/

[Sergey Kolekonov (skolekonov)]

I checked a cloud-init package from the link you provided and wasn't able to reproduce the bug. So I think that the package from RHEL is also suitable.

[Ilya Shakhat (shakhat)]

Since the issue is caused by cloudinit package and nor related to GRE nor Neutron, I'd consider renaming it to "CentOS 6 cloud image with cloud-init-0.7.5 can not retrieve OpenStack metadata"

[Vasilios Tzanoudakis (vtzanoudakis)]

cloud-init is working for me.

Setup:
2 Controllers HA
ubuntu+GRE Setup
Ceph for all
fuel 5.1.1 (28/10/2014)

using Centos 6.6 cloud image from http://cloud.centos.org/centos/6/images/CentOS-6-x86_64-GenericCloud.qcow2

and epel repo from here : rpm -ivh http://ftp-stud.hs-esslingen.de/pub/epel/6/i386/epel-release-6-8.noarch.rpm

yum install cloud-utils -y

and metadata service is working just fine.

The only thing that doesn't work is auto root fs resize.

I hope this helps

[Alexander Ignatov (aignatov)]

Moved to Invalid since there are no actions we can do with external cloud images with broken cloud-init

[Alexander Bozhenko (alexbozhenko)]

I don't agree that it is invalid.
Looks like the issue is in bundle mos 6.0 only + centos cloud image.
So it probably can be fixed on our side somehow?
Isn't it better to investigate this possibility than advice ALL our customers to build cloud-init rpm without centos patches, and then modify their images ;) ?

[Dmitry Borodaenko (angdraug)]

Reopened as per comment #20. Assigning to docs to incorporate the information from comments #13-#17 in MOS 6.1 release notes as the first priority. Once that is done, lets get back to discussing if there's anything we can do to improve this situation inthe code.

[Paul Mathews (pmathews-i)]

I fixed this for Iwassa by rebuilding the latest f23 RPM (cloud-init-0.7.6-4.20140218bzr1060.fc23.src.rpm) for cent6. I hacked the sources a bit (I couldn't figure out how to explicitly tell it to build for sysvinit), but it fixes key injection.

[Sergey Kolekonov (skolekonov)]

This issue should also be added to release notes for 6.0.x release.

[Sergey Kolekonov (skolekonov)]

Release notes:

Instances metadata on CentOS 6.6 cloud images with cloud-init 0.7.5 may be unset in Neutron environments
When an environment is deployed with one of the Neutron topologies and CentOS 6.6 cloud images are used, instances can be unreachable via ssh due to cloud-init failure because users' keypairs and the rest of configuration data may not be set correctly.

[Bellantuono Daniel (kelfen)]

It seems that the problem there is also on Centos 7 images.

[Timur Nurlygayanov (tnurlygayanov)]

Is it really In Progress for MOS 6.1? Do we have any commits on review with the fix?

[Sergey Kolekonov (skolekonov)]

I was unable to reproduce the original problem on the latest CentOS 7 official cloud images and MOS 7.0 (ISO #187).

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 7.0 and moved to 8.0 (see comments #20-24 for current workaround).

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 6.0 and 6.1 - use the workaround above

[Ilya Shakhat (shakhat)]

Won't fix since it's not Neutron issue (see comments above)