[SRU] Custom vendor data causes cloud-init failure on 0.7.5

[Edmund Rhudy]

[Impact]

When a vendor data json provides a dictionary without a 'cloud-init' key, cloud-init renders a non functional user-data, so any configuration (i.e. ssh public keys to use) is missed.

This prevents cloud providers from publishing a vendor data that is not intended to be consumed by cloud-init.

This patch checks for the existence of 'cloud-init' key and tries to get None, a string or a list as value, if this process fails or cloud-init key is missing the vendor data is set to None.

[Test Case]

* deploy an OpenStack cloud (easy right? :) )
  - the easiest way is to branch https://code.launchpad.net/~ost-maintainers/openstack-charm-testing/trunk and run: juju deployer -c default.yaml -d -v -s 10 trusty-kilo
* configure vendor data
  - Edit /etc/nova/nova.conf in neutron-gateway unit(s), include the following two lines:
    vendordata_driver=nova.api.metadata.vendordata_json.JsonFileVendorData
    vendordata_jsonfile_path=/etc/nova/vendordata.json
  - Create /etc/nova/vendordata.json in neutron-gateway unit(s) with the following content:
    {"custom": {"a": 1, "b": [2, 3]}}
  - Restart nova-api-metadata (sudo service nova-api-metadata restart)
* Launch an instance using trusty

Expected result:
- the new instance is launched and is accesible according to the configuration used

Actual result:
- cloud-init fails to configure the ssh public key

[Regression Potential]

* This patch is already part of Vivid and there are no known issues.
* This proposed fix was tested with a custom image and no issues were detected.

[Other Info]

I encountered this issue when adding custom vendor data via nova-compute. Originally the bug manifested as SSH host key generation failing to fire when vendor data was present (example vendor data below).

{"msg": "", "uuid": "4996e2b67d2941818646481453de1efe", "users": [{"username": "erhudy", "sshPublicKeys": [], "uuid": "erhudy"}], "name": "TestTenant"}

I launched a volume-backed instance, waited for it to fail, then terminated it and mounted its root volume to examine the logs. What I found was that cloud-init was failing to process vendor-data into MIME multipart (note the absence of the line that indicates that cloud-init is writing vendor-data.txt.i):

2015-06-25 21:41:02,178 - util.py[DEBUG]: Writing to /var/lib/cloud/instance/obj.pkl - wb: [256] 9751 bytes
2015-06-25 21:41:02,178 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/65c9fb0c-0700-4f87-a22f-c59534e98dfb/user-data.txt - wb: [384] 0 bytes
2015-06-25 21:41:02,184 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/65c9fb0c-0700-4f87-a22f-c59534e98dfb/user-data.txt.i - wb: [384] 345 bytes
2015-06-25 21:41:02,185 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/65c9fb0c-0700-4f87-a22f-c59534e98dfb/vendor-data.txt - wb: [384] 234 bytes
2015-06-25 21:41:02,185 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)

After following the call chain all the way down, I found the problematic code in user_data.py:

# Coverts a raw string into a mime message
def convert_string(raw_data, headers=None):
    if not raw_data:
        raw_data = ''
    if not headers:
        headers = {}
    data = util.decomp_gzip(raw_data)
    if "mime-version:" in data[0:4096].lower():
        msg = email.message_from_string(data)
        for (key, val) in headers.iteritems():
            _replace_header(msg, key, val)
    else:
        mtype = headers.get(CONTENT_TYPE, NOT_MULTIPART_TYPE)
        maintype, subtype = mtype.split("/", 1)
        msg = MIMEBase(maintype, subtype, *headers)
        msg.set_payload(data)
    return msg

raw_data in the case that is failing is a dictionary rather than the expected string, so slicing into data causes a TypeError: unhashable type exception.

I think this bug was fixed after a fashion in 0.7.7, where the call to util.decomp_gzip() is now wrapped by util.decode_binary(), which appears to always return a string.

[Scott Moser (smoser)]

marking trunk fix-released and ubuntu fixed-released per bug openers suggestion that it was fixed in 0.7.7 . I have not reproduced though.

[Edmund Rhudy (erhudy)]

I can reliably reproduce this issue via the following steps:

1) Create blah.py under nova/api/metadata/ with these contents:

from nova.api.metadata import base

class CustomVendordata(base.VendorDataDriver):
    def __init__(self, *args, **kwargs):
        super(BcpcMetadata, self).__init__(*args, **kwargs)

    def get(self):
        return {"msg": "", "uuid": "4996e2b67d2941818646481453de1efe", "users": [{"username": "erhudy", "sshPublicKeys": [], "uuid": "erhudy"}], "name": "TestTenant"}

2) Configure Nova to use that class as the vendordata_driver in nova.conf:

[DEFAULT]
vendordata_driver = nova.api.metadata.blah.CustomVendordata

3) Restart nova-api.

4) Launch an Ubuntu cloud instance on that hypervisor.

5) Experience cloud-init fury:

2015-07-22 15:30:56,075 - util.py[WARNING]: Running ssh-authkey-fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_ssh_authkey_fingerprints.pyc'>) failed
ec2:
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
-----END SSH HOST KEY KEYS-----

[Edmund Rhudy (erhudy)]

Just doing something simple like

data = str(util.decomp_gzip(raw_data))

causes it to work, because data's now a string and hashable and so the range data[0:4096] works (whether the content is actually what it's supposed to be is another question).

[Edmund Rhudy (erhudy)]

Screwed up the above script, it should be

from nova.api.metadata import base

class CustomVendordata(base.VendorDataDriver):
    def __init__(self, *args, **kwargs):
        super(CustomVendordata, self).__init__(*args, **kwargs)

    def get(self):
        return {"msg": "", "uuid": "4996e2b67d2941818646481453de1efe", "users": [{"username": "erhudy", "sshPublicKeys": [], "uuid": "erhudy"}], "name": "TestTenant"}

[Edmund Rhudy (erhudy)]

Changing the class to return a string rather than a dictionary avoids the cloud-init explosion. That runs counter to all the documentation that says "use JSON", but it's the workaround we'll use for now (serializing to YAML).

[Felipe Reyes (freyes)]

A vendor data file should be a dict with a cloud-init key and the value for that key is expected to be a string or a list[0], the docs provide a simple example[1]

Here is another example that will add a user called cloudy, upgrade the system and install htop:

{"cloud-init": "#cloud-config\nusers:\n - name: cloudy\n ssh-import-id: cloudy\npackage_upgrade: True\npackages:\n - htop"}

So a yaml cloud-config compatible has to be passed a string.

[0] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/helpers/openstack.py#L470
[1] http://cloudinit.readthedocs.org/en/latest/topics/datasources.html?highlight=vendor#vendor-data

[Felipe Reyes (freyes)]

cloud-init misbehaves when a vendor data json comes with information that won't be consumed by it, a fix was added in rev 1013[0]. I backported this patch to Trusty and prepared a patched image, having a custom vendor data doesn't break cloud-init functionality.

I'll submit a SRU to fix this in Trusty as soon as possible.

[0] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/1013

[Felipe Reyes (freyes)]

Utopic is already EOL, so I'm marking it as Invalid

[Felipe Reyes (freyes)]

Rebased patch on top of latest cloud-init available in trusty-updates.

[Chris J Arges (arges)]

Hello Edmund, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Felipe Reyes (freyes)]

The package available in trusty-proposed fixes this bug.

$nova console-log 7a2e57a7-df5c-4c61-8f69-c98a41a1de5f
...
ubuntu login: Cloud-init v. 0.7.5 running 'modules:final' at Tue, 22 Sep 2015 02:39:29 +0000. Up 26.16 seconds.
2015-09-22 02:39:30,077 - util.py[WARNING]: Running ssh-authkey-fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_ssh_authkey_fingerprints.pyc'>) failed
ec2:
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
-----END SSH HOST KEY KEYS-----
Cloud-init v. 0.7.5 finished at Tue, 22 Sep 2015 02:39:30 +0000. Datasource DataSourceOpenStack [net,ver=2]. Up 26.33 seconds

$ nova console-log trusty-patched
...
Cloud-init v. 0.7.5 running 'modules:final' at Tue, 22 Sep 2015 02:41:53 +0000. Up 30.54 seconds.
ci-info: ++++++++++Authorized keys from /home/ubuntu/.ssh/authorized_keys for user ubuntu+++++++++++
ci-info: +---------+-------------------------------------------------+---------+-------------------+
ci-info: | Keytype | Fingerprint (md5) | Options | Comment |
ci-info: +---------+-------------------------------------------------+---------+-------------------+
ci-info: | ssh-rsa | bd:37:fc:87:b5:3a:c6:4b:86:cf:ab:e1:29:81:aa:7f | - | Generated-by-Nova |
ci-info: +---------+-------------------------------------------------+---------+-------------------+
ec2:
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 1024 35:8e:8c:4d:86:ff:2c:a0:01:da:f0:fd:db:dc:da:94 root@trusty-patched (DSA)
ec2: 256 72:f7:f1:6e:5d:74:ad:ad:13:25:2a:3f:25:83:8b:5c root@trusty-patched (ECDSA)
ec2: 256 33:cb:4f:2a:70:08:30:8f:dc:3e:60:68:1a:19:a9:cc root@trusty-patched (ED25519)
ec2: 2048 60:cb:95:fe:11:1f:01:14:fc:44:6e:a6:43:1b:81:31 root@trusty-patched (RSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGp6OpR/cqHVcQmks95ANFCbgi67E+Opr4CmhOIDZl1zRkT6hwW2qGZa3P0Y9PAisNorgaidbpq+Vfi13MKsV5o= root@trusty-patched
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPy5DB2nOuBzAL+R+0x4xPn3bfdGCSlLgEK1B7PIUzyB root@trusty-patched
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCY7NO6k1FBQ27JfcQSuewZ74NNCWhC0ZllDtGi3j/WoQooGn7zRWcY36yibvFu7JP1k9ajrPqUD9P0wlDLhfdSaOzyKWOEqP/pSW2/o1R+o/L7+mOFdZoOblzjd+hHvjkZ+xXMRAn0o/fvRo7kButtKyijscGvUZ8yAFRRNauseJxzsMBBdX0/AWLRNG6LFMjrsWwgooDOwjCNK7tRSU8KulBEGMEMrvnScbNmoDvwYwUZFVK8tW8MSeWH+pbD8GR6rJjSmiTikkH1ChZI6o+A+AkHhVaNoH044Ddea6x3CvwEYO9HXOADmZZfwUgx2cym6V6D/jw4QtqriXCdspf root@trusty-patched
-----END SSH HOST KEY KEYS-----
Cloud-init v. 0.7.5 finished at Tue, 22 Sep 2015 02:41:54 +0000. Datasource DataSourceOpenStack [net,ver=2]. Up 30.81 seconds

[Stéphane Graber (stgraber)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.5-0ubuntu1.11

---------------
cloud-init (0.7.5-0ubuntu1.11) trusty; urgency=medium

  [ Felipe Reyes ]
  * d/patches/fix-consumption-of-vendor-data.patch:
    - Fix consumption of vendor-data in OpenStack to allow namespacing
      (LP: #1469260).

  [ Scott Moser ]
  * d/patches/lp-1461242-generate-ed25519-host-keys.patch:
    - ssh: generate ed25519 host keys if supported (LP: #1461242)

 -- Scott Moser <email address hidden> Fri, 11 Sep 2015 20:22:00 -0400