[SRU] Custom vendor data causes cloud-init failure on 0.7.5

Bug #1469260 reported by Edmund Rhudy
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
Medium
Unassigned
cloud-init (Ubuntu)
Fix Released
Medium
Unassigned
Trusty
Fix Released
Medium
Felipe Reyes
Utopic
Invalid
Medium
Unassigned

Bug Description

[Impact]

When a vendor data json provides a dictionary without a 'cloud-init' key, cloud-init renders a non functional user-data, so any configuration (i.e. ssh public keys to use) is missed.

This prevents cloud providers from publishing a vendor data that is not intended to be consumed by cloud-init.

This patch checks for the existence of 'cloud-init' key and tries to get None, a string or a list as value, if this process fails or cloud-init key is missing the vendor data is set to None.

[Test Case]

* deploy an OpenStack cloud (easy right? :) )
  - the easiest way is to branch https://code.launchpad.net/~ost-maintainers/openstack-charm-testing/trunk and run: juju deployer -c default.yaml -d -v -s 10 trusty-kilo
* configure vendor data
  - Edit /etc/nova/nova.conf in neutron-gateway unit(s), include the following two lines:
    vendordata_driver=nova.api.metadata.vendordata_json.JsonFileVendorData
    vendordata_jsonfile_path=/etc/nova/vendordata.json
  - Create /etc/nova/vendordata.json in neutron-gateway unit(s) with the following content:
    {"custom": {"a": 1, "b": [2, 3]}}
  - Restart nova-api-metadata (sudo service nova-api-metadata restart)
* Launch an instance using trusty

Expected result:
- the new instance is launched and is accesible according to the configuration used

Actual result:
- cloud-init fails to configure the ssh public key

[Regression Potential]

* This patch is already part of Vivid and there are no known issues.
* This proposed fix was tested with a custom image and no issues were detected.

[Other Info]

I encountered this issue when adding custom vendor data via nova-compute. Originally the bug manifested as SSH host key generation failing to fire when vendor data was present (example vendor data below).

{"msg": "", "uuid": "4996e2b67d2941818646481453de1efe", "users": [{"username": "erhudy", "sshPublicKeys": [], "uuid": "erhudy"}], "name": "TestTenant"}

I launched a volume-backed instance, waited for it to fail, then terminated it and mounted its root volume to examine the logs. What I found was that cloud-init was failing to process vendor-data into MIME multipart (note the absence of the line that indicates that cloud-init is writing vendor-data.txt.i):

2015-06-25 21:41:02,178 - util.py[DEBUG]: Writing to /var/lib/cloud/instance/obj.pkl - wb: [256] 9751 bytes
2015-06-25 21:41:02,178 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/65c9fb0c-0700-4f87-a22f-c59534e98dfb/user-data.txt - wb: [384] 0 bytes
2015-06-25 21:41:02,184 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/65c9fb0c-0700-4f87-a22f-c59534e98dfb/user-data.txt.i - wb: [384] 345 bytes
2015-06-25 21:41:02,185 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/65c9fb0c-0700-4f87-a22f-c59534e98dfb/vendor-data.txt - wb: [384] 234 bytes
2015-06-25 21:41:02,185 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)

After following the call chain all the way down, I found the problematic code in user_data.py:

# Coverts a raw string into a mime message
def convert_string(raw_data, headers=None):
    if not raw_data:
        raw_data = ''
    if not headers:
        headers = {}
    data = util.decomp_gzip(raw_data)
    if "mime-version:" in data[0:4096].lower():
        msg = email.message_from_string(data)
        for (key, val) in headers.iteritems():
            _replace_header(msg, key, val)
    else:
        mtype = headers.get(CONTENT_TYPE, NOT_MULTIPART_TYPE)
        maintype, subtype = mtype.split("/", 1)
        msg = MIMEBase(maintype, subtype, *headers)
        msg.set_payload(data)
    return msg

raw_data in the case that is failing is a dictionary rather than the expected string, so slicing into data causes a TypeError: unhashable type exception.

I think this bug was fixed after a fashion in 0.7.7, where the call to util.decomp_gzip() is now wrapped by util.decode_binary(), which appears to always return a string.

Related branches

Revision history for this message
Scott Moser (smoser) wrote :

marking trunk fix-released and ubuntu fixed-released per bug openers suggestion that it was fixed in 0.7.7 . I have not reproduced though.

Changed in cloud-init (Ubuntu):
status: New → Fix Released
Changed in cloud-init:
status: New → Fix Released
Changed in cloud-init (Ubuntu):
importance: Undecided → Medium
Changed in cloud-init:
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Trusty):
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Utopic):
importance: Undecided → Medium
Felipe Reyes (freyes)
Changed in cloud-init (Ubuntu Trusty):
assignee: nobody → Felipe Reyes (freyes)
Revision history for this message
Edmund Rhudy (erhudy) wrote :

I can reliably reproduce this issue via the following steps:

1) Create blah.py under nova/api/metadata/ with these contents:

from nova.api.metadata import base

class CustomVendordata(base.VendorDataDriver):
    def __init__(self, *args, **kwargs):
        super(BcpcMetadata, self).__init__(*args, **kwargs)

    def get(self):
        return {"msg": "", "uuid": "4996e2b67d2941818646481453de1efe", "users": [{"username": "erhudy", "sshPublicKeys": [], "uuid": "erhudy"}], "name": "TestTenant"}

2) Configure Nova to use that class as the vendordata_driver in nova.conf:

[DEFAULT]
vendordata_driver = nova.api.metadata.blah.CustomVendordata

3) Restart nova-api.

4) Launch an Ubuntu cloud instance on that hypervisor.

5) Experience cloud-init fury:

2015-07-22 15:30:56,075 - util.py[WARNING]: Running ssh-authkey-fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_ssh_authkey_fingerprints.pyc'>) failed
ec2:
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
-----END SSH HOST KEY KEYS-----

Revision history for this message
Edmund Rhudy (erhudy) wrote :

Just doing something simple like

data = str(util.decomp_gzip(raw_data))

causes it to work, because data's now a string and hashable and so the range data[0:4096] works (whether the content is actually what it's supposed to be is another question).

Revision history for this message
Edmund Rhudy (erhudy) wrote :

Screwed up the above script, it should be

from nova.api.metadata import base

class CustomVendordata(base.VendorDataDriver):
    def __init__(self, *args, **kwargs):
        super(CustomVendordata, self).__init__(*args, **kwargs)

    def get(self):
        return {"msg": "", "uuid": "4996e2b67d2941818646481453de1efe", "users": [{"username": "erhudy", "sshPublicKeys": [], "uuid": "erhudy"}], "name": "TestTenant"}

Revision history for this message
Edmund Rhudy (erhudy) wrote :

Changing the class to return a string rather than a dictionary avoids the cloud-init explosion. That runs counter to all the documentation that says "use JSON", but it's the workaround we'll use for now (serializing to YAML).

Revision history for this message
Felipe Reyes (freyes) wrote :

A vendor data file should be a dict with a cloud-init key and the value for that key is expected to be a string or a list[0], the docs provide a simple example[1]

Here is another example that will add a user called cloudy, upgrade the system and install htop:

{"cloud-init": "#cloud-config\nusers:\n - name: cloudy\n ssh-import-id: cloudy\npackage_upgrade: True\npackages:\n - htop"}

So a yaml cloud-config compatible has to be passed a string.

[0] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/helpers/openstack.py#L470
[1] http://cloudinit.readthedocs.org/en/latest/topics/datasources.html?highlight=vendor#vendor-data

Revision history for this message
Felipe Reyes (freyes) wrote :

cloud-init misbehaves when a vendor data json comes with information that won't be consumed by it, a fix was added in rev 1013[0]. I backported this patch to Trusty and prepared a patched image, having a custom vendor data doesn't break cloud-init functionality.

I'll submit a SRU to fix this in Trusty as soon as possible.

[0] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/1013

Revision history for this message
Felipe Reyes (freyes) wrote :

Utopic is already EOL, so I'm marking it as Invalid

Changed in cloud-init (Ubuntu Utopic):
status: New → Invalid
tags: added: sts
tags: added: openstack
Felipe Reyes (freyes)
description: updated
summary: - Custom vendor data causes cloud-init failure on 0.7.5
+ [SRU] Custom vendor data causes cloud-init failure on 0.7.5
Changed in cloud-init (Ubuntu Trusty):
status: New → In Progress
Revision history for this message
Felipe Reyes (freyes) wrote :

Rebased patch on top of latest cloud-init available in trusty-updates.

Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Edmund, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Felipe Reyes (freyes) wrote :

The package available in trusty-proposed fixes this bug.

$nova console-log 7a2e57a7-df5c-4c61-8f69-c98a41a1de5f
...
ubuntu login: Cloud-init v. 0.7.5 running 'modules:final' at Tue, 22 Sep 2015 02:39:29 +0000. Up 26.16 seconds.
2015-09-22 02:39:30,077 - util.py[WARNING]: Running ssh-authkey-fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_ssh_authkey_fingerprints.pyc'>) failed
ec2:
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
-----END SSH HOST KEY KEYS-----
Cloud-init v. 0.7.5 finished at Tue, 22 Sep 2015 02:39:30 +0000. Datasource DataSourceOpenStack [net,ver=2]. Up 26.33 seconds

$ nova console-log trusty-patched
...
Cloud-init v. 0.7.5 running 'modules:final' at Tue, 22 Sep 2015 02:41:53 +0000. Up 30.54 seconds.
ci-info: ++++++++++Authorized keys from /home/ubuntu/.ssh/authorized_keys for user ubuntu+++++++++++
ci-info: +---------+-------------------------------------------------+---------+-------------------+
ci-info: | Keytype | Fingerprint (md5) | Options | Comment |
ci-info: +---------+-------------------------------------------------+---------+-------------------+
ci-info: | ssh-rsa | bd:37:fc:87:b5:3a:c6:4b:86:cf:ab:e1:29:81:aa:7f | - | Generated-by-Nova |
ci-info: +---------+-------------------------------------------------+---------+-------------------+
ec2:
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 1024 35:8e:8c:4d:86:ff:2c:a0:01:da:f0:fd:db:dc:da:94 root@trusty-patched (DSA)
ec2: 256 72:f7:f1:6e:5d:74:ad:ad:13:25:2a:3f:25:83:8b:5c root@trusty-patched (ECDSA)
ec2: 256 33:cb:4f:2a:70:08:30:8f:dc:3e:60:68:1a:19:a9:cc root@trusty-patched (ED25519)
ec2: 2048 60:cb:95:fe:11:1f:01:14:fc:44:6e:a6:43:1b:81:31 root@trusty-patched (RSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGp6OpR/cqHVcQmks95ANFCbgi67E+Opr4CmhOIDZl1zRkT6hwW2qGZa3P0Y9PAisNorgaidbpq+Vfi13MKsV5o= root@trusty-patched
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPy5DB2nOuBzAL+R+0x4xPn3bfdGCSlLgEK1B7PIUzyB root@trusty-patched
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCY7NO6k1FBQ27JfcQSuewZ74NNCWhC0ZllDtGi3j/WoQooGn7zRWcY36yibvFu7JP1k9ajrPqUD9P0wlDLhfdSaOzyKWOEqP/pSW2/o1R+o/L7+mOFdZoOblzjd+hHvjkZ+xXMRAn0o/fvRo7kButtKyijscGvUZ8yAFRRNauseJxzsMBBdX0/AWLRNG6LFMjrsWwgooDOwjCNK7tRSU8KulBEGMEMrvnScbNmoDvwYwUZFVK8tW8MSeWH+pbD8GR6rJjSmiTikkH1ChZI6o+A+AkHhVaNoH044Ddea6x3CvwEYO9HXOADmZZfwUgx2cym6V6D/jw4QtqriXCdspf root@trusty-patched
-----END SSH HOST KEY KEYS-----
Cloud-init v. 0.7.5 finished at Tue, 22 Sep 2015 02:41:54 +0000. Datasource DataSourceOpenStack [net,ver=2]. Up 30.81 seconds

tags: added: verification-done
removed: verification-needed
Revision history for this message
Stéphane Graber (stgraber) wrote : Update Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.5-0ubuntu1.11

---------------
cloud-init (0.7.5-0ubuntu1.11) trusty; urgency=medium

  [ Felipe Reyes ]
  * d/patches/fix-consumption-of-vendor-data.patch:
    - Fix consumption of vendor-data in OpenStack to allow namespacing
      (LP: #1469260).

  [ Scott Moser ]
  * d/patches/lp-1461242-generate-ed25519-host-keys.patch:
    - ssh: generate ed25519 host keys if supported (LP: #1461242)

 -- Scott Moser <email address hidden> Fri, 11 Sep 2015 20:22:00 -0400

Changed in cloud-init (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.