cloud-init user-data mime conversion fails on base64 encoded data

[Ben Howard]

Cloud-init's conversion of user-data to mime fails when the user-data is base64 encoded due to the Python2 to Python3 switch. base64.b64decode in Python 2 returns a string, whilst Python3 returns a byte stream.

Consider:
      import base64

      hi = "aGkK"
      print(type(base64.b64decode(hi)).__name__)
      if 'hi' in str(base64.b64decode(hi)):
         print("works")
      if 'hi' in base64.b64decode(hi):
         print("works on Py2")

ben@prongs:~$ python /tmp/proof.py
str
works
works on Py2
ben@prongs:~$ python3 /tmp/proof.py
bytes
works
Traceback (most recent call last):
  File "/tmp/proof.py", line 10, in <module>
    if 'hi' in base64.b64decode(hi):
TypeError: Type str doesn't support the buffer API

[Ben Howard (utlemming)]

Root cause appears to be that cloud-init is _not_ running the init modules. I'm still not sure what is preventing the init modules from running. But it is clear from the logs that it is skipping the init modules and going straight to the config modules.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1423972/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Ben Howard (utlemming)]

apport information

[Ben Howard (utlemming)]

apport information

[Ben Howard (utlemming)]

Cloud-init is failing silently and exiting with an exit code of 1.

{
 "v1": {
  "init-local": {
   "finished": 1424466074.5986445,
   "start": 1424466074.3635466,
   "errors": []
  },
  "modules-final": {
   "finished": 1424466081.407799,
   "start": 1424466081.248662,
   "errors": []
  },
  "stage": null,
  "modules-config": {
   "finished": 1424466080.7607048,
   "start": 1424466080.3998847,
   "errors": []
  },
  "datasource": null,
  "init": {
   "finished": 1424466408.8851824,
   "start": 1424466408.6761255,
   "errors": [
    "'str' does not support the buffer interface"
   ]
  }
 }
}

[Ben Howard (utlemming)]

Okay, traced this tricky little bug down. The bug was hidden due to a try/except block capturing the exception and then ignoring it.

Traceback (most recent call last):
  File "/usr/bin/cloud-init", line 617, in <module>
    sys.exit(main())
  File "/usr/bin/cloud-init", line 613, in main
    get_uptime=True, func=functor, args=(name, args))
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1990, in log_time
    ret = func(*args, **kwargs)
  File "/usr/bin/cloud-init", line 497, in status_wrapper
    ret = functor(name, args)
  File "/usr/bin/cloud-init", line 264, in main_init
    init.update()
  File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 326, in update
    self._store_userdata()
  File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 332, in _store_userdata
    processed_ud = "%s" % (self.datasource.get_userdata())
  File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 82, in get_userdata
    self.userdata = self.ud_proc.process(self.get_userdata_raw())
  File "/usr/lib/python3/dist-packages/cloudinit/user_data.py", line 97, in process
    self._process_msg(convert_string(blob), accumulating_msg)
  File "/usr/lib/python3/dist-packages/cloudinit/user_data.py", line 340, in convert_string
    if "mime-version:" in data[0:4096].lower():
TypeError: 'str' does not support the buffer interface

[Ben Howard (utlemming)]

Confirmed patch and submitted MP upstream. While my patch works, it assumes that user-data is always a string, which may not hold true. Asking smoser for a review.

[Ben Howard (utlemming)]

ppa:utlemming/lp1423972 has patch applied

[Scott Moser (smoser)]

--- cloudinit/user_data.py 2015-02-20 22:19:07 +0000
+++ cloudinit/user_data.py 2015-02-23 20:08:06 +0000
@@ -336,8 +336,8 @@
         raw_data = ''
     if not headers:
         headers = {}
- data = util.decomp_gzip(raw_data)
- if "mime-version:" in str(data[0:4096]).lower():
+ data = util.decode_binary(util.decomp_gzip(raw_data))
+ if "mime-version:" in data[0:4096]).lower():
         msg = email.message_from_string(data)
         for (key, val) in headers.items():
             _replace_header(msg, key, val)

I think the above is a sane patch (as sane as it can be at this point).
but i'm not convinced yet. need to actually test.