base64 user-data broken in OVF datasource

[Julian Vassev]

==== Begin SRU Template ====
[Impact]
Users of the OVF datasource can not provide user-data that is base64 encoded.

[Test Case]
$ rel=xenial
$ burl="http://cloud-images.ubuntu.com/daily/server/$rel/current/"
$ disk="$rel-server-cloudimg-amd64-disk1.img"
$ wget "$burl/$disk" -O "$disk"

# prepare the ovftransport.iso
$ git clone http://git.launchpad.net/cloud-init
$ ( cd cloud-init/doc/sources/ovf && ./make-iso ovf-env.xml.tmpl user-data --output - ) > ovftransport.iso

# create a temp disk for patching and patch in -proposed
$ qemu-img create -f qcow2 -b "$disk" disk.img
$ sudo mount-image-callback --system-resolvconf disk.img -- chroot _MOUNTPOINT_ \
    sh -c 'l=/etc/apt/sources.list.d/proposed.list;
       echo deb http://archive.ubuntu.com/ubuntu/ xenial-proposed main > "$l" &&
       apt-get -q update && apt-get install -qy cloud-init'

$ qemu-system-x86_64 -enable-kvm \
   -device virtio-net-pci,netdev=net00 -netdev type=user,id=net00 \
   -hda disk.img -cdrom ovftransport.iso -m 512 -curses -snapshot

# you will see an 'ovfdemo' login prompt, log in as root:passw0rd
# verify it worked by cat /var/lib/cloud/instance/user-data.txt

[Regression Potential]
very low chance for regression, a guest with user-data would be broken previously.

==== End SRU Template ====

I am using cloudint 0.7.7~bzr1256-0ubuntu1~16.04.1 on Ubuntu 16.04.1 LTS.

When I pass yaml-formatted user-data over the OVF datasource over the ISO transport the yaml string fails to be parsed.

I tracked this to minidom's inability to handle newlines in an attribute. The xml bellow works for coreos but breaks under cloud-init:

<PropertySection>
         <Property oe:key="user-data" oe:value="#cloud-config
---
users:
  - name: "test"
    passwd: "$1$NVdKo9MI$PlHasfYsufCHbP1Hh9TMz/"
    groups:
      - "sudo"
"/>
</PropertySection>

If I use base64-encoded user-data I get this error:

  Sep 01 12:07:43 sof2-lab3-dhcp371 cloud-init[3248]: 2016-09-01 12:07:43,854 - __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'b'I2Nsb3VkLWNvbmZpZwotLS0K'...'
  Sep 01 12:07:43 sof2-lab3-dhcp371 cloud-init[3248]: [CLOUDINIT] __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'b'I2Nsb3VkLWNvbmZpZwotLS0K'...'

Is there a way to pass user-data as a single-line string that doesn't confuse minidom?

[Scott Moser (smoser)]

fwiw, i'm pretty sure it is invalid ovf to have newlines in the value of property.
minidom might be doing the right thing.

the fix for this in cloud-init is to allow you to pass in base64 encoded string in that user-data.

that path was broken in python3.

Julian put a branch to fix that at https://code.launchpad.net/~jvassev/cloud-init/+git/cloud-init/+ref/fix-b64decode-ovf .

I've put a branch up at https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/305496 with a unit test, and i'm going to pull that in now.

[Scott Moser (smoser)]

fixed in 0.7.8.

[Chris J Arges (arges)]

Hello Julian, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7-31-g65ace7b-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Julian Vassev (jvassev)]

Hi Chris,
Upgrading to 0.7.7-31-g65ace7b-0ubuntu1~16.04.1 from -proposed fixes my issue.

[Martin Pitt (pitti)]

Hello Julian, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-1-g3705bb5-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

verified as shown in SRU template.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.8-1-g3705bb5-0ubuntu1~16.04.1

---------------
cloud-init (0.7.8-1-g3705bb5-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream release 0.7.8.
  * New upstream snapshot.
    - systemd: put cloud-init.target After multi-user.target (LP: #1623868)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * debian/control: add Breaks of older versions of walinuxagent (LP: #1623570)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/control: fix missing dependency on python3-serial,
    and make SmartOS datasource work.
  * debian/cloud-init.templates fix capitalisation in template so
    dpkg-reconfigure works to select OpenStack. (LP: #1575727)
  * d/README.source, d/control, d/new-upstream-snapshot, d/rules: sync
    with yakkety for changes due to move to git.
  * d/rules: change PYVER=python3 to PYVER=3 to adjust to upstream change.
  * debian/rules, debian/cloud-init.install: remove install file
    to ensure expected files are collected into cloud-init deb.
    (LP: #1615745)
  * debian/dirs: remove obsolete / unused file.
  * upstream move from bzr to git.
  * New upstream snapshot.
    - Allow link type of null in network_data.json [Jon Grimm] (LP: #1621968)
    - DataSourceOVF: fix user-data as base64 with python3 (LP: #1619394)
    - remove obsolete .bzrignore
    - systemd: Better support package and upgrade. (LP: #1576692, #1621336)
    - tests: cleanup tempdirs in apt_source tests
    - apt config conversion: treat empty string as not provided. (LP: #1621180)
    - Fix typo in default keys for phone_home [Roland Sommer] (LP: #1607810)
    - salt minion: update default pki directory for newer salt minion.
      (LP: #1609899)
    - bddeb: add --release flag to specify the release in changelog.
    - apt-config: allow both old and new format to be present.
      [Christian Ehrhardt] (LP: #1616831)
    - python2.6: fix dict comprehension usage in _lsb_release. [Joshua Harlow]
    - Add a module that can configure spacewalk. [Joshua Harlow]
    - add install option for openrc [Matthew Thode]
    - Generate a dummy bond name for OpenStack (LP: #1605749)
    - network: fix get_interface_mac for bond slave, read_sys_net for ENOTDIR
    - azure dhclient-hook cleanups
    - Minor cleanups to atomic_helper and add unit tests.
    - Fix Gentoo net config generation [Matthew Thode]
    - distros: fix get_primary_arch method use of os.uname [Andrew Jorgensen]
    - Apt: add new apt configuration format [Christian Ehrhardt]
    - Get Azure endpoint server from DHCP client [Brent Baude]
    - DigitalOcean: use the v1.json endpoint [Ben Howard]
    - MAAS: add vendor-data support (LP: #1612313)
    - Upgrade to a configobj package new enough to work [Joshua Harlow]
    - ConfigDrive: recognize 'tap' as a link type. (LP: #1610784)
    - NoCloud: fix bug providing network-interfaces via meta-data.
      (LP: 1577982)
    - Add distro tags on config modules that should have it [Joshua Harlow]
    - ChangeLog: update changelog for previous commit.
    - add ntp config module [Ryan Harper]
    - SmartOS: more improvement...

[Chris J Arges (arges)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2715