read_seeded broken

[Scott Moser]

=== Begin SRU Information ===
[Impact]
cloud-init cannot read a "seed". Many of the data-sources allow a seed from a url. This is useful in testing, especially with the NoCloud seed from the command line.

One of the functions in cloud-init simply does not work with python-3, and the patch makes that work. Patch is applied to upstream and present in wily.

[Test Case]
Run the attached test-bug-1455233. Without the patch applied, the system will boot and show error like in the serial.log:
  util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceNoCloud.DataSourceNoCloudNet'> failed

Additionally, a simple test case can be done more directly by simply running
 $ echo "my-userdata" > user-data
 $ echo "instance-id: FOO" > meta-data
 $ python -m SimpleHTTPServer 8999 &
 $ python3 -c 'from cloudinit import util; print(util.read_seeded("http://localhost:8999/", retries=0))'
127.0.0.1 - - [09/Jun/2015 01:24:05] "GET /meta-data HTTP/1.1" 200 -
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 842, in read_seeded
    if md_resp.ok():
AttributeError: 'str' object has no attribute 'ok'

The added test case in the build process pushes the code through this function.

[Regression Potential]
This code was 100% broken in python3, so likelyhood of regression is very low.
=== End SRU Information ===

util.read_seeded uses load_tfile_or_url, but then treats the return value as if it was a response.

this regressed in revno 1067.

[Scott Moser (smoser)]

fixed in trunk at 1102

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1106-0ubuntu1

---------------
cloud-init (0.7.7~bzr1106-0ubuntu1) wily; urgency=medium

  * New upstream snapshot.
    * Azure: remove strict dependency on walinux-agent, but still utilize
      it for the time being.
    * fix read_seeded that is used in seeding user-data and meta-data
      from additional locations (LP: #1455233)
    * fix bug preventing partitioning of disks in many cases. (LP: #1311463)
    * Azure: do not override hostname if user has set it (LP: #1375252)
    * Fix GCE datasource not handling per-instance SSH keys (LP: #1403617)
    * Allow specifying of uid in user/group config.

 -- Scott Moser <email address hidden> Fri, 15 May 2015 17:04:19 -0400

[Scott Moser (smoser)]

running this will:
 a.) download amd64 vivid cloud iamge
 b.) convert it to qcow2 (removing compression)
 c.) create a datasource seed and put make that available on port 32999
 d.) boot a kvm guest with kernel command line pointing at the seed .

running it with PATCH_IMAGE=cloud-inig.deb will patch the image with the provided cloud-init.deb and then run.

[Chris J Arges (arges)]

Hello Scott, or anyone else affected,

Accepted cloud-init into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1091-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

verified.

 * downloaded test-bug-1455233 from attachment
 * ran it, it downloaded daily/server/vivid/20150609/vivid-server-cloudimg-amd64-disk1.img
 * it booted vm, and on console i see failure:
[ 13.659828] cloud-init[729]: 2015-06-10 18:13:05,254 - util.py[WARNING]: Getting data from <class cloudinit.sources.DataSourceNoCloud.DataSourceNoCloudNet'> failed
 and in serial.log, it shows
  util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceNoCloud.DataSourceNoCloudNet'> failed

 * updated the image to -proposed cloud image:
   # work around 1463963 to get us some space
   $ sudo mount-image-callback --system-resolvconf --system-mounts \
       vivid-server-cloudimg-amd64-disk1.qcow2 -- sh -ec \
       'dev=$(awk "\$2 == mp { print \$1 }" "mp=$MOUNTPOINT" /proc/mounts);
        resize2fs $dev'

   $ sudo mount-image-callback --system-resolvconf --system-mounts \
      vivid-server-cloudimg-amd64-disk1.qcow2 -- \
      chroot _MOUNTPOINT_ sh -c '
        rel=$(lsb_release -sc);
        echo "deb http://archive.ubuntu.com/ubuntu $rel-proposed main" |
        tee /etc/apt/sources.list.d/proposed.list &&
        apt-get update -qy && apt-get install cloud-init grub-legacy-ec2 &&
        apt-cache policy cloud-init'
   ...
   cloud-init:
     Installed: 0.7.7~bzr1091-0ubuntu3
     Candidate: 0.7.7~bzr1091-0ubuntu3
     Version table:
    *** 0.7.7~bzr1091-0ubuntu3 0
           500 http://archive.ubuntu.com/ubuntu/ vivid-proposed/main amd64 Packages
           100 /var/lib/dpkg/status
        0.7.7~bzr1091-0ubuntu2 0
           500 http://archive.ubuntu.com/ubuntu/ vivid-updates/main amd64 Packages
        0.7.7~bzr1091-0ubuntu1 0
           500 http://archive.ubuntu.com/ubuntu/ vivid/main amd64 Packages

 * ran ./test-bug-1455233
   logged in at login prompt with 'ubuntu' and 'passw0rd'

[Scott Moser (smoser)]

Then, just for fun inside the booted system I verified additionally with:
$ python3 -c 'from cloudinit import util; print(util.read_seeded("http://localhost:8999/", retries=0))'
({'instance-id': 'FOO'}, b'my-userdata\n')

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1091-0ubuntu3

---------------
cloud-init (0.7.7~bzr1091-0ubuntu3) vivid-proposed; urgency=medium

  * d/README.source, debian/cherry-pick-rev: improve packaging tool
  * fix read_seeded method needed for seeding cloud-init via kernel
    command line (LP: #1455233).

 -- Scott Moser <email address hidden> Mon, 08 Jun 2015 15:39:32 -0400

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Scott Moser (smoser)]

This is fixed in cloud-init 0.7.7