[SRU] xenial cloud-init update

[Scott Moser]

---- Begin SRU Template ----
[Impact]
Please see individual bugs listed for impact. Some significant fixes added here are:
 - user_data: fix error when user-data is not utf-8 decodable (LP: #1532072)
 - support networking information being read from dreamcompute and older
   versions of openstack.
 - support reading and applying networking information on SmartOS virtual
   machine guests.
 - fix reading networking from openstack network_data.json
 - support for renaming devices in a container (LP: #1579130).
 - remove blocking of udev rules (LP: #1577844, LP: #1571761)
 - Apt sources configuration improvements (LP: #1574113)
 - cloud-config specified on kernel command line will now override
   system settings (LP: #1582323).
 - Paths: fix instance path if datasource's id has a '/'. (LP: #1575938)
 - Config Drive: fix check_instance_id signature. (LP: #1575055)

[Test Case]
Most of the changes here are related to data provided by the cloud and properly configuring an instance in response to that data.

Platforms:
 Azure
 NoCloud with network_config
 NoCloud without network_config
 Google GCE
 Openstack Config Drive
 Openstack Metadata Service
 LXD
 Joyent
 Dreamcompute
 MAAS (ephemeral env and installed environment)

For each of these platforms, the following should be done once with user-data provided and once without.
 - Boot the latest available xenial instance (1 with user-data, 1 without)
 - Enable proposed and upgrade
 - save original /var/log/cloud-init.log and /var/log/cloud-init-output.log
 - reboot, check /var/log/cloud-init.log for WARN
   this checks that the upgrade from a previous version did not recognize any
   failures.
 - clean the instance to make it seem as much as possible like a first boot.
   ( cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done )
 - reboot, check /var/log/cloud-init.log for WARN
   this mimics first boot in as close to a real scenario as possible without generating new images with the patched cloud-init.
 - reboot again, check /var/log/cloud-init.log for WARN
   this time check for second-boot issues.

[Regression Potential]
The changes here that are most likely to cause regression are those related to network configuration. cloud-init is now doing a better job of configuring the sytem as the datasource indicated it should be configured. That could, however, have fallout network would be configured differently than before.

[Other Info]
This does represent significant changes. Cloud-init is stabalizing toward a 0.7.8 release, and it is important to get these changes out. They are present in yakkety, and available in a daily ppa at https://code.launchpad.net/~cloud-init-dev/+archive/ubuntu/daily , but being present exposes the new code to many more users.

---- End SRU Template ----

There are several fixes in the current yaketty version of cloud-init that need to make it back to 16.04. These are both bug fixes and more general improvements in code.

This bug is a 'SRU metabug' to cover that SRU. Below is a changelog format with individual bugs listed. The uploaded version will have the bug references removed, but I will mark bugs individually as fix-released and reference this bug.

  * debian/new-upstream-snapshot: minor change supporting revision
    passed in as an argument.
  * New upstream snapshot.
    - user_data: fix error when user-data is not utf-8 decodable (LP: #1532072)
    - write_files: if no permissions are provided, use the default without
      logging a warning.
    - do not write /etc/systemd/network/50-cloud-init-*.link files
      (LP: #1594546)
    - fix several potential errors identified by pylint.
    - move 'main' into cloudinit/cmd/ for easier testing
    - Remove trailing dot from GCE metadata URL (LP: #1581200) [Phil Roche]
    - Refactor cloudinit networking module to improve testing
    - Change missing Cheetah log warning to debug [Andrew Jorgensen]
    - network configuration improvements
      - centrally handle 'dsmode' (DataSource mode) to be 'local' or 'net.
      - support networking information being read on dreamcompute
      - support reading and applying networking information on SmartOS
      - improve reading networking from openstack network_data.json
      - support for renaming devices in a container (LP: #1579130).
      - remove blocking of udev rules (LP: #1577844, LP: #1571761)
    - Apt sources configuration improvements (LP: #1574113)
    - cloud-config specified on kernel command line will now override
      system settings (LP: #1582323).
    - fix timestamp in reporting events.
    - Paths: fix instance path if datasource's id has a '/'. (LP: #1575938)
    - Config Drive: fix check_instance_id signature. (LP: #1575055)
    - cloudstack: Only use DHCPv4 lease files as a datasource (LP: #1576273)

[Chris J Arges (arges)]

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1245-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1246-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

I'm walking through testing of all this as described, i'll post more information tomorrow, but so far it looks good. I've tested sucessfully google compute, azure, openstack config drive, openstack metadata service, dreamcompute and maas at this point.

[Patricia Gaughen (gaughen)]

Dreamhost has validated this works for them.

[Scott Moser (smoser)]

I'm marking this as verification-done.
On each of the platforms below, I
  * launched an instance of current xenial with userdata and without userdata
  * upgraded and collected logs
  * rebooted and collected logs (to verify reboot was fine)
  * cleaned instance and rebooted and collecte logs (to verify first boot)

I did not verify do the no-userdata case on nocloud as it cloud-localds
didn't support it.

                       \userdata/
 platform | Y | N | image
 gce | X | X | /ubuntu-os-cloud/ubuntu-1604-xenial-v20160627
 azure | X | X | Ubuntu_DAILY_BUILD-xenial-16_04-LTS-amd64-server-20160627-en-us-30GB
 nocloud network config | X | | daily cloud-image download 20160627
 nocloud | X | | daily cloud-image download 20160627
 openstack config drive | X | X | ubuntu-released/ubuntu-xenial-16.04-amd64-server-20160627-disk1.img
 openstack md service | X | X | ubuntu-released/ubuntu-xenial-16.04-amd64-server-20160627-disk1.img
 lxd | X | X | ubuntu 16.04 LTS amd64 (daily) (20160627)
 joyent | X | X | ubuntu-certified-15.10 (20160627)
 dreamcompute | X | X | Ubuntu-16.04 (12f6a911-00a2-42eb-8712-d930da2da81f) (serial 20160223-165330)
 MAAS | X | X | ubuntu/xenial amd64/hwe-x 20160627 daily

[Scott Moser (smoser)]

the formatting in the comment above is bad due to non fixed font. it looks better at
 http://paste.ubuntu.com/18196053/

[Juan J. Martínez (jjmartinez)]

Tested 0.7.7~bzr1246-0ubuntu1~16.04.1 and fixed the issue for us.

When will this get into Xenial repos?

Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1246-0ubuntu1~16.04.1

---------------
cloud-init (0.7.7~bzr1246-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream snapshot.
    - fix restoring from a datasource that did not have dsmode (LP: #1596690)

cloud-init (0.7.7~bzr1245-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/new-upstream-snapshot: minor change supporting revision
    passed in as an argument.
  * debian/control: Build-Depends on python3-unittest2
  * SRU Upstream to 16.04 (LP: #1595302).
    - user_data: fix error when user-data is not utf-8 decodable
    - write_files: if no permissions are provided, use the default without
      logging a warning.
    - do not write /etc/systemd/network/50-cloud-init-*.link files
    - fix several potential errors identified by pylint.
    - move 'main' into cloudinit/cmd/ for easier testing
    - Remove trailing dot from GCE metadata URL [Phil Roche]
    - Refactor cloudinit networking module to improve testing
    - Change missing Cheetah log warning to debug [Andrew Jorgensen]
    - network configuration improvements
      - centrally handle 'dsmode' (DataSource mode) to be 'local' or 'net.
      - support networking information being read on dreamcompute
      - support reading and applying networking information on SmartOS
      - improve reading networking from openstack network_data.json
      - support for renaming devices in a container.
      - remove blocking of udev rules
    - Apt sources configuration improvements
    - cloud-config specified on kernel command line will now override
      system settings.
    - fix timestamp in reporting events.
    - Paths: fix instance path if datasource's id has a '/'.
    - Config Drive: fix check_instance_id signature.
    - cloudstack: Only use DHCPv4 lease files as a datasource

 -- Scott Moser <email address hidden> Mon, 27 Jun 2016 16:31:37 -0400

[Chris J Arges (arges)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.