Support Joyent lx-brand environment in smartos datasource

[Robert C Jennings]

[Impact]

 * Cloud-init does not support the datasource presented to LX-Brand
   zones in SmartOS. Without this support cloud-init does not work
   on the Joyent cloud platform's container service.

 * The metadata is provided in the same format as already supported
   for KVM-based instances for the SmartOS datasource (with
   communication over a serial port). For LX-Brand zones the
   metadata is accessed via a socket file.

[Test Case]

 * Install cloud-init on an lx-brand zone image and
   run 'cloud-init init'. No metadata will be found without
   this patch.

 * With the patched cloud-init, run 'cloud-init init' on the
   new lx-brand zones images. The metadata should be discovered
   and the instance should be recognized as being provisioned.
   Perform the same testing on existing KVM-based SmartOS instances
   to ensure there is no regression.

[Regression Potential]

 * [Trusty only] This SRU backports support for the V2 metadata format
   that has been present since the Vivid release. This is well tested
   code and has no user-facing impact. This was done because the
   V2 datasource allows for serial or socket communication. The existing
   Joyent datasource uses the serial port provided in KVM instances,
   however LX-brand zones have a socket instead @
   /native/.zonecontrol/metadata.sock. Additionally the V2 datasource
   provides checksums and response length fields that allow for
   validation of results. Additional testing was performed on Trusty
   SmartOS and KVM instances to validate these changes.

 * To keep code common for KVM (serial) and LX-Brand (socket)
   communication the code had to switch to read byte-by-byte from
   the metadata source file object to avoid reading past the end of
   input on the socket which would block indefinitely. This is a change
   from using readline and will be slower but not significantly so.

 * This builds on the existing SmartOS datasource to enable LX-Brand.
   There is risk that this will break the datasouce on existing KVM
   instances. To mitigate this risk testing has been performed on
   both environments with Trusty, Vivid (out of support today),
   Wily, and Xenial.

[Other Info]

 * This is python 2/3 byte/string safe on Vivid, Wily, and Trusty
   where cloud-init supports python 3.

==== Original Description ====

The Joyent Metadata Protocol Specification (Version 2)[1] allows for serial or socket communication. The existing Joyent datasource uses the serial port provided in KVM instances, however LX-brand zones have a socket instead @ /native/.zonecontrol/metadata.sock.

Detecting we are in a Joyent LX-brand zone is possible by looking at '/bin/uname -v' which will report "BrandZ virtual linux" or checking for the existence of a /native mount where '/native/usr/bin/uname -s' reports
SunOS.

[Robert C Jennings (rcj)]

Patch for trusty attached, branch @
https://git.launchpad.net/~rcj/ubuntu/+source/cloud-init?h=features%2Ftrusty%2Fsdc-lxbrand

[Robert C Jennings (rcj)]

Patch for vivid attached, git branch @
https://git.launchpad.net/~rcj/ubuntu/+source/cloud-init?h=features%2Fvivid%2Fsdc-lxbrand

[Robert C Jennings (rcj)]

Patch for wily attached, git branch @
https://git.launchpad.net/~rcj/ubuntu/+source/cloud-init?h=features%2Fwily%2Fsdc-lxbrand

[Robert C Jennings (rcj)]

Patch for xenial attached, git branch @
https://git.launchpad.net/~rcj/ubuntu/+source/cloud-init?h=features%2Fmaster%2Fsdc-lxbrand

[Robert C Jennings (rcj)]

SmartOS: Add support for Joyent LX-Brand Zones

LX-brand zones on Joyent's SmartOS use a different metadata source
(socket file) than the KVM-based SmartOS virtualization (serial port).
This patch adds support for recognizing the different flavors of
virtualization on SmartOS and setting up a metadata source file object.
After the file object is created, the rest of the code for the datasource
can remain common.

This patch reads the metadata byte-by-byte rather than using readline
because we can not perform a readline on the file-like object for the
socket as this block indefintely waiting for an EOF that is will not be
sent by the host platform. This patch also moves to V2 metadata as it
provides checksum validation and makes reading the metadata much more
reliable.

[Robert C Jennings (rcj)]

Assigning to Scott for patch review. I will update the description with SRU template.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "trusty.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Robert C Jennings (rcj)]

I am going to delete the patches and post them again. I need to update them from the final patch that was merged upstream. They were here prematurely.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1160-0ubuntu1

---------------
cloud-init (0.7.7~bzr1160-0ubuntu1) xenial; urgency=medium

  * New upstream snapshot.
    * SmartOS: Add support for Joyent LX-Brand Zones (LP: #1540965)
    * systemd/power_state: fix power_state when cloud-final exited failure
      (LP: #1449318)

 -- Scott Moser <email address hidden> Thu, 04 Feb 2016 17:22:36 -0500

[Robert C Jennings (rcj)]

The code was accepted upstream and published for Xenial in cloud-init 0.7.7~bzr1160-0ubuntu1 .

[Robert C Jennings (rcj)]

SRU templated added and ready for upload/sponsorship.

Scott, can you ack the upload?
Ben, can you sponsor this?

Thanks.

[Robert C Jennings (rcj)]

New trusty.debdiff based on feedback from Ben. I have cleaned up the changelog (version and path to patch), fixed the diff to have the correct from address, and changed dmi_data() to call util.read_dmi_data() rather than use a local implementation of that util function.

[Robert C Jennings (rcj)]

New wily.debdiff based on feedback from Ben. I have cleaned up the changelog (version and path to patch).

[Ben Howard (darkmuggle-deactivatedaccount)]

Reviewed and upload for 14.04 and 15.10. Pending acceptance into proposed.

[Chris J Arges (arges)]

Hello Robert, or anyone else affected,

Accepted cloud-init into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1149-0ubuntu6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Robert, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.16 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.5-0ubuntu1.16

---------------
cloud-init (0.7.5-0ubuntu1.16) trusty; urgency=medium

  * Joyent Smart DataOS:
    - d/patches/lp-1540965-SmartOS-Add-support-for-Joyent-LX-Brand-Zones.patch:
      SmartOS: Add support for Joyent LX-Brand Zones (LP: #1540965)

 -- Robert C Jennings <email address hidden> Tue, 02 Feb 2016 09:32:02 -0600

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1149-0ubuntu6

---------------
cloud-init (0.7.7~bzr1149-0ubuntu6) wily; urgency=medium

  * Joyent Smart DataOS:
    - d/patches/lp-1540965-SmartOS-Add-support-for-Joyent-LX-Brand-Zones.patch:
      SmartOS: Add support for Joyent LX-Brand Zones (LP: #1540965)

 -- Robert C Jennings <email address hidden> Tue, 02 Feb 2016 16:41:52 -0600