[SRU] cloud-init SmartOS datasource provider is not compliant to datadictionary

Bug #1272115 reported by Ben Howard
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
Fix Released
High
Unassigned
Precise
Fix Released
Medium
Unassigned
Saucy
Fix Released
Medium
Unassigned
Trusty
Fix Released
High
Unassigned

Bug Description

[IMPACT]
Cloud-init provisioning on SmartOS is not compliant per the DataDictionary published on 25 September. The result is that cloud-init provisioned Ubuntu on SmartOS hosts will not behave as user expect. This SRU makes cloud-init's SmartOS datasource compliant on the DataDictionary specification.

[Test Case]
1. Install nodejs
2. Install the CLI tool, "sudo npm install smartdc -g"
3. Launch instance on SmartOS based cloud, with metadata tag of "cloud-init:user-data" set with cloud-init user-data.
4. Install SRU's on instance
5. Run:
   * rm -rf /var/lib/cloud
   * [12.04] cloud-init-cfg init
   * [13.10] cloud-init --debug init
   * Confirm that instance first-boot completed
   * Confirm that cloud-init user-data was consummed
6. From workstation, create a user-script file with the following:
   `touch /root/cloud-init.$(date +%s)`
7. From workstation, run "sdc-updatemachinemetadata <UUID> --metadata-file user-script=<user-script-file>
8. Reboot
9. Confirm that /var/db/user-script exists and that /root/cloud-init.<stamp exists>
10. From workstation, run "sdc-deletemachinemetadata <UUID> --metadata user-script
11. Reboot
12. Confirm that /var/db/user-script is gone
13. Create user-data file with random stuff
14. From workstation, run "sdc-updatemachinemetadata <UUID> --metadata-file user-data=<user-data-file>
15. Reboot
16. Confirm that /var/db/user-data exists

[Regression Potential]
This has low regression potential and will only affect users who are relying on the broken behavior.

[ORIGINAL REPORT]

The SmartOS provider is not compliant to the datadictionary [1] due to a change in the definition.

Specifically:
1. the datasource conflates user-data for the user and cloud-init. cloud-init should use namespaced user-data
2. the datasource should run user-scripts each boot
3. user-scripts should be assumed to be executable with bash unless file magic is present
4. the datacenter name is now scoped to be sdc:datacenter

In order to provide for proper operation on SmartOS, these definiciency need to be addressed.

[1] http://us-east.manta.joyent.com/jmc/public/mdata/datadict.html

Related branches

affects: ubuntu → cloud-init (Ubuntu)
Changed in cloud-init (Ubuntu):
milestone: none → ubuntu-14.04-beta-1
Revision history for this message
Ben Howard (darkmuggle-deactivatedaccount) wrote :

Initial work done, needing to run through a complete build/test cycle.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.5~bzr933-0ubuntu1

---------------
cloud-init (0.7.5~bzr933-0ubuntu1) trusty; urgency=medium

  * debian/control: bump Standards-Version to 3.9.5
  * debian/control: drop boto dependency no longer required in trunk.
  * New upstream snapshot.
    * ConfigDrive: consider partitions labelled correctly as possible sources.
    * find root filesystem for resizing in cases where there is no initramfs
    * removal of dependency on python-boto
    * vendor-data support, and usage of that in Joyent datasource.
    * change default output to be logged to /var/log/cloud-init-output.log
    * SeLinuxGuard: Cast file path to string. (LP: #1260072)
    * drop support for resizing via parted (LP: #1212492)
    * SmartOS: changes to address changes in platform (LP: #1272115)
    * FreeBSD support.
 -- Scott Moser <email address hidden> Fri, 24 Jan 2014 22:41:57 -0500

Changed in cloud-init (Ubuntu):
status: In Progress → Fix Released
Scott Moser (smoser)
Changed in cloud-init (Ubuntu Precise):
status: New → In Progress
Changed in cloud-init (Ubuntu Saucy):
status: New → In Progress
Changed in cloud-init (Ubuntu Precise):
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Saucy):
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Precise):
assignee: nobody → Ben Howard (utlemming)
Changed in cloud-init (Ubuntu Saucy):
assignee: nobody → Ben Howard (utlemming)
summary: - cloud-init SmartOS datasource provider is not compliant to
+ [SRU] cloud-init SmartOS datasource provider is not compliant to
datadictionary
description: updated
Changed in cloud-init (Ubuntu Precise):
status: In Progress → New
Changed in cloud-init (Ubuntu Saucy):
status: In Progress → New
Revision history for this message
Chris Halse Rogers (raof) wrote :

Hm. In “Regression potential” you say “will only affect users who are relying on the broken behavior”.

What effect will this have on users who *are* relying on the current behaviour? Do you have any estimate of how many users might be relying on it? There's a high bar to clear for an SRU that breaks a user's current setup - that's a deliberate regression.

Revision history for this message
Ben Howard (darkmuggle-deactivatedaccount) wrote :

> What effect will this have on users who *are* relying on the current behaviour?

The effect will be minimial, if any. SmartOS has the concept of user-data, which is a channel for sending raw data. In the case of cloud-init, users would be more likely to be confused by something acting on the data. By having cloud-init follow the documented use-case, cloud-init becomes functional.

Further, I would argue that if cloud-init is acting on user-data, there is actually a danger depending on the payload of the user-data. For example, if the user-data payload is a destructive script that is supposed to be run by something other than cloud-init it could result in system failure.

Given that we introduced SmartOS recently wtih 13.10, the impact here will be minimial. There is no public cloud vendor that is currently using cloud-init. Finally, this change was requested by the SmartOS upstream maintainers.

Revision history for this message
Chris Halse Rogers (raof) wrote : Please test proposed package

Hello Ben, or anyone else affected,

Accepted cloud-init into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.7.3-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Saucy):
status: New → Fix Committed
tags: added: verification-needed
Changed in cloud-init (Ubuntu Precise):
status: New → Fix Committed
Revision history for this message
Chris Halse Rogers (raof) wrote :

Hello Ben, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Robert C Jennings (rcj) wrote :

Verified on saucy with success. Attaching cloud-init.log for saucy.

Revision history for this message
Robert C Jennings (rcj) wrote :

Log file and notes from verification on precise.
- Testing was successful on a host with plain text in the cloud-init:user-data metadata field.
- Testing failed on a host which had base64 data in the cloud-init:user-data metadata field.

I also saw that the user-script ran twice, producing two files in /root/

Revision history for this message
Ben Howard (darkmuggle-deactivatedaccount) wrote :

@rcj: this is as expected. The cloud-init user-data needs to have a metadata key of base64_keys="cloud-init:user-data" in order for it to work.

The logs indicate that the instance your on is lacking a ubuntu user. Can you re-run the test, but create the ubuntu user before running the test?

Revision history for this message
Ben Howard (darkmuggle-deactivatedaccount) wrote :

Marking as verification-done. The logs from comment #8, actually indicate that the test cases succeeded, however, it was failing due to using a non-official Ubuntu image.

I verified the test cases. This SRU is good to go for release.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.10

---------------
cloud-init (0.6.3-0ubuntu1.10) precise-proposed; urgency=low

  * debian/patches/lp-1272115-fix_smartos_compliance.patch: Fix compliance for
    SmartOS data handling (LP: #1272115):
    - cloud-init user-data is now named spaced as "cloud-init:user-data"
    - user-data is written to /var/db/mdata-user-data
    - user-scripts are fetched and executed each boot
    - datacenter/region is now named spaced as "sdc:datacenter"
 -- Ben Howard <email address hidden> Mon, 27 Jan 2014 13:57:24 -0700

Changed in cloud-init (Ubuntu Precise):
status: Fix Committed → Fix Released
Revision history for this message
Colin Watson (cjwatson) wrote : Update Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.3-0ubuntu2.1

---------------
cloud-init (0.7.3-0ubuntu2.1) saucy-proposed; urgency=low

  * debian/patches/lp-1272115-fix_smartos_compliance.patch: Fix compliance for
    SmartOS data handling (LP: #1272115):
    - straight backport from 14.04 LTS development of SmartOS datasource
      and datasource test suite, sans vendordata.
    - cloud-init user-data is now named spaced as "cloud-init:user-data"
    - user-data is written to /var/db/mdata-user-data
    - user-scripts are fetched and executed each boot
    - datacenter/region is now named spaced as "sdc:datacenter"
 -- Ben Howard <email address hidden> Tue, 28 Jan 2014 11:19:14 -0700

Changed in cloud-init (Ubuntu Saucy):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.