[SRU] Azure: cloud-init should use VM unique ID

Bug #1506187 reported by Stephen A. Zarkos
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
Undecided
Dan Watkins
cloud-init (Ubuntu)
Fix Released
Critical
Unassigned
Precise
Fix Released
High
Dan Watkins
Trusty
Fix Released
Critical
Unassigned
Vivid
Won't Fix
Critical
Unassigned
Wily
Fix Released
Critical
Unassigned
Xenial
Fix Released
Critical
Unassigned

Bug Description

SRU JUSTIFICATION

[IMPACT] On Azure, the InstanceID is currently detected via a fabric provided XML file. With the new CRP stack, this ID is not guaranteed to be stable. As a result instances may go re-provision upon reboot.

[FIX] Use DMI data to detect the instance ID and migrate existing instances to the new ID.

[REGRESSION POTENTIAL] The fix is both in the cloud-init code and in the packaging. If the instance ID is not properly migrated, then a reboot may trigger re-provisioning.

[TEST CASES]
1. Boot instance on Azure.
2. Apply cloud-init from -proposed. A migration message should apply.
3. Get the new instance ID:
   $ sudo cat /sys/class/dmi/id/product_uuid
4. Confirm that /var/lib/cloud/instance is a symlink to /var/lib/cloud/instances/<UUID from step 3>
5. Re-install cloud-init and confirm that migration message is NOT displayed.

[TEST CASE 2]
1. Build new cloud-image from -proposed
2. Boot up instance
3. Confirm that /sys/class/dmi/id/product_uuid is used to get instance ID (see /var/log/cloud-init.log)

[ORIGINAL REPORT]
The Azure datasource currently uses the InstanceID from the SharedConfig.xml file. On our new CRP stack, this ID is not guaranteed to be stable and could change if the VM is deallocated. If the InstanceID changes then cloud-init will attempt to reprovision the VM, which could result in temporary loss of access to the VM.

Instead cloud-init should switch to use the VM Unique ID, which is guaranteed to be stable everywhere for the lifetime of the VM. The VM unique ID is explained here: https://azure.microsoft.com/en-us/blog/accessing-and-using-azure-vm-unique-id/

In short, the unique ID is available via DMI, and can be accessed with the command 'dmidecode | grep UUID' or even easier via sysfs in the file "/sys/devices/virtual/dmi/id/product_uuid".

Steve

Related branches

Revision history for this message
Dan Watkins (oddbloke) wrote :

Hi Steve,

Thanks for letting us know! Before I get started, can you confirm that we can safely remove all retrieval of instance ID from SharedConfig.xml? (That is to say, are there any Azure platforms etc. where we might not have the DMI value available?)

Thanks,

Dan

Changed in cloud-init (Ubuntu):
assignee: nobody → Dan Watkins (daniel-thewatkins)
status: New → Confirmed
Revision history for this message
Stephen A. Zarkos (stevez) wrote :

Yes, this is supported in all prod environments. You can remove the dependency on InstanceID.

Thanks!
Steve

Changed in cloud-init (Ubuntu):
importance: Undecided → Critical
summary: - Azure: cloud-init should use VM unique ID
+ [SRU] Azure: cloud-init should use VM unique ID
Dan Watkins (oddbloke)
Changed in cloud-init:
status: New → In Progress
assignee: nobody → Dan Watkins (daniel-thewatkins)
Changed in cloud-init (Ubuntu Xenial):
assignee: Dan Watkins (daniel-thewatkins) → nobody
description: updated
tags: added: patch
Revision history for this message
Ben Howard (darkmuggle-deactivatedaccount) wrote :

Per feedback, instance migration logic is proposed to be in the preinst.

Revision history for this message
Ben Howard (darkmuggle-deactivatedaccount) wrote :

Attached debdiff of all patch and logic applied for instance ID migration.

Changed in cloud-init (Ubuntu Precise):
assignee: nobody → Ben Howard (utlemming)
Changed in cloud-init (Ubuntu Trusty):
assignee: nobody → Ben Howard (utlemming)
Changed in cloud-init (Ubuntu Vivid):
assignee: nobody → Ben Howard (utlemming)
Changed in cloud-init (Ubuntu Wily):
assignee: nobody → Ben Howard (utlemming)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.7~bzr1156-0ubuntu1

---------------
cloud-init (0.7.7~bzr1156-0ubuntu1) xenial; urgency=medium

  * New upstream snapshot.
  * d/cloud-init.preinst: migrate Azure instance ID from old ID to stable
    ID (LP: #1506187).

 -- Ben Howard <email address hidden> Tue, 17 Nov 2015 11:59:49 -0700

Changed in cloud-init (Ubuntu Xenial):
status: Confirmed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Stephen, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Trusty):
status: New → Fix Committed
tags: added: verification-needed
Mathew Hodson (mhodson)
Changed in cloud-init (Ubuntu Precise):
importance: Undecided → Critical
Changed in cloud-init (Ubuntu Trusty):
importance: Undecided → Critical
Changed in cloud-init (Ubuntu Vivid):
importance: Undecided → Critical
Changed in cloud-init (Ubuntu Wily):
importance: Undecided → Critical
Revision history for this message
Stephen A. Zarkos (stevez) wrote :

I tested cloud-init 0.7.5-0ubuntu1.15 from -proposed for Trusty on Azure and confirmed that the new package resolves the issue.

The upgrade path is also working as expected. I first created an Ubuntu 14.04 VM using ARM. After shutting down the VM via the Azure portal (portal.azure.com) and then restarting it I saw that the Instance ID changed (you can see this in /var/log/waagent.log). However, the symlink in /var/lib/cloud/instances/ that links the old Instance ID to the VM unique ID had been created, and so cloud-init did not attempt to reprovision the VM.

Thanks!
Steve

tags: added: verification-done-trusty
removed: verification-needed
Revision history for this message
Martin Pitt (pitti) wrote :

Hello Stephen, or anyone else affected,

Accepted cloud-init into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1149-0ubuntu5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Wily):
status: New → Fix Committed
tags: added: verification-needed
Revision history for this message
Ben Howard (darkmuggle-deactivatedaccount) wrote :

Verified for 15.10 and 14.04.

tags: added: verification-done verification-done-wily
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.5-0ubuntu1.15

---------------
cloud-init (0.7.5-0ubuntu1.15) trusty; urgency=medium

  * Microsoft Azure:
    - d/patches/lp-1506244-azure-ssh-key-values.patch: AZURE: Add support
      and preference for fabric provided public SSH public key values over
      fingerprints (LP: #1506244).
    - use stable VM instance ID over SharedConfig.xml (LP: #1506187):
      - d/patches/lp-1506187-azure_use_unique_vm_id.patch: use DMI data for
        the stable VM instance ID
      - d/cloud-init.preinst: migrate existing instances to stable VM instance
        ID on upgrade from prior versions of cloud-init.

 -- Ben Howard <email address hidden> Tue, 17 Nov 2015 10:02:24 -0700

Changed in cloud-init (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.7~bzr1149-0ubuntu5

---------------
cloud-init (0.7.7~bzr1149-0ubuntu5) wily; urgency=medium

  * Microsoft Azure: use stable VM instance ID over SharedConfig.xml
    (LP: #1506187):
    - d/patches/lp-1506187-azure_use_unique_vm_id.patch: use DMI data for the
      stable VM instance ID
    - d/cloud-init.preinst: migrate existing instances to stable VM instance
      ID on upgrade from prior versions of cloud-init.

 -- Ben Howard <email address hidden> Fri, 20 Nov 2015 17:26:09 -0700

Changed in cloud-init (Ubuntu Wily):
status: Fix Committed → Fix Released
Dan Watkins (oddbloke)
Changed in cloud-init:
status: In Progress → Fix Released
Changed in cloud-init (Ubuntu Precise):
importance: Critical → High
Changed in cloud-init (Ubuntu Vivid):
status: New → Won't Fix
Changed in cloud-init (Ubuntu Precise):
assignee: Ben Howard (utlemming) → nobody
Dan Watkins (oddbloke)
Changed in cloud-init (Ubuntu Precise):
assignee: nobody → Dan Watkins (daniel-thewatkins)
status: New → In Progress
Revision history for this message
Dan Watkins (oddbloke) wrote :
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Stephen, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Precise):
status: In Progress → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Revision history for this message
Dan Watkins (oddbloke) wrote :

I've performed the upgrade test case, and am now booting an instance from a -proposed image to test the install case.

tags: added: verification-done-precise
tags: removed: verification-needed
Revision history for this message
Dan Watkins (oddbloke) wrote :

The first boot test case also passed, so this is good to go.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.25

---------------
cloud-init (0.6.3-0ubuntu1.25) precise; urgency=medium

  * Microsoft Azure: Use stable VM instance ID over SharedConfig.xml
    (LP: #1506187)
    - d/patches/lp-1506187-azure_use_unique_vm_id.patch: use DMI data for
      the stable VM instance ID
    - d/cloud-init.preinst: migrate existing instances to stable VM instance
      ID on upgrade from prior versions of cloud-init.

 -- Daniel Watkins <email address hidden> Mon, 25 Apr 2016 16:53:07 -0400

Changed in cloud-init (Ubuntu Precise):
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.