[SRU] cloud-init should check/format Azure empheral disks each boot

[Ben Howard]

SRU Justification

[IMPACT] Users on Windows Azure are guaranteed to have the ephemeral device as ext4 for the first boot. Subsequent boots may result in fuse mounted NTFS file system.

[TEST CASE]
Defined in comment 3.

[Regression Potential] Low. This change is scope only to the Windows Azure datasource and the behavior complies with expected behavior of Windows Azure: the ephemeral disk is not durable between boots. From Microsoft documentation:
> Because data on a resource disk may not be durable across reboots, it is often used by
> applications and processes running in the virtual machine for transient and temporary
> storage of data. It is also used to store page or swap files for the operating system.
(See http://www.windowsazure.com/en-us/documentation/articles/storage-windows-attach-disk/)

Even so, the change to Cloud-init is scope to only replace the ephemeral disk when the disk is 1) NTFS; 2) has a label of "Temporary Storage"; and 3) has no files on it. When the disk matches, cloud-init will turn the code paths for formating the ephemeral disk for that boot only.

[ORIGINAL REPORT]

On Windows Azure, the ephemeral disk should be treated as ephemeral per boot, not per instance.

Microsoft has informed us that under the following conditions an ephemeral disk may disappear:
1. The user resizes the instance
2. A fault causes the instance to move from one physical host to another
3. A machine is shutdown and then started again

Essentially, on Azure, the ephemeral disk is extremely ephemeral. Users who hit any of the above situations are discovering that /mnt is mount with their default NTFS file system.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: cloud-init 0.7.5~bzr964-0ubuntu1 [modified: usr/lib/python2.7/dist-packages/cloudinit/config/cc_disk_setup.py usr/lib/python2.7/dist-packages/cloudinit/config/cc_final_message.py usr/lib/python2.7/dist-packages/cloudinit/config/cc_seed_random.py usr/lib/python2.7/dist-packages/cloudinit/sources/DataSourceAzure.py usr/lib/python2.7/dist-packages/cloudinit/sources/DataSourceCloudSigma.py usr/lib/python2.7/dist-packages/cloudinit/sources/DataSourceSmartOS.py]
ProcVersionSignature: Ubuntu 3.13.0-17.37-generic 3.13.6
Uname: Linux 3.13.0-17-generic x86_64
ApportVersion: 2.13.3-0ubuntu1
Architecture: amd64
Date: Fri Mar 14 17:53:20 2014
PackageArchitecture: all
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)

[Ben Howard (darkmuggle-deactivatedaccount)]

Uploaded branch that makes the disk formating set the semaphore per device and sets disk formating to "always" for the Azure datasource.

[Ben Howard (darkmuggle-deactivatedaccount)]

Test cases:

[ NEW Filesystem created ]
- NTFS file system with label of "Temporary Storage"
  mkfs.ntfs -L "Temporary Storage" -f /dev/sdb1

[ NO New Filesystem created ]
- NTFS file system with label of "Temporary Storage" w/ a file
  mkfs.ntfs -L "Temporary Storage" -f /dev/sdb1; mount /dev/sdb1 /mnt; touch /mnt/test

- NTFS file system, no label
  mkfs.ntfs -f /dev/sdb1

- ext4 file system with label of "Temporary Storage"
  mkfs.ext4 -L "Temporary Storage" /dev/sdb1

- no file system _after_ first boot
  wipefs --all /dev/sdb1

The no file system after first boot should cover the exotic file system and on-disk data formats including LUKS, tar, DRDB and others.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.5~bzr969-0ubuntu1

---------------
cloud-init (0.7.5~bzr969-0ubuntu1) trusty; urgency=medium

  * New upstream snapshot.
    * Azure: Reformat ephemeral disk if it got re-provisioned
      by the cloud on any reboot (LP: #1292648)
    * final_message: fix replacement of upper case keynames (LP: #1286164)
    * seed_random: do not capture output. Correctly provide
      environment variable RANDOM_SEED_FILE to command.
    * CloudSigma: support base64 encoded user-data
 -- Scott Moser <email address hidden> Wed, 19 Mar 2014 14:04:34 -0400

[Dave Walker (davewalker)]

Hello Ben, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dave Walker (davewalker)]

Hello Ben, or anyone else affected,

Accepted cloud-init into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.7.3-0ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ben Howard (darkmuggle-deactivatedaccount)]

Performed through validation testing on multiple instances. Marking as verification done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.12

---------------
cloud-init (0.6.3-0ubuntu1.12) precise-proposed; urgency=low

  * debian/patches/lp-1269626-azure_new_instance.patch: fix handling of new
    instances on Windows Azure. Backport of fix from 14.04 (LP: #1269626).
  * debian/patches/lp-1292648-azure-format-ephemeral-new.patch: Azure,
    re-format ephemeral disk if necessary (LP: #1292648).
 -- Ben Howard <email address hidden> Tue, 18 Mar 2014 10:58:12 -0600

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.3-0ubuntu2.2

---------------
cloud-init (0.7.3-0ubuntu2.2) saucy-proposed; urgency=low

  * debian/patches/lp-1269626-azure_new_instance.patch:
    fix handling of new instances on Windows Azure (LP: #1269626).
  * debian/patches/lp-1292648-azure-format-ephemeral-new.patch:
    re-format ephemeral disk if necessary (LP: #1292648).
 -- Ben Howard <email address hidden> Wed, 19 Mar 2014 16:31:51 -0600

[Ben Howard (darkmuggle-deactivatedaccount)]

This was fixed released a long time ago.

[trialotto (trialotto)]

@Ben,

Bug returns, the issue is still present in 0.7.5-0ubuntu1.3, when partitioning the ephemeral disk.

Interestingly, a partitioning of form

- [ 66, [33,82]] does return the bug (the whole /dev/sdb becomes ntfs after shutdown). Bug is persistent through resizing.

- [ [ 33,82], 66] does not yield any problem at all. Persisten through resizing (resizing does not matter in this case).

Maybe I did something wrong with the cloud config syntax, but that should not be the case.

Still testing all scenario´s. Any idea of the strange situation?

Kind regards

[Paul Meyer (paul-meyer)]

@Ben,

I'm seeing a ~100% repro of this bug on the current 16.04 image in Azure on VM resize. Create a D2_v2 VM, check that fstab has /dev/disk/cloud/azure_resource-part1 on /mnt and that /mnt is mounted as ext4. In that case, provisioning went correctly. Now after resizing this VM, e.g. to a D1_v2, /mnt will be mounted as fuseblk with NTFS as the underlying filesystem. The relevant logs from cloud-init:

Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating new filesystem.
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Checking /dev/sdb against default devices
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Manual request of partition 1 for /dev/sdb1
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Checking device /dev/sdb1
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/blkid', '-c', '/dev/null', '/dev/sdb1'] with allowed return codes [0, 2] (shell=False, capture=True)
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Device /dev/sdb1 has Temporary Storage ntfs
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Device /dev/sdb1 is cleared for formating
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: File system None will be created on /dev/sdb1
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] util.py[DEBUG]: Running command ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps'] with allowed return codes [0] (shell=False, capture=True)
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating file system None on /dev/sdb1
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/sdb1
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/mkfs.ext4', '/dev/sdb1'] with allowed return codes [0] (shell=False, capture=True)
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.038 seconds
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Cloud-init v. 0.7.7 running 'modules:config' at Thu, 04 Aug 2016 18:19:38 +0000. Up 18.80 seconds.
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: 2016-08-04 18:19:38,478 - util.py[WARNING]: Failed during filesystem operation
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']':
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Unexpected error while running command.
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Command: ['/sbin/mkfs.ext4', '/dev/sdb1']
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Exit code: 1
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Reason: -
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Stdout: ''
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Stderr: 'mke2fs 1.42.13 (17-May-2015)\n/dev/sdb1 is mounted; will not make a filesystem here!\n'

The mnt.mount systemd unit was started before cloud-config.service was started, but that looks by design, looking at the ...

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2433