[SRU] cloud-init must check/format Azure ephemeral disks each boot

[Stephen A. Zarkos]

[Impact]
Sometimes when rebooting an instance in Azure, a different ephemeral disk will be presented. Azure ephemeral disks are presented as NTFS disks, but we reformat them to ext4. With the last cloud-init upload, we shifted to using /dev/disk symlinks to mount these disks (in case they are not presented as the same physical device). Unfortunately, the code that determines if we have a new ephemeral disk was not updated to handle symlinks, so never detects a new disk.

[Test Case]
1) Boot an Azure instance and install the new cloud-init.
2) Change the size of the instance using the Azure web interface (as this near-guarantees that the ephemeral disk will be replaced with a new one). This will reboot the instance.
3) Once the instance is rebooted, SSH in and confirm that:
 a) An ext4 ephemeral disk is mounted at /mnt, and
 b) cloud-init.log indicates that a fabric formatted ephemeral disk was found on this boot.

[Regression Potential]
Limited; two LOCs change, to dereference symlinks instead of using paths verbatim.

[Original Report]
Ubuntu 14.04.3 (20150805) on Azure with cloud-init package 0.7.5-0ubuntu1.8.

On Azure cloud-init prepares the ephemeral device as ext4 for the first boot. However, if the VM is ever moved to another Host for any reason, then a new ephemeral disk might be provided to the VM. This ephemeral disk is NTFS formatted, so for subsequent reboots cloud-init must detect and reformat the new disk as ext4. However, with cloud-init 0.7.5-0ubuntu1.8 subsequent boots may result in fuse mounted NTFS file system.

This issue occurred in earlier versions of cloud-init, but was fixed with bug 1292648 (https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1292648). So this appears to be a regression.

Repro:
  - Create an Ubuntu 14.04.3 VM on Azure
  - Resize the VM to a larger size (this typically moves the VM)
  - Log in and run 'blkid' to show an ntfs formatted ephemeral disk:

    # blkid
    /dev/sdb1: LABEL="Temporary Storage" UUID="A43C43DD3C43A95E" TYPE="ntfs"

Expected results:
  - After resizing the ephemeral disk should be formatted as ext4.

[Ben Howard (utlemming)]

From /var/log/cloud-init.log:

Sep 1 15:05:49 daily-trusty-0901-5a83547c4a [CLOUDINIT] cc_disk_setup.py[DEBUG]: Device /dev/sdb1 is cleared for formating
Sep 1 15:05:49 daily-trusty-0901-5a83547c4a [CLOUDINIT] cc_disk_setup.py[DEBUG]: File system None will be created on /dev/sdb1
Sep 1 15:05:49 daily-trusty-0901-5a83547c4a [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating file system None on /dev/sdb1
Sep 1 15:05:49 daily-trusty-0901-5a83547c4a [CLOUDINIT] cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/sdb1
Sep 1 15:05:49 daily-trusty-0901-5a83547c4a [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/mkfs.ext4', '/dev/sdb1'] with allowed
return codes [0] (shell=False, capture=True)
Sep 1 15:05:49 daily-trusty-0901-5a83547c4a [CLOUDINIT] util.py[DEBUG]: Creating fs for /dev/disk/azure/resource took 0.009 seconds
Sep 1 15:05:49 daily-trusty-0901-5a83547c4a [CLOUDINIT] util.py[WARNING]: Failed during filesystem operation#012Failed to exec of '['/
sbin/mkfs.ext4', '/dev/sdb1']':#012Unexpected error while running command.#012Command: ['/sbin/mkfs.ext4', '/dev/sdb1']#012Exit code: 1
#012Reason: -#012Stdout: ''#012Stderr: 'mke2fs 1.42.9 (4-Feb-2014)\n/dev/sdb1 is mounted; will not make a filesystem here!\n'
Sep 1 15:05:49 daily-trusty-0901-5a83547c4a [CLOUDINIT] util.py[DEBUG]: Failed during filesystem operation#012Failed to exec of '['/sb
in/mkfs.ext4', '/dev/sdb1']':#012Unexpected error while running command.#012Command: ['/sbin/mkfs.ext4', '/dev/sdb1']#012Exit code: 1#0
12Reason: -#012Stdout: ''#012Stderr: 'mke2fs 1.42.9 (4-Feb-2014)\n/dev/sdb1 is mounted; will not make a filesystem here!\n'#012Tracebac
k (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/cloudinit/config/cc_disk_setup.py", line 75, in handle#012 fu
nc=mkfs, args=(definition,))#012 File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 1875, in log_time#012 ret = func(*
args, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/cloudinit/config/cc_disk_setup.py", line 861, in mkfs#012 raise Exceptio
n("Failed to exec of '%s':\n%s" % (fs_cmd, e))#012Exception: Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']':#012Unexpected error
while running command.#012Command: ['/sbin/mkfs.ext4', '/dev/sdb1']#012Exit code: 1#012Reason: -#012Stdout: ''#012Stderr: 'mke2fs 1.42.
9 (4-Feb-2014)\n/dev/sdb1 is mounted; will not make a filesystem here!\n'
Sep 1 15:05:49 daily-trusty-0901-5a83547c4a [CLOUDINIT] util.py[DEBUG]: Writing to /var/lib/cloud/instances/aab62d9d9f19492cb676f42410
5b60ea/sem/config_mounts - wb: [420] 19 bytes

[Ben Howard (utlemming)]

ApportVersion: 2.14.1-0ubuntu3.12
Architecture: amd64
DistroRelease: Ubuntu 14.04
Package: cloud-init 0.7.5-0ubuntu1.8
PackageArchitecture: all
ProcVersionSignature: Ubuntu 3.19.0-26.28~14.04.1-generic 3.19.8-ckt4
Tags: trusty uec-images
Uname: Linux 3.19.0-26-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True

[Ben Howard (utlemming)]

apport information

[Ben Howard (utlemming)]

apport information

[Dan Watkins (daniel-thewatkins)]

I've worked this out: we check if a disk is mounted (in order to unmount it) using the /dev/disk symlink, but it's the dereferenced device that appears in the mount table. If we dereference the symlink before checking the mount table, this goes away.

This is what the attached branch does.

[Dan Watkins (daniel-thewatkins)]

I've refreshed the patches for trusty and vivid to include the changes that hit -proposed earlier today. precise-proposed didn't have a change, so that debdiff remains unchanged.

[Ben Howard (utlemming)]

I've tested the patches and confirmed that they work. Uploads are pending in -proposed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1141-0ubuntu1

---------------
cloud-init (0.7.7~bzr1141-0ubuntu1) wily; urgency=medium

  * New upstream snapshot.
    * fix bug in cloud-init main preventing any usage.

cloud-init (0.7.7~bzr1140-0ubuntu1) wily; urgency=medium

  * New upstream snapshot.
    * Handle symlink mount points in mount_cb (LP: #1490796).
    * sync curtin reporting changes back to cloud-init.

 -- Scott Moser <email address hidden> Thu, 03 Sep 2015 11:36:37 -0400

[Brian Murray (brian-murray)]

Hello Stephen, or anyone else affected,

Accepted cloud-init into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1091-0ubuntu7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Stephen, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Stephen, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Stephen A. Zarkos (stevez)]

Tested cloud-init/0.7.5-0ubuntu1.10 and it resolves the issue for me.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.20

---------------
cloud-init (0.6.3-0ubuntu1.20) precise; urgency=medium

  * debian/patches/lp-1490796-azure-fix-mount_cb-for-symlinks.patch:
      - Fix a regression caused by switching to /dev/disk symlinks
        (LP: #1490796).

 -- Daniel Watkins <email address hidden> Wed, 02 Sep 2015 13:24:28 +0100

[Stéphane Graber (stgraber)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.5-0ubuntu1.10

---------------
cloud-init (0.7.5-0ubuntu1.10) trusty; urgency=medium

  [ Daniel Watkins ]
  * d/patches/lp-1490796-azure-fix-mount_cb-for-symlinks.patch:
      - Fix a regression caused by switching to /dev/disk symlinks
        (LP: #1490796).

 -- Ben Howard <email address hidden> Wed, 02 Sep 2015 10:57:30 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1091-0ubuntu7

---------------
cloud-init (0.7.7~bzr1091-0ubuntu7) vivid; urgency=medium

  [ Daniel Waktins ]
  * d/patches/lp-1490796-azure-fix-mount_cb-for-symlinks.patch:
      - Fix a regression caused by switching to /dev/disk symlinks
        (LP: #1490796).

 -- Ben Howard <email address hidden> Wed, 02 Sep 2015 11:18:30 -0600