Azure reboot with unformatted ephemeral drive won't mount reformatted volume

[Jason Zions]

If an Azure VM is rebooted after being moved to a different host (e.g. after a deallocate operation or after a service-heal to remove a bad host from service), the ephemeral drive exposed to the VM is reset to the default state (NTFS format). The Azure data source detects this and marks cc_disk_setup and cc_mounts to be run. While cc_disk_setup reformats the volume as desired, cc_mounts determines that the appropriate mount request was already in /etc/fstab (as setup during initial provisioning). Since the normal boot process would already have mounted everything according to fstab, the cc_mounts logic is "no mount -a is required". This is not true in this scenario.

[Ryan Harper (raharper)]

Could you supply the systemd journal from the system and a cloud-init collect-logs?

I'd like to confirm that the .mount units for the ephemeral drive were attempted, but failed due to the disk not being ext4 (yet).

[Jason Zions (jasonzio)]

I confirmed that myself in the logs; the timestamp on the failed mnt.mount is about three seconds earlier than the timestamp on the cloud-init startup message. The error message is indeed a complaint about not being able to mount an "ntfs" formatted volume. I'll try to find a relevant repro and get you the logs you seek.

[Jason Zions (jasonzio)]

Per your request, the results of collect-logs

[Jason Zions (jasonzio)]

The contents of /var/log/messages on this CentOS system, corresponding to the VM start-after-deallocate operation covered by the previously-uploaded collect-logs bundle. In particular, see Apr 23 00:06:05 (date/times in UTC) for the attempt to mount the NTFS ephemeral volume. Meanwhile, in cloud-init.log, you'll see that cc_disk_setup doesn't run until 00:06:27.

When this VM was first created, /var/log/messages said nothing at all about /dev/disk/cloud/azure_resource-part1 or about NTFS, which is exactly as one would expect for the first-time-ever boot of the VM.

[Server Team CI bot (server-team-bot)]

This bug is fixed with commit acc25d8d to cloud-init on branch master.
To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=acc25d8d

[Chad Smith (chad.smith)]

This bug is believed to be fixed in cloud-init in version 19.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3370