Activity log for bug #1611074

Date Who What changed Old value New value Message
2016-08-08 18:26:11 Paul Meyer bug added bug
2016-08-08 18:26:11 Paul Meyer attachment added cloud-init.log https://bugs.launchpad.net/bugs/1611074/+attachment/4716876/+files/cloud-init.log
2016-08-08 21:56:34 Paul Meyer bug task added cloud-init
2016-08-26 15:34:24 Joshua Powers bug added subscriber Ubuntu Server Team
2016-08-31 19:04:33 Jon Grimm bug added subscriber Jon Grimm
2016-09-02 17:40:12 Jon Grimm cloud-init (Ubuntu): assignee Scott Moser (smoser)
2016-09-07 15:55:50 Martin Pitt bug added subscriber Martin Pitt
2016-09-17 02:43:49 Scott Moser cloud-init: status New Confirmed
2016-09-17 02:43:51 Scott Moser cloud-init (Ubuntu): status New Confirmed
2016-09-17 02:43:55 Scott Moser cloud-init: importance Undecided High
2016-09-17 02:43:57 Scott Moser cloud-init (Ubuntu): importance Undecided High
2016-09-19 20:09:18 Launchpad Janitor merge proposal linked https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/306145
2016-09-20 18:33:59 Scott Moser description After resizing a 16.04 VM on Azure, the VM is presented with a new ephemeral drive (of a different size), which initially is NTFS formatted. Cloud-init tries to format the appropriate partition ext4, but fails because it is mounted. Cloud-init has unmount logic for exactly this case in the get_data call on the Azure data source, but this is never called because fresh cache is found. Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: start: init-network/check-cache: attempting to read from cache [trust] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False) Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Read 5950 bytes from /var/lib/cloud/instance/obj.pkl Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] stages.py[DEBUG]: restored from cache: DataSourceAzureNet [seed=/dev/sr0] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/check-cache: SUCCESS: restored from cache: DataSourceAzureNet [seed=/dev/sr0] ... Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating file system None on /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/mkfs.ext4', '/dev/sdb1'] with allowed return codes [0] (shell=False, capture=True) Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.052 seconds Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[WARNING]: Failed during filesystem operation#012Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']':#012Unexpected error while running command.#012Command: ['/sbin/mkfs.ext4', '/dev/sdb1']#012Exit code: 1#012Reason: -#012Stdout: ''#012Stderr: 'mke2fs 1.42.13 (17-May-2015)\n/dev/sdb1 is mounted; will not make a filesystem here!\n' $ lsb_release -rd Description: Ubuntu 16.04.1 LTS Release: 16.04 $ cat /etc/cloud/build.info build_name: server serial: 20160721 ~$ dpkg -l cloud-init Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=====================-=====================-===================================================================== ii cloud-init 0.7.7~bzr1256-0ubuntu all Init scripts for cloud instances We're seeing ~100% repro of this bug on resize, where the only success cases are caused by another bug that messes up fstab and prevents mounting of the drive. === Begin SRU Template === [Impact] In some cases, cloud-init writes entries to /etc/fstab, and on azure it will even format a disk for mounting and then write the entry for that 'ephemeral' disk there. A supported operation on Azure is to "resize" the system. When you do this the system is shut down, resized (given larger/faster disks and more CPU) and then brought back up. In that process, the "ephemeral" disk re-initialized to its original NTFS format. The designed goal is for cloud-init to recognize this situation and re-format the disk to ext4. The problem is that the mount of that disk happens before cloud-init can reformat. Thats because the entry in fstab has 'auto' and is automatically mounted. The end result is that after resize operation the user will be left with the ephemeral disk mounted at /mnt and having a ntfs filesystem rather than ext4. [Test Case] The text in comment 3 describes how to recreate by the original reporter. Another way to do this is to just re-format the ephemeral disk as ntfs and then reboot. The result *should* be that after reboot it comes back up and has an ext4 filesystem on it. 1.) boot system on azure (for this, i use https://gist.github.com/smoser/5806147, but you can use web ui or any other way). 2.) unmount the ephemeral disk $ umount /mnt 3.) repartition it so that mkfs.ntfs does less and is faster This is not strictly necessary, but mkfs.ntfs can take upwards of 20 minutes. shrinking /dev/sdb2 to be 200M means it will finish in < 1 minute. $ disk=/dev/disk/cloud/azure_resource $ part=/dev/disk/cloud/azure_resource-part1 $ echo "2048,$((2*1024*100)),7" | sudo sfdisk "$disk" $ time mkfs.ntfs --quick "$part" 4.) reboot 5.) expect that /proc/mounts has /dev/disk/cloud/azure_resource-part1 as ext4 and that fstab has x-systemd.requires in it. $ awk '$2 == "/mnt" { print $0 }' /proc/mounts /dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0 $ awk '$2 == "/mnt" { print $0 }' /etc/fstab /dev/sdb1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2 [Regression Potential] Regression is unlikely. Likely failure case is just that the problem is not correctly fixed, and the user ends up with either an NTFS formated disk that is mounted at /mnt or there is nothing mounted at /mnt. === End SRU Template === After resizing a 16.04 VM on Azure, the VM is presented with a new ephemeral drive (of a different size), which initially is NTFS formatted. Cloud-init tries to format the appropriate partition ext4, but fails because it is mounted. Cloud-init has unmount logic for exactly this case in the get_data call on the Azure data source, but this is never called because fresh cache is found. Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: start: init-network/check-cache: attempting to read from cache [trust] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False) Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Read 5950 bytes from /var/lib/cloud/instance/obj.pkl Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] stages.py[DEBUG]: restored from cache: DataSourceAzureNet [seed=/dev/sr0] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/check-cache: SUCCESS: restored from cache: DataSourceAzureNet [seed=/dev/sr0] ... Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating file system None on /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/mkfs.ext4', '/dev/sdb1'] with allowed return codes [0] (shell=False, capture=True) Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.052 seconds Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[WARNING]: Failed during filesystem operation#012Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']':#012Unexpected error while running command.#012Command: ['/sbin/mkfs.ext4', '/dev/sdb1']#012Exit code: 1#012Reason: -#012Stdout: ''#012Stderr: 'mke2fs 1.42.13 (17-May-2015)\n/dev/sdb1 is mounted; will not make a filesystem here!\n' $ lsb_release -rd Description: Ubuntu 16.04.1 LTS Release: 16.04 $ cat /etc/cloud/build.info build_name: server serial: 20160721 ~$ dpkg -l cloud-init Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=====================-=====================-===================================================================== ii cloud-init 0.7.7~bzr1256-0ubuntu all Init scripts for cloud instances We're seeing ~100% repro of this bug on resize, where the only success cases are caused by another bug that messes up fstab and prevents mounting of the drive.
2016-09-20 19:56:32 Launchpad Janitor cloud-init (Ubuntu): status Confirmed Fix Released
2016-10-03 19:25:22 Scott Moser cloud-init: status Confirmed Fix Committed
2016-10-03 19:27:45 Scott Moser description === Begin SRU Template === [Impact] In some cases, cloud-init writes entries to /etc/fstab, and on azure it will even format a disk for mounting and then write the entry for that 'ephemeral' disk there. A supported operation on Azure is to "resize" the system. When you do this the system is shut down, resized (given larger/faster disks and more CPU) and then brought back up. In that process, the "ephemeral" disk re-initialized to its original NTFS format. The designed goal is for cloud-init to recognize this situation and re-format the disk to ext4. The problem is that the mount of that disk happens before cloud-init can reformat. Thats because the entry in fstab has 'auto' and is automatically mounted. The end result is that after resize operation the user will be left with the ephemeral disk mounted at /mnt and having a ntfs filesystem rather than ext4. [Test Case] The text in comment 3 describes how to recreate by the original reporter. Another way to do this is to just re-format the ephemeral disk as ntfs and then reboot. The result *should* be that after reboot it comes back up and has an ext4 filesystem on it. 1.) boot system on azure (for this, i use https://gist.github.com/smoser/5806147, but you can use web ui or any other way). 2.) unmount the ephemeral disk $ umount /mnt 3.) repartition it so that mkfs.ntfs does less and is faster This is not strictly necessary, but mkfs.ntfs can take upwards of 20 minutes. shrinking /dev/sdb2 to be 200M means it will finish in < 1 minute. $ disk=/dev/disk/cloud/azure_resource $ part=/dev/disk/cloud/azure_resource-part1 $ echo "2048,$((2*1024*100)),7" | sudo sfdisk "$disk" $ time mkfs.ntfs --quick "$part" 4.) reboot 5.) expect that /proc/mounts has /dev/disk/cloud/azure_resource-part1 as ext4 and that fstab has x-systemd.requires in it. $ awk '$2 == "/mnt" { print $0 }' /proc/mounts /dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0 $ awk '$2 == "/mnt" { print $0 }' /etc/fstab /dev/sdb1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2 [Regression Potential] Regression is unlikely. Likely failure case is just that the problem is not correctly fixed, and the user ends up with either an NTFS formated disk that is mounted at /mnt or there is nothing mounted at /mnt. === End SRU Template === After resizing a 16.04 VM on Azure, the VM is presented with a new ephemeral drive (of a different size), which initially is NTFS formatted. Cloud-init tries to format the appropriate partition ext4, but fails because it is mounted. Cloud-init has unmount logic for exactly this case in the get_data call on the Azure data source, but this is never called because fresh cache is found. Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: start: init-network/check-cache: attempting to read from cache [trust] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False) Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Read 5950 bytes from /var/lib/cloud/instance/obj.pkl Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] stages.py[DEBUG]: restored from cache: DataSourceAzureNet [seed=/dev/sr0] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/check-cache: SUCCESS: restored from cache: DataSourceAzureNet [seed=/dev/sr0] ... Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating file system None on /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/mkfs.ext4', '/dev/sdb1'] with allowed return codes [0] (shell=False, capture=True) Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.052 seconds Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[WARNING]: Failed during filesystem operation#012Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']':#012Unexpected error while running command.#012Command: ['/sbin/mkfs.ext4', '/dev/sdb1']#012Exit code: 1#012Reason: -#012Stdout: ''#012Stderr: 'mke2fs 1.42.13 (17-May-2015)\n/dev/sdb1 is mounted; will not make a filesystem here!\n' $ lsb_release -rd Description: Ubuntu 16.04.1 LTS Release: 16.04 $ cat /etc/cloud/build.info build_name: server serial: 20160721 ~$ dpkg -l cloud-init Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=====================-=====================-===================================================================== ii cloud-init 0.7.7~bzr1256-0ubuntu all Init scripts for cloud instances We're seeing ~100% repro of this bug on resize, where the only success cases are caused by another bug that messes up fstab and prevents mounting of the drive. === Begin SRU Template === [Impact] In some cases, cloud-init writes entries to /etc/fstab, and on azure it will even format a disk for mounting and then write the entry for that 'ephemeral' disk there. A supported operation on Azure is to "resize" the system. When you do this the system is shut down, resized (given larger/faster disks and more CPU) and then brought back up. In that process, the "ephemeral" disk re-initialized to its original NTFS format. The designed goal is for cloud-init to recognize this situation and re-format the disk to ext4. The problem is that the mount of that disk happens before cloud-init can reformat. Thats because the entry in fstab has 'auto' and is automatically mounted. The end result is that after resize operation the user will be left with the ephemeral disk mounted at /mnt and having a ntfs filesystem rather than ext4. [Test Case] The text in comment 3 describes how to recreate by the original reporter. Another way to do this is to just re-format the ephemeral disk as ntfs and then reboot. The result *should* be that after reboot it comes back up and has an ext4 filesystem on it. 1.) boot system on azure   (for this, i use https://gist.github.com/smoser/5806147, but you can    use web ui or any other way). 2.) unmount the ephemeral disk    $ umount /mnt 3.) repartition it so that mkfs.ntfs does less and is faster    This is not strictly necessary, but mkfs.ntfs can take upwards of    20 minutes. shrinking /dev/sdb2 to be 200M means it will finish    in < 1 minute.    $ disk=/dev/disk/cloud/azure_resource    $ part=/dev/disk/cloud/azure_resource-part1    $ echo "2048,$((2*1024*100)),7" | sudo sfdisk "$disk"    $ time mkfs.ntfs --quick "$part" 4.) reboot 5.) expect that /proc/mounts has /dev/disk/cloud/azure_resource-part1 as ext4     and that fstab has x-systemd.requires in it.     $ awk '$2 == "/mnt" { print $0 }' /proc/mounts     /dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0     $ awk '$2 == "/mnt" { print $0 }' /etc/fstab     /dev/sdb1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2 [Regression Potential] Regression is unlikely. Likely failure case is just that the problem is not correctly fixed, and the user ends up with either an NTFS formated disk that is mounted at /mnt or there is nothing mounted at /mnt. === End SRU Template === After resizing a 16.04 VM on Azure, the VM is presented with a new ephemeral drive (of a different size), which initially is NTFS formatted. Cloud-init tries to format the appropriate partition ext4, but fails because it is mounted. Cloud-init has unmount logic for exactly this case in the get_data call on the Azure data source, but this is never called because fresh cache is found. Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: start: init-network/check-cache: attempting to read from cache [trust] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False) Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Read 5950 bytes from /var/lib/cloud/instance/obj.pkl Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] stages.py[DEBUG]: restored from cache: DataSourceAzureNet [seed=/dev/sr0] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/check-cache: SUCCESS: restored from cache: DataSourceAzureNet [seed=/dev/sr0] ... Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating file system None on /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/mkfs.ext4', '/dev/sdb1'] with allowed return codes [0] (shell=False, capture=True) Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.052 seconds Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[WARNING]: Failed during filesystem operation#012Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']':#012Unexpected error while running command.#012Command: ['/sbin/mkfs.ext4', '/dev/sdb1']#012Exit code: 1#012Reason: -#012Stdout: ''#012Stderr: 'mke2fs 1.42.13 (17-May-2015)\n/dev/sdb1 is mounted; will not make a filesystem here!\n' $ lsb_release -rd Description: Ubuntu 16.04.1 LTS Release: 16.04 $ cat /etc/cloud/build.info build_name: server serial: 20160721 ~$ dpkg -l cloud-init Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=====================-=====================-===================================================================== ii cloud-init 0.7.7~bzr1256-0ubuntu all Init scripts for cloud instances We're seeing ~100% repro of this bug on resize, where the only success cases are caused by another bug that messes up fstab and prevents mounting of the drive. Related bugs: bug 1629868: cloud-init times out because of no dbus
2016-10-31 16:53:26 Stephen A. Zarkos bug added subscriber Stephen A. Zarkos
2016-11-07 21:19:00 Scott Moser nominated for series Ubuntu Xenial
2016-11-07 21:19:00 Scott Moser bug task added cloud-init (Ubuntu Xenial)
2016-11-07 21:19:22 Scott Moser cloud-init (Ubuntu Xenial): status New Confirmed
2016-11-07 21:19:25 Scott Moser cloud-init (Ubuntu Xenial): importance Undecided Medium
2016-11-09 18:06:44 Scott Moser description === Begin SRU Template === [Impact] In some cases, cloud-init writes entries to /etc/fstab, and on azure it will even format a disk for mounting and then write the entry for that 'ephemeral' disk there. A supported operation on Azure is to "resize" the system. When you do this the system is shut down, resized (given larger/faster disks and more CPU) and then brought back up. In that process, the "ephemeral" disk re-initialized to its original NTFS format. The designed goal is for cloud-init to recognize this situation and re-format the disk to ext4. The problem is that the mount of that disk happens before cloud-init can reformat. Thats because the entry in fstab has 'auto' and is automatically mounted. The end result is that after resize operation the user will be left with the ephemeral disk mounted at /mnt and having a ntfs filesystem rather than ext4. [Test Case] The text in comment 3 describes how to recreate by the original reporter. Another way to do this is to just re-format the ephemeral disk as ntfs and then reboot. The result *should* be that after reboot it comes back up and has an ext4 filesystem on it. 1.) boot system on azure   (for this, i use https://gist.github.com/smoser/5806147, but you can    use web ui or any other way). 2.) unmount the ephemeral disk    $ umount /mnt 3.) repartition it so that mkfs.ntfs does less and is faster    This is not strictly necessary, but mkfs.ntfs can take upwards of    20 minutes. shrinking /dev/sdb2 to be 200M means it will finish    in < 1 minute.    $ disk=/dev/disk/cloud/azure_resource    $ part=/dev/disk/cloud/azure_resource-part1    $ echo "2048,$((2*1024*100)),7" | sudo sfdisk "$disk"    $ time mkfs.ntfs --quick "$part" 4.) reboot 5.) expect that /proc/mounts has /dev/disk/cloud/azure_resource-part1 as ext4     and that fstab has x-systemd.requires in it.     $ awk '$2 == "/mnt" { print $0 }' /proc/mounts     /dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0     $ awk '$2 == "/mnt" { print $0 }' /etc/fstab     /dev/sdb1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2 [Regression Potential] Regression is unlikely. Likely failure case is just that the problem is not correctly fixed, and the user ends up with either an NTFS formated disk that is mounted at /mnt or there is nothing mounted at /mnt. === End SRU Template === After resizing a 16.04 VM on Azure, the VM is presented with a new ephemeral drive (of a different size), which initially is NTFS formatted. Cloud-init tries to format the appropriate partition ext4, but fails because it is mounted. Cloud-init has unmount logic for exactly this case in the get_data call on the Azure data source, but this is never called because fresh cache is found. Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: start: init-network/check-cache: attempting to read from cache [trust] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False) Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Read 5950 bytes from /var/lib/cloud/instance/obj.pkl Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] stages.py[DEBUG]: restored from cache: DataSourceAzureNet [seed=/dev/sr0] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/check-cache: SUCCESS: restored from cache: DataSourceAzureNet [seed=/dev/sr0] ... Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating file system None on /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/mkfs.ext4', '/dev/sdb1'] with allowed return codes [0] (shell=False, capture=True) Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.052 seconds Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[WARNING]: Failed during filesystem operation#012Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']':#012Unexpected error while running command.#012Command: ['/sbin/mkfs.ext4', '/dev/sdb1']#012Exit code: 1#012Reason: -#012Stdout: ''#012Stderr: 'mke2fs 1.42.13 (17-May-2015)\n/dev/sdb1 is mounted; will not make a filesystem here!\n' $ lsb_release -rd Description: Ubuntu 16.04.1 LTS Release: 16.04 $ cat /etc/cloud/build.info build_name: server serial: 20160721 ~$ dpkg -l cloud-init Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=====================-=====================-===================================================================== ii cloud-init 0.7.7~bzr1256-0ubuntu all Init scripts for cloud instances We're seeing ~100% repro of this bug on resize, where the only success cases are caused by another bug that messes up fstab and prevents mounting of the drive. Related bugs: bug 1629868: cloud-init times out because of no dbus === Begin SRU Template === [Impact] In some cases, cloud-init writes entries to /etc/fstab, and on azure it will even format a disk for mounting and then write the entry for that 'ephemeral' disk there. A supported operation on Azure is to "resize" the system. When you do this the system is shut down, resized (given larger/faster disks and more CPU) and then brought back up. In that process, the "ephemeral" disk re-initialized to its original NTFS format. The designed goal is for cloud-init to recognize this situation and re-format the disk to ext4. The problem is that the mount of that disk happens before cloud-init can reformat. Thats because the entry in fstab has 'auto' and is automatically mounted. The end result is that after resize operation the user will be left with the ephemeral disk mounted at /mnt and having a ntfs filesystem rather than ext4. [Test Case] The text in comment 3 describes how to recreate by the original reporter. Another way to do this is to just re-format the ephemeral disk as ntfs and then reboot. The result *should* be that after reboot it comes back up and has an ext4 filesystem on it. 1.) boot system on azure   (for this, i use https://gist.github.com/smoser/5806147, but you can    use web ui or any other way). Save output of journalctl --no-pager > journalctl.orig systemctl status --no-pager > systemctl-status.orig systemctl --no-pager > systemctl.orig 2.) unmount the ephemeral disk    $ umount /mnt 3.) repartition it so that mkfs.ntfs does less and is faster    This is not strictly necessary, but mkfs.ntfs can take upwards of    20 minutes. shrinking /dev/sdb2 to be 200M means it will finish    in < 1 minute.    $ disk=/dev/disk/cloud/azure_resource    $ part=/dev/disk/cloud/azure_resource-part1    $ echo "2048,$((2*1024*100)),7" | sudo sfdisk "$disk"    $ time mkfs.ntfs --quick "$part" 4.) reboot 5.) expect that /proc/mounts has /dev/disk/cloud/azure_resource-part1 as ext4     and that fstab has x-systemd.requires in it.     $ awk '$2 == "/mnt" { print $0 }' /proc/mounts     /dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0     $ awk '$2 == "/mnt" { print $0 }' /etc/fstab     /dev/sdb1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2 6.) collect journal and systemctl information as described in step 1 above. Compare output, specifically looking for case insensitve "breaks" [Regression Potential] Regression is unlikely. Likely failure case is just that the problem is not correctly fixed, and the user ends up with either an NTFS formated disk that is mounted at /mnt or there is nothing mounted at /mnt. === End SRU Template === After resizing a 16.04 VM on Azure, the VM is presented with a new ephemeral drive (of a different size), which initially is NTFS formatted. Cloud-init tries to format the appropriate partition ext4, but fails because it is mounted. Cloud-init has unmount logic for exactly this case in the get_data call on the Azure data source, but this is never called because fresh cache is found. Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: start: init-network/check-cache: attempting to read from cache [trust] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False) Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Read 5950 bytes from /var/lib/cloud/instance/obj.pkl Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] stages.py[DEBUG]: restored from cache: DataSourceAzureNet [seed=/dev/sr0] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/check-cache: SUCCESS: restored from cache: DataSourceAzureNet [seed=/dev/sr0] ... Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating file system None on /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/mkfs.ext4', '/dev/sdb1'] with allowed return codes [0] (shell=False, capture=True) Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.052 seconds Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[WARNING]: Failed during filesystem operation#012Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']':#012Unexpected error while running command.#012Command: ['/sbin/mkfs.ext4', '/dev/sdb1']#012Exit code: 1#012Reason: -#012Stdout: ''#012Stderr: 'mke2fs 1.42.13 (17-May-2015)\n/dev/sdb1 is mounted; will not make a filesystem here!\n' $ lsb_release -rd Description: Ubuntu 16.04.1 LTS Release: 16.04 $ cat /etc/cloud/build.info build_name: server serial: 20160721 ~$ dpkg -l cloud-init Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=====================-=====================-===================================================================== ii cloud-init 0.7.7~bzr1256-0ubuntu all Init scripts for cloud instances We're seeing ~100% repro of this bug on resize, where the only success cases are caused by another bug that messes up fstab and prevents mounting of the drive. Related bugs:  bug 1629868: cloud-init times out because of no dbus
2016-11-15 23:30:19 Steve Langasek cloud-init (Ubuntu Xenial): status Confirmed Fix Committed
2016-11-15 23:30:22 Steve Langasek bug added subscriber Ubuntu Stable Release Updates Team
2016-11-15 23:30:24 Steve Langasek bug added subscriber SRU Verification
2016-11-15 23:30:31 Steve Langasek tags verification-needed
2016-11-16 20:23:16 Paul Meyer tags verification-needed verification-failed
2016-11-16 21:22:53 Paul Meyer attachment added log repro with 0.7.8-47-gb6561a1-0u https://bugs.launchpad.net/cloud-init/+bug/1611074/+attachment/4778323/+files/cloud-init.log
2016-11-16 22:25:12 Matt Bearup attachment added logs.tar.gz https://bugs.launchpad.net/cloud-init/+bug/1611074/+attachment/4778347/+files/logs.tar.gz
2016-11-16 22:37:53 Paul Meyer attachment added cloud-init.log https://bugs.launchpad.net/cloud-init/+bug/1611074/+attachment/4778349/+files/cloud-init.log
2016-11-17 19:53:01 Launchpad Janitor merge proposal linked https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/311205
2016-11-17 20:51:48 Scott Moser cloud-init (Ubuntu): status Fix Released Confirmed
2016-11-17 20:51:54 Scott Moser cloud-init (Ubuntu Xenial): status Fix Committed Confirmed
2016-11-18 22:37:14 Launchpad Janitor cloud-init (Ubuntu): status Confirmed Fix Released
2016-11-19 01:55:57 Steve Langasek cloud-init (Ubuntu Xenial): status Confirmed Fix Committed
2016-11-19 01:56:05 Steve Langasek tags verification-failed
2016-11-19 01:56:06 Steve Langasek tags verification-needed
2016-11-21 15:31:05 Scott Moser tags verification-needed verification-done
2016-11-22 17:58:08 Scott Moser description === Begin SRU Template === [Impact] In some cases, cloud-init writes entries to /etc/fstab, and on azure it will even format a disk for mounting and then write the entry for that 'ephemeral' disk there. A supported operation on Azure is to "resize" the system. When you do this the system is shut down, resized (given larger/faster disks and more CPU) and then brought back up. In that process, the "ephemeral" disk re-initialized to its original NTFS format. The designed goal is for cloud-init to recognize this situation and re-format the disk to ext4. The problem is that the mount of that disk happens before cloud-init can reformat. Thats because the entry in fstab has 'auto' and is automatically mounted. The end result is that after resize operation the user will be left with the ephemeral disk mounted at /mnt and having a ntfs filesystem rather than ext4. [Test Case] The text in comment 3 describes how to recreate by the original reporter. Another way to do this is to just re-format the ephemeral disk as ntfs and then reboot. The result *should* be that after reboot it comes back up and has an ext4 filesystem on it. 1.) boot system on azure   (for this, i use https://gist.github.com/smoser/5806147, but you can    use web ui or any other way). Save output of journalctl --no-pager > journalctl.orig systemctl status --no-pager > systemctl-status.orig systemctl --no-pager > systemctl.orig 2.) unmount the ephemeral disk    $ umount /mnt 3.) repartition it so that mkfs.ntfs does less and is faster    This is not strictly necessary, but mkfs.ntfs can take upwards of    20 minutes. shrinking /dev/sdb2 to be 200M means it will finish    in < 1 minute.    $ disk=/dev/disk/cloud/azure_resource    $ part=/dev/disk/cloud/azure_resource-part1    $ echo "2048,$((2*1024*100)),7" | sudo sfdisk "$disk"    $ time mkfs.ntfs --quick "$part" 4.) reboot 5.) expect that /proc/mounts has /dev/disk/cloud/azure_resource-part1 as ext4     and that fstab has x-systemd.requires in it.     $ awk '$2 == "/mnt" { print $0 }' /proc/mounts     /dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0     $ awk '$2 == "/mnt" { print $0 }' /etc/fstab     /dev/sdb1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2 6.) collect journal and systemctl information as described in step 1 above. Compare output, specifically looking for case insensitve "breaks" [Regression Potential] Regression is unlikely. Likely failure case is just that the problem is not correctly fixed, and the user ends up with either an NTFS formated disk that is mounted at /mnt or there is nothing mounted at /mnt. === End SRU Template === After resizing a 16.04 VM on Azure, the VM is presented with a new ephemeral drive (of a different size), which initially is NTFS formatted. Cloud-init tries to format the appropriate partition ext4, but fails because it is mounted. Cloud-init has unmount logic for exactly this case in the get_data call on the Azure data source, but this is never called because fresh cache is found. Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: start: init-network/check-cache: attempting to read from cache [trust] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False) Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Read 5950 bytes from /var/lib/cloud/instance/obj.pkl Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] stages.py[DEBUG]: restored from cache: DataSourceAzureNet [seed=/dev/sr0] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/check-cache: SUCCESS: restored from cache: DataSourceAzureNet [seed=/dev/sr0] ... Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating file system None on /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/mkfs.ext4', '/dev/sdb1'] with allowed return codes [0] (shell=False, capture=True) Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.052 seconds Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[WARNING]: Failed during filesystem operation#012Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']':#012Unexpected error while running command.#012Command: ['/sbin/mkfs.ext4', '/dev/sdb1']#012Exit code: 1#012Reason: -#012Stdout: ''#012Stderr: 'mke2fs 1.42.13 (17-May-2015)\n/dev/sdb1 is mounted; will not make a filesystem here!\n' $ lsb_release -rd Description: Ubuntu 16.04.1 LTS Release: 16.04 $ cat /etc/cloud/build.info build_name: server serial: 20160721 ~$ dpkg -l cloud-init Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=====================-=====================-===================================================================== ii cloud-init 0.7.7~bzr1256-0ubuntu all Init scripts for cloud instances We're seeing ~100% repro of this bug on resize, where the only success cases are caused by another bug that messes up fstab and prevents mounting of the drive. Related bugs:  bug 1629868: cloud-init times out because of no dbus === Begin SRU Template === [Impact] In some cases, cloud-init writes entries to /etc/fstab, and on azure it will even format a disk for mounting and then write the entry for that 'ephemeral' disk there. A supported operation on Azure is to "resize" the system. When you do this the system is shut down, resized (given larger/faster disks and more CPU) and then brought back up. In that process, the "ephemeral" disk re-initialized to its original NTFS format. The designed goal is for cloud-init to recognize this situation and re-format the disk to ext4. The problem is that the mount of that disk happens before cloud-init can reformat. Thats because the entry in fstab has 'auto' and is automatically mounted. The end result is that after resize operation the user will be left with the ephemeral disk mounted at /mnt and having a ntfs filesystem rather than ext4. [Test Case] The text in comment 3 describes how to recreate by the original reporter. Another way to do this is to just re-format the ephemeral disk as ntfs and then reboot. The result *should* be that after reboot it comes back up and has an ext4 filesystem on it. 1.) boot system on azure   (for this, i use https://gist.github.com/smoser/5806147, but you can    use web ui or any other way).    Save output of      journalctl --no-pager > journalctl.orig      systemctl status --no-pager > systemctl-status.orig      systemctl --no-pager > systemctl.orig 2.) unmount the ephemeral disk    $ umount /mnt 3.) repartition it so that mkfs.ntfs does less and is faster    This is not strictly necessary, but mkfs.ntfs can take upwards of    20 minutes. shrinking /dev/sdb2 to be 200M means it will finish    in < 1 minute.    $ disk=/dev/disk/cloud/azure_resource    $ part=/dev/disk/cloud/azure_resource-part1    $ echo "2048,$((2*1024*100)),7" | sudo sfdisk "$disk"    $ time mkfs.ntfs --quick "$part" 4.) reboot 5.) expect that /proc/mounts has /dev/disk/cloud/azure_resource-part1 as ext4     and that fstab has x-systemd.requires in it.     $ awk '$2 == "/mnt" { print $0 }' /proc/mounts     /dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0     $ awk '$2 == "/mnt" { print $0 }' /etc/fstab     /dev/sdb1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2 6.) collect journal and systemctl information as described in step 1 above.     Compare output, specifically looking for case insensitve "breaks" [Regression Potential] Regression is unlikely. Likely failure case is just that the problem is not correctly fixed, and the user ends up with either an NTFS formated disk that is mounted at /mnt or there is nothing mounted at /mnt. === End SRU Template === After resizing a 16.04 VM on Azure, the VM is presented with a new ephemeral drive (of a different size), which initially is NTFS formatted. Cloud-init tries to format the appropriate partition ext4, but fails because it is mounted. Cloud-init has unmount logic for exactly this case in the get_data call on the Azure data source, but this is never called because fresh cache is found. Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: start: init-network/check-cache: attempting to read from cache [trust] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False) Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Read 5950 bytes from /var/lib/cloud/instance/obj.pkl Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] stages.py[DEBUG]: restored from cache: DataSourceAzureNet [seed=/dev/sr0] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/check-cache: SUCCESS: restored from cache: DataSourceAzureNet [seed=/dev/sr0] ... Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating file system None on /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/mkfs.ext4', '/dev/sdb1'] with allowed return codes [0] (shell=False, capture=True) Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.052 seconds Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[WARNING]: Failed during filesystem operation#012Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']':#012Unexpected error while running command.#012Command: ['/sbin/mkfs.ext4', '/dev/sdb1']#012Exit code: 1#012Reason: -#012Stdout: ''#012Stderr: 'mke2fs 1.42.13 (17-May-2015)\n/dev/sdb1 is mounted; will not make a filesystem here!\n' $ lsb_release -rd Description: Ubuntu 16.04.1 LTS Release: 16.04 $ cat /etc/cloud/build.info build_name: server serial: 20160721 ~$ dpkg -l cloud-init Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=====================-=====================-===================================================================== ii cloud-init 0.7.7~bzr1256-0ubuntu all Init scripts for cloud instances We're seeing ~100% repro of this bug on resize, where the only success cases are caused by another bug that messes up fstab and prevents mounting of the drive. Related bugs:  bug 1629868: cloud-init times out because of no dbus bug 1603222: Azure: incorrect entry in fstab for ephemeral disk
2016-11-24 20:00:36 Launchpad Janitor cloud-init (Ubuntu Xenial): status Fix Committed Fix Released
2016-11-24 20:00:55 Adam Conrad removed subscriber Ubuntu Stable Release Updates Team
2016-11-29 23:41:06 Chris Halse Rogers cloud-init (Ubuntu Yakkety): status New Fix Committed
2016-11-29 23:41:08 Chris Halse Rogers bug added subscriber Ubuntu Stable Release Updates Team
2016-11-29 23:41:20 Chris Halse Rogers tags verification-done
2016-11-29 23:41:22 Chris Halse Rogers tags verification-needed
2016-12-08 02:38:54 Mathew Hodson cloud-init (Ubuntu Yakkety): importance Undecided Medium
2016-12-19 18:06:10 Scott Moser tags verification-needed verification-done
2016-12-21 01:05:10 Launchpad Janitor cloud-init (Ubuntu Yakkety): status Fix Committed Fix Released
2016-12-23 17:42:04 Scott Moser cloud-init: status Fix Committed Fix Released
2023-05-10 15:25:22 James Falcon bug watch added https://github.com/canonical/cloud-init/issues/2705