fs_setup always creates new filesystem with partition 'auto'

[Lee Faris]

=== Begin SRU Template ===
[Impact]
On instance first boot cloud-init may create a filesystem when
it should have re-used an existing filesystem.

[Test Case]
The test case launches an instance, assuming that has an old version
of cloud-init inside. The user-data will not be valid for the default
configuration of disks. (Most openstack instances would have a
single 'ephemeral' disk in addition to root, with an ext4 filesystem
labelled 'ephemeral0'). We will then upgrade instance to proposed.
And create a filesystem on /dev/vdb1 that *should* match.

1. launch an instance in openstack with the following user-data.
   |#cloud-config
   |fs_setup:
   | - label: mydata
   | device: /dev/vdb
   | filesystem: ext4
   | partition: auto
   |mounts:
   | - ["/dev/vdb1", "/mnt"]

   $ cat > my-userdata.txt <<EOF
   #cloud-config
   fs_setup:
     - label: mydata
       device: /dev/vdb
       filesystem: ext4
       partition: auto
   mounts:
     - ["/dev/vdb1", "/mnt"]
   EOF

   $ openstack server create --user-data=my-userdata.txt \
      --key-name=brickies --flavor=m1.small .... my-test

2. ssh in, prepare the /dev/vdb to have a partition, and upgrade

  # run attached 'disk-setup'. This will partition the disk
  # and wipe any filesystem data off, basically making it a partitioned
  # but otherwise empty disk.
  $ sudo ./disk-setup
  umount: /mnt: not mounted
  wiping /dev/vdb
  partitioning /dev/vdb
  /dev/vdb: PTUUID="9920db9b-a1ff-4f44-834c-8fcc42bd5821" PTTYPE="gpt"
  /dev/vdb1: PARTUUID="1accde8d-8880-4a93-913a-61eee2e92535"

3. enable proposed, upgrade
4. clean out state and reboot
   sudo rm -Rf /var/lib/cloud /var/log/cloud-init*
   sudo sed -i '/comment=cloudconfig/d' /etc/fstab
   sudo reboot

5. ssh back in and look around.

   # cloud-init should have created a filesystem on /dev/vdb1
   # and mounted it at /mnt.

   $ grep /mnt /proc/mounts
   /dev/vdb1 /mnt ext4 rw,relatime,data=ordered 0 0

   # and have a filesystem 'mydata'
   $ sudo blkid /dev/vdb1
   /dev/vdb1: LABEL="mydata" UUID="79090091-800e-4348-a7a9-8c7a26ed18f7" TYPE="ext4" PARTUUID="b8ef8d3f-acb4-4bd0-ba2d-fdfb45c2e8f0"

   # put a file on there, then clean up and reboot.
   # we will expect that that this time, cloud-init will just re-use
   # the existing filesystem rather than making another.
   $ echo hi mom | sudo tee -a /mnt/my-important-data.txt

6. ssh in and expect /mnt/my-important-data.txt

   $ cat /mnt/my-important-data.txt
   hi mom

[Regression Potential]
Potentially this could re-use a partition that the user wanted reformatted.

[Other Info]
Upstream commit:
 https://git.launchpad.net/cloud-init/commit/?id=4a2b2f87ec48c2

=== End SRU Template ===

# cloud-init -v
cloud-init 0.7.5

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty

AMI: ami-1721ff77 - Ubuntu 14.04 20160314

fs_setup fails to detect an existing filesystem and creates a new one when using the following configuration:

fs_setup:
    label: None
    filesystem: ext4
    device: /dev/xvdf
    partition: auto

There error seems to be here - https://git.launchpad.net/cloud-init/tree/cloudinit/config/cc_disk_setup.py#n216
This line sets definition['partition'] to None.

I believe " definition['partition'] = part" should be a part of the above conditional.

[Jonathan Ballet (multani)]

I'm facing the same problem here.

I'm trying to get this configuration into cloud-init:

    disk_setup:
      /dev/xvdb:
        overwrite: false
        layout: true
    fs_setup:
      - device: /dev/xvdb1
        filesystem: ext4
        label: test
        overwrite: false
        partition: auto

For the context, it's applied on a EBS disk on Amazon, and my hope was to be able to keep the content of the partition accross different machines.

I've tested @leelynnef 's patch and it fixes the problem for me.

[Brian Murray (brian-murray)]

Hello Lee, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-90-g61eb03fe-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Lee, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-90-g61eb03fe-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

$ dpkg-query --show cloud-init
cloud-init 0.7.9-90-g61eb03fe-0ubuntu1~16.10.1

$ grep /mnt /proc/mounts
/dev/vdb1 /mnt ext4 rw,relatime,data=ordered 0 0

$ sudo blkid /dev/vdb1
/dev/vdb1: LABEL="mydata" UUID="20a0d834-2aca-450c-8d4d-a9e0a3fa74da" TYPE="ext4" PARTUUID="4d3c51ff-449e-4497-9ddc-54143b531095"

$ echo $(date -R) hi mom | sudo tee -a /mnt/my-important-data.txt
Mon, 17 Apr 2017 14:21:07 +0000 hi mom

$ sudo rm -Rf /var/lib/cloud /var/log/cloud-init*
$ sudo reboot

...
# ssh back in

$ grep cc_disk_setup.py /var/log/cloud-init.log
2017-04-17 14:22:05,078 - stages.py[DEBUG]: Running module disk_setup (<module 'cloudinit.config.cc_disk_setup' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py'>) with frequency once-per-instance
2017-04-17 14:22:05,080 - cc_disk_setup.py[DEBUG]: setting up filesystems: [{'label': 'mydata', 'partition': 'auto', 'device': '/dev/vdb', 'filesystem': 'ext4'}]
2017-04-17 14:22:05,166 - cc_disk_setup.py[DEBUG]: Creating new filesystem.
2017-04-17 14:22:05,166 - cc_disk_setup.py[DEBUG]: Checking /dev/vdb against default devices
2017-04-17 14:22:05,166 - cc_disk_setup.py[DEBUG]: Identifying device to create mydata filesytem on
2017-04-17 14:22:05,176 - cc_disk_setup.py[DEBUG]: Automatic device for /dev/vdb identified as /dev/vdb1
2017-04-17 14:22:05,176 - cc_disk_setup.py[DEBUG]: Found filesystem match, skipping formating.

$ cat /mnt/my-important-data.txt
Mon, 17 Apr 2017 14:21:07 +0000 hi mom

[Scott Moser (smoser)]

## setup
$ git clone git://git.launchpad.net/~smoser/cloud-init/+git/sru-info
$ ./sru-info/bin/enable-proposed
$ ( sudo apt-get -q update && sudo apt-get install -qy cloud-init ) </dev/null
$ dpkg-query --show cloud-init
$ sudo ./sru-info/bugs/lp-1634678/disk-setup
$ sudo ./sru-info/bin/do-reboot clean save=orig

## ssh back in
$ grep cc_disk_setup.py /var/log/cloud-init.log
2017-04-17 14:43:48,488 - stages.py[DEBUG]: Running module disk_setup (<module 'cloudinit.config.cc_disk_setup' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py'>) with frequency once-per-instance
2017-04-17 14:43:48,491 - cc_disk_setup.py[DEBUG]: setting up filesystems: [{'partition': 'auto', 'label': 'mydata', 'device': '/dev/vdb', 'filesystem': 'ext4'}]
2017-04-17 14:43:48,521 - cc_disk_setup.py[DEBUG]: Creating new filesystem.
2017-04-17 14:43:48,522 - cc_disk_setup.py[DEBUG]: Checking /dev/vdb against default devices
2017-04-17 14:43:48,522 - cc_disk_setup.py[DEBUG]: Identifying device to create mydata filesytem on
2017-04-17 14:43:48,536 - cc_disk_setup.py[DEBUG]: Automatic device for /dev/vdb identified as /dev/vdb1
2017-04-17 14:43:48,536 - cc_disk_setup.py[DEBUG]: File system type 'ext4' with label 'mydata' will be created on /dev/vdb1
2017-04-17 14:43:48,542 - cc_disk_setup.py[DEBUG]: Creating file system mydata on /dev/vdb1
2017-04-17 14:43:48,542 - cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/vdb1 -L mydata

$ grep /mnt /proc/mounts
/dev/vdb1 /mnt ext4 rw,relatime,data=ordered 0 0
$ sudo blkid /dev/vdb1
/dev/vdb1: LABEL="mydata" UUID="3f2d4ff8-2863-4842-b4ec-e54fc3de9a2f" TYPE="ext4" PARTUUID="9ebe6f52-30c4-4523-b771-e6857edbbd65"
$ echo $(date -R) hi mom | sudo tee -a /mnt/my-important-data.txt
Mon, 17 Apr 2017 14:45:27 +0000 hi mom
$ sudo ./sru-info/bin/do-reboot clean save=orig2

## one more time.
## we did not format this time.
$ grep cc_disk_setup.py /var/log/cloud-init.log
2017-04-17 14:46:22,927 - stages.py[DEBUG]: Running module disk_setup (<module 'cloudinit.config.cc_disk_setup' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py'>) with frequency once-per-instance
2017-04-17 14:46:22,929 - cc_disk_setup.py[DEBUG]: setting up filesystems: [{'filesystem': 'ext4', 'partition': 'auto', 'label': 'mydata', 'device': '/dev/vdb'}]
2017-04-17 14:46:22,948 - cc_disk_setup.py[DEBUG]: Creating new filesystem.
2017-04-17 14:46:22,949 - cc_disk_setup.py[DEBUG]: Checking /dev/vdb against default devices
2017-04-17 14:46:22,949 - cc_disk_setup.py[DEBUG]: Identifying device to create mydata filesytem on
2017-04-17 14:46:22,964 - cc_disk_setup.py[DEBUG]: Automatic device for /dev/vdb identified as /dev/vdb1
2017-04-17 14:46:22,964 - cc_disk_setup.py[DEBUG]: Found filesystem match, skipping formating.

$ cat /mnt/my-important-data.txt
Mon, 17 Apr 2017 14:45:27 +0000 hi mom

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-90-g61eb03fe-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - OpenStack: add 'dvs' to the list of physical link types. (LP: #1674946)
    - Fix bug that resulted in an attempt to rename bonds or vlans.
      (LP: #1669860)
    - tests: update OpenNebula and Digital Ocean to not rely on host
      interfaces.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without
      initramfs. [Steve Langasek] (LP: #1677376)
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)
    - Fix filesystem creation when using "partition: auto"
      [Jonathan Ballet] (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive.
      (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3. [root]
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)
    - cloudinit.net: add network config v2 parsing and rendering [Ryan Harper]
    - net: Fix incorrect call to isfile [Joshua Powers] (LP: #1674317)
    - net: add renderers for automatically selecting the renderer.
    - doc: fix config drive doc with regard to unpartitioned disks.
      (LP: #1673818)
    - test: Adding integratiron test for password as list [Joshua Powers]
    - render_network_state: switch arguments around, do not require target
    - support 'loopback' as a device type.
    - Integration Testing: improve testcase subclassing [Wesley Wiedenmeier]
    - gitignore: adding doc/rtd_html [Joshua Powers]
    - doc: add instructions for running integration tests via tox.
      [Joshua Powers]
    - test: avoid differences in 'date' output due to daylight savings.
    - Fix chef config module in omnibus install. [Jeremy Melvin] (LP: #1583837)
    - Add feature flags to cloudinit.version. [Wesley Wiedenmeier]
    - tox: add a citest environment
    - Support chpasswd/list being a list in addition to a string.
      [Sergio Lystopad] (LP: #1665694)
    - doc: Fix configuration example for cc_set_passwords module.
      [Sergio Lystopad] (LP: #1665773)
    - ...

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - OpenStack: add 'dvs' to the list of physical link types. (LP: #1674946)
    - Fix bug that resulted in an attempt to rename bonds or vlans.
      (LP: #1669860)
    - tests: update OpenNebula and Digital Ocean to not rely on host
      interfaces.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without
      initramfs. [Steve Langasek] (LP: #1677376)
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)
    - Fix filesystem creation when using "partition: auto"
      [Jonathan Ballet] (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive.
      (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3. [root]
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)
    - cloudinit.net: add network config v2 parsing and rendering [Ryan Harper]
    - net: Fix incorrect call to isfile [Joshua Powers] (LP: #1674317)
    - net: add renderers for automatically selecting the renderer.
    - doc: fix config drive doc with regard to unpartitioned disks.
      (LP: #1673818)
    - test: Adding integratiron test for password as list [Joshua Powers]
    - render_network_state: switch arguments around, do not require target
    - support 'loopback' as a device type.
    - Integration Testing: improve testcase subclassing [Wesley Wiedenmeier]
    - gitignore: adding doc/rtd_html [Joshua Powers]
    - doc: add instructions for running integration tests via tox.
      [Joshua Powers]
    - test: avoid differences in 'date' output due to daylight savings.
    - Fix chef config module in omnibus install. [Jeremy Melvin] (LP: #1583837)
    - Add feature flags to cloudinit.version. [Wesley Wiedenmeier]
    - tox: add a citest environment
    - Support chpasswd/list being a list in addition to a string.
      [Sergio Lystopad] (LP: #1665694)
    - doc: Fix configuration example for cc_set_passwords module.
      [Sergio Lystopad] (LP: #1665773...

[Scott Moser (smoser)]

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2741