growing partitions does not work when booted without initramfs

[Scott Moser]

[SRU Justification]
When booted without an initramfs, the root device will be /dev/root, not a
named device. There is partial support for this when resizing filesystems,
but not for growing partitions, without which it doesn't do much good.

A system boots significantly slower with an initramfs than without, and we care about boot speed; a user should not have to have to boot with an initramfs to take advantage of root partition resizing.

[Test case]
Because there are not yet any official Ubuntu images that boot without initramfs, the test case is somewhat opaque. Here is a test case that should work with public branches, though I will be doing validation in GCE using a private branch.
1. Build livecd-rootfs from lp:~vorlon/livecd-rootfs/minimizing/ and upload it to a ppa.
2. bzr branch lp:launchpad-buildd
3. run the following setup commands:
export BUILDID=cloud-init-test
export CHROOT_ROOT=$HOME/build-$BUILDID/chroot-autobuild
wget http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-root.tar.xz -O /tmp/root.tar.xz
mkdir -p $CHROOT_ROOT/build
tar -C $CHROOT_ROOT -x -f /tmp/root.tar.xz
rm $CHROOT_ROOT/etc/resolv.conf
launchpad-buildd/mount-chroot $BUILDID
launchpad-buildd/update-debian-chroot $BUILDID
4. Run a build without -proposed enabled:
launchpad-buildd/buildlivefs --arch amd64 --project ubuntu-cpc --subproject minimize --series <release> --build-id $BUILDID --image-target=none
5. Boot the resulting image ($HOME/build-$BUILDID/chroot-autobuild/build/binary/boot/disk.ext4) in an openstack environment.
6. Verify that the image boots but does not expand the root partition.
7. Run a build again with -proposed enabled:
launchpad-buildd/buildlivefs --arch amd64 --project ubuntu-cpc --subproject minimize --series <release> --build-id $BUILDID --image-target=none --proposed
8. Boot the resulting image in an openstack environment.
9. Verify that the image boots, and that this time the root partition is expanded to use the full disk.

[Regression potential]
Users with existing initramfsless images that are configured with the default of resizing the root partition may be surprised when this starts to work. However, this only applies to first boot of an image, so only newly-mastered images would be affected, so this should be caught by any image CI before the affected users put such an image into production if it has serious impact for them.

Related bugs:
 * bug 1684869: growing root partition does not always work with root=PARTUUID=
 * bug 1685291: RFC: virtio and virtio-scsi should be built in

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-87-gd23543eb-0ubuntu1

---------------
cloud-init (0.7.9-87-gd23543eb-0ubuntu1) zesty; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without initramfs.
      (LP: #1677376) [Steve Langasek]
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)

 -- Scott Moser <email address hidden> Thu, 30 Mar 2017 16:46:43 -0400

[Brian Murray (brian-murray)]

Hello Scott, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-90-g61eb03fe-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-90-g61eb03fe-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Verified for xenial. Image with cloud-init 0.7.9-48-g1c795b9-0ubuntu1~16.04.1:

2017-04-17 22:16:49,814 - cc_growpart.py[DEBUG]: No 'growpart' entry in cfg. Using default: {'mode': 'auto', 'ignore_growroot_disabled': False, 'devices': ['/' ]}
2017-04-17 22:16:49,814 - util.py[DEBUG]: Running command ['growpart', '--help'] with allowed return codes [0] (shell=False, capture=True)
2017-04-17 22:16:49,826 - util.py[DEBUG]: Reading from /proc/1069/mountinfo (quiet=False)
2017-04-17 22:16:49,826 - util.py[DEBUG]: Read 2661 bytes from /proc/1069/mountinfo
2017-04-17 22:16:49,826 - util.py[DEBUG]: resize_devices took 0.001 seconds
2017-04-17 22:16:49,826 - cc_growpart.py[DEBUG]: '/' SKIPPED: stat of '/dev/root' failed: [Errno 2] No such file or directory: '/dev/root'
[...]
2017-04-17 22:16:49,827 - cc_resizefs.py[DEBUG]: resize_info: dev=/dev/root mnt_point=/ path=/
[...]
2017-04-17 22:16:49,832 - cc_resizefs.py[DEBUG]: Converted /dev/root to '/dev/PARTUUID=0dde3c84-01' per kernel cmdline
2017-04-17 22:16:49,832 - cc_resizefs.py[WARNING]: Device '/dev/PARTUUID=0dde3c84-01' did not exist. cannot resize: dev=/dev/root mnt_point=/ path=/

Image with cloud-init 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1:

2017-04-17 22:56:47,456 - cc_growpart.py[DEBUG]: No 'growpart' entry in cfg. Using default: {'mode': 'auto', 'ignore_growroot_disabled': False, 'devices': ['/']}
2017-04-17 22:56:47,456 - util.py[DEBUG]: Running command ['growpart', '--help'] with allowed return codes [0] (shell=False, capture=True)
2017-04-17 22:56:47,493 - util.py[DEBUG]: Reading from /proc/1064/mountinfo (quiet=False)
2017-04-17 22:56:47,493 - util.py[DEBUG]: Read 2661 bytes from /proc/1064/mountinfo
[...]
2017-04-17 22:56:47,498 - util.py[DEBUG]: Running command ['blkid', '-tPARTUUID=94eb3eec-01', '-odevice'] with allowed return codes [0, 2] (shell=False, capture=True)
2017-04-17 22:56:47,522 - util.py[DEBUG]: Reading from /sys/class/block/sda1/partition (quiet=False)
2017-04-17 22:56:47,522 - util.py[DEBUG]: Read 2 bytes from /sys/class/block/sda1/partition
2017-04-17 22:56:47,522 - util.py[DEBUG]: Reading from /sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/dev (quiet=False)
2017-04-17 22:56:47,522 - util.py[DEBUG]: Read 4 bytes from /sys/devices/pci0000:00/0000:00:03.0/virtio0/host0/target0:0:1/0:0:1:0/block/sda/dev
2017-04-17 22:56:47,522 - util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/sda', '1'] with allowed return codes [0] (shell=False, capture=True)
2017-04-17 22:56:47,731 - util.py[DEBUG]: Running command ['growpart', '/dev/sda', '1'] with allowed return codes [0] (shell=False, capture=True)
2017-04-17 22:56:47,770 - util.py[DEBUG]: resize_devices took 0.277 seconds
2017-04-17 22:56:47,770 - cc_growpart.py[INFO]: '/' resized: changed (/dev/sda, 1) from 2359296000 to 10736352768
[...]
2017-04-17 22:56:47,771 - cc_resizefs.py[DEBUG]: resize_info: dev=/dev/root mnt_point=/ path=/
[...]
2017-04-17 22:56:47,780 - util.py[DEBUG]: Running command ['blkid', '-tPARTUUID=94eb3eec-01', '-odevice'] with allowed return codes [0, 2] (shell=False, capture=True)
2017-04-17 22:56:47,794 - cc_resizefs.py[DEBUG]: Converted /dev/root to '/dev/sd...

[Steve Langasek (vorlon)]

For yakkety, I've published a test image w/o the fixed cloud-init to https://people.canonical.com/~vorlon/lp.1677376-yakkety-bad.ext4.xz, if someone would be able to test it.

Unfortunately I don't have a test image with the fixed cloud-init yet, because yakkety-proposed is currently unbuildable due to golang-related uninstallabilities.

[Steve Langasek (vorlon)]

'Good' image with cloud-init from yakkety-proposed manually added for testing, now at <https://people.canonical.com/~vorlon/lp.1677376-yakkety-good.ext4.xz>.

If someone could test these in an openstack per the test case, that would help move this SRU forward.

[Steve Langasek (vorlon)]

Here's for yakkety. Image with cloud-init 0.7.9-48-g1c795b9-0ubuntu1~16.10.1:

2017-04-19 20:35:34,470 - cc_growpart.py[DEBUG]: No 'growpart' entry in cfg. Using default: {'devices': ['/'], 'mode': 'auto', 'ignore_growroot_disabled': False}
2017-04-19 20:35:34,470 - util.py[DEBUG]: Running command ['growpart', '--help'] with allowed return codes [0] (shell=False, capture=True)
2017-04-19 20:35:34,502 - util.py[DEBUG]: Reading from /proc/1112/mountinfo (quiet=False)
2017-04-19 20:35:34,502 - util.py[DEBUG]: Read 2661 bytes from /proc/1112/mountinfo
2017-04-19 20:35:34,503 - util.py[DEBUG]: resize_devices took 0.001 seconds
2017-04-19 20:35:34,503 - cc_growpart.py[DEBUG]: '/' SKIPPED: stat of '/dev/root' failed: [Errno 2] No such file or directory: '/dev/root'
[...]
2017-04-19 20:35:34,504 - cc_resizefs.py[DEBUG]: resize_info: dev=/dev/root mnt_point=/ path=/
[...]
2017-04-19 20:35:34,509 - cc_resizefs.py[DEBUG]: Converted /dev/root to '/dev/PARTUUID=19fa8722-01' per kernel cmdline
2017-04-19 20:35:34,509 - cc_resizefs.py[WARNING]: Device '/dev/PARTUUID=19fa8722-01' did not exist. cannot resize: dev=/dev/root mnt_point=/ path=/

Image with cloud-init 0.7.9-90-g61eb03fe-0ubuntu1~16.10.1:

2017-04-19 21:41:21,850 - cc_growpart.py[DEBUG]: No 'growpart' entry in cfg. Using default: {'ignore_growroot_disabled': False, 'devices': ['/'], 'mode': 'auto'}
2017-04-19 21:41:21,850 - util.py[DEBUG]: Running command ['growpart', '--help'] with allowed return codes [0] (shell=False, capture=True)
2017-04-19 21:41:21,888 - util.py[DEBUG]: Reading from /proc/1113/mountinfo (quiet=False)
2017-04-19 21:41:21,888 - util.py[DEBUG]: Read 2661 bytes from /proc/1113/mountinfo
[...]
2017-04-19 21:41:21,897 - util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/sda', '1'] with allowed return codes [0] (shell=False, capture=True)
2017-04-19 21:41:22,105 - util.py[DEBUG]: Running command ['growpart', '/dev/sda', '1'] with allowed return codes [0] (shell=False, capture=True)
2017-04-19 21:41:22,139 - util.py[DEBUG]: resize_devices took 0.251 seconds
2017-04-19 21:41:22,139 - cc_growpart.py[INFO]: '/' resized: changed (/dev/sda, 1) from 2359296000 to 10736352768
[...]
2017-04-19 21:41:22,140 - cc_resizefs.py[DEBUG]: resize_info: dev=/dev/root mnt_point=/ path=/
[...]
2017-04-19 21:41:22,149 - cc_resizefs.py[DEBUG]: Converted /dev/root to '/dev/disk/by-partuuid/a5711545-01' per kernel cmdline
2017-04-19 21:41:22,149 - cc_resizefs.py[DEBUG]: Resizing / (ext4) using resize2fs /dev/disk/by-partuuid/a5711545-01
2017-04-19 21:41:22,149 - util.py[DEBUG]: Running command ('resize2fs', '/dev/disk/by-partuuid/a5711545-01') with allowed return codes [0] (shell=False, capture=True)
2017-04-19 21:41:22,253 - util.py[DEBUG]: Resizing took 0.104 seconds
2017-04-19 21:41:22,254 - cc_resizefs.py[DEBUG]: Resized root filesystem (type=ext4, val=True)

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-90-g61eb03fe-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - OpenStack: add 'dvs' to the list of physical link types. (LP: #1674946)
    - Fix bug that resulted in an attempt to rename bonds or vlans.
      (LP: #1669860)
    - tests: update OpenNebula and Digital Ocean to not rely on host
      interfaces.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without
      initramfs. [Steve Langasek] (LP: #1677376)
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)
    - Fix filesystem creation when using "partition: auto"
      [Jonathan Ballet] (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive.
      (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3. [root]
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)
    - cloudinit.net: add network config v2 parsing and rendering [Ryan Harper]
    - net: Fix incorrect call to isfile [Joshua Powers] (LP: #1674317)
    - net: add renderers for automatically selecting the renderer.
    - doc: fix config drive doc with regard to unpartitioned disks.
      (LP: #1673818)
    - test: Adding integratiron test for password as list [Joshua Powers]
    - render_network_state: switch arguments around, do not require target
    - support 'loopback' as a device type.
    - Integration Testing: improve testcase subclassing [Wesley Wiedenmeier]
    - gitignore: adding doc/rtd_html [Joshua Powers]
    - doc: add instructions for running integration tests via tox.
      [Joshua Powers]
    - test: avoid differences in 'date' output due to daylight savings.
    - Fix chef config module in omnibus install. [Jeremy Melvin] (LP: #1583837)
    - Add feature flags to cloudinit.version. [Wesley Wiedenmeier]
    - tox: add a citest environment
    - Support chpasswd/list being a list in addition to a string.
      [Sergio Lystopad] (LP: #1665694)
    - doc: Fix configuration example for cc_set_passwords module.
      [Sergio Lystopad] (LP: #1665773)
    - ...

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - OpenStack: add 'dvs' to the list of physical link types. (LP: #1674946)
    - Fix bug that resulted in an attempt to rename bonds or vlans.
      (LP: #1669860)
    - tests: update OpenNebula and Digital Ocean to not rely on host
      interfaces.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without
      initramfs. [Steve Langasek] (LP: #1677376)
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)
    - Fix filesystem creation when using "partition: auto"
      [Jonathan Ballet] (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive.
      (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3. [root]
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)
    - cloudinit.net: add network config v2 parsing and rendering [Ryan Harper]
    - net: Fix incorrect call to isfile [Joshua Powers] (LP: #1674317)
    - net: add renderers for automatically selecting the renderer.
    - doc: fix config drive doc with regard to unpartitioned disks.
      (LP: #1673818)
    - test: Adding integratiron test for password as list [Joshua Powers]
    - render_network_state: switch arguments around, do not require target
    - support 'loopback' as a device type.
    - Integration Testing: improve testcase subclassing [Wesley Wiedenmeier]
    - gitignore: adding doc/rtd_html [Joshua Powers]
    - doc: add instructions for running integration tests via tox.
      [Joshua Powers]
    - test: avoid differences in 'date' output due to daylight savings.
    - Fix chef config module in omnibus install. [Jeremy Melvin] (LP: #1583837)
    - Add feature flags to cloudinit.version. [Wesley Wiedenmeier]
    - tox: add a citest environment
    - Support chpasswd/list being a list in addition to a string.
      [Sergio Lystopad] (LP: #1665694)
    - doc: Fix configuration example for cc_set_passwords module.
      [Sergio Lystopad] (LP: #1665773...

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2847