MBR disk setup fails because sfdisk no longer accepts M as a valid unit

[Dan Watkins]

=== Begin SRU Template ===
[Impact]
Cloud-init has function to partition disks on devices.
Creating partitions on a disk no longer works with sfdisk as recent versions of
sfdisk no accept the unit 'M' as input, this function is broken.

[Test Case]
1. Launch an instance with provided user-data

   On Azure, this will work:
     #cloud-config
     disk_setup:
       ephemeral0:
           table_type: mbr
           layout: [66, [33, 82]]
           overwrite: True
     fs_setup:
       - device: ephemeral0.1
         filesystem: ext4
       - device: ephemeral0.2
         filesystem: swap
     mounts:
       - ["ephemeral0.1", "/mnt2"]
       - ["ephemeral0.2", "none", "swap", "sw", "0", "0"]

   On a typical kvm openstack use:
     #cloud-config
     disk_setup:
       /dev/vdb:
           table_type: mbr
           layout: [66, [33, 82]]
           overwrite: True
     fs_setup:
       - device: /dev/vdb1
         filesystem: ext4
       - device: /dev/vdb2
         filesystem: swap
     mounts:
       - ["/dev/vdb1", "/mnt2"]
       - ["/dev/vdb2", "none", "swap", "sw", "0", "0"]

2. Add proposed, update, and reboot as fresh instance.

   # enable proposed
   echo deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main | sudo tee /etc/apt/sources.list.d/proposed.list
   sudo apt-get -qy update && sudo apt-get -qy install cloud-init </dev/null

   # clear out previous state in cloud-init
   sudo rm -Rf /var/lib/cloud /var/log/cloud-init*

   # wipe the partition table and filesystem from the disk
   sudo dd if=/dev/zero of=/dev/[sv]db bs=1M count=10

   # remove cloud-init written entries in /etc/fstab to mimic clean first boot
   sudo sed -i '/cloudconfig/d' /etc/fstab
   sudo reboot

3. login to system and look around.

   The target disk should be partitioned with 2 partitions.

   $ grep [sv]db /proc/partitions
   253 16 41943040 vdb
   253 17 27682406 vdb1
   253 18 14259200 vdb2

   There should entries in /etc/fstab for the configured devices
   $ cat /etc/fstab
   LABEL=cloudimg-rootfs / ext4 defaults 0 0
   LABEL=UEFI /boot/efi vfat defaults 0 0
   /dev/vdb1 /mnt2 auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2
   /dev/vdb2 none swap sw,comment=cloudconfig 0 0

   A 'mount -a' will mount the necessary devices.
   Note, these should ideally be already mounted, but an as-yet unidentified
   issue (to be fixed in bug 1642383) means they are not mounted on first
   boot.
   $ mount -a
   $ df -h /mnt2
   Filesystem Size Used Avail Use% Mounted on
   /dev/vdb1 26G 45M 25G 1% /mnt2

   There should be swap used.
   $ free
             total used free shared buff/cache available
   Mem: 2047792 73984 1804508 3252 169300 1825896
   Swap: 14259196 0 14259196

   $ grep [vs]db2 /proc/swaps
   /dev/vdb2 partition 14259196 0 -1

[Regression Potential]
Change from using Megabytes to Sectors could have caused math errors,
most likely with sector sizes other than 512.
Partitioning with sfdisk was broken previously, so it wont be worse.

=== End SRU Template ===

Specifically, we get the following output in cc_disk_setup.exec_mkpart_mbr:

sfdisk: --Linux option is unnecessary and deprecated
sfdisk: unsupported unit 'M'

and the manpage says:

       -u, --unit S
              Deprecated option. Only the sector unit is supported.

So we'll need to shift to using sectors.

Related bugs:
 * bug 1642383: Unable to configure swap space on ephemeral disk in Azure

[Dan Watkins (oddbloke)]

I'll look at this in our next iteration (i.e. the second half of June).

[Robert C Jennings (rcj)]

Removed "wily" in title as it is EOL but this issue still exists in supported releases.

[Scott Moser (smoser)]

For reference, this is the commit that added the deprecated message on --Linux and insists on sectors.
http://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=35ce145f6adc8ca15555bf976fd69af41879e161

[Robie Basak (racb)]

Hello Dan, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-49-g9e904bb-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

$ dpkg-query --show cloud-init
cloud-init 0.7.8-49-g9e904bb-0ubuntu1~16.04.2

$ cat /etc/cloud/build.info
build_name: server
serial: 20161214

$ df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 1018216 0 1018216 0% /dev
tmpfs 204836 3180 201656 2% /run
/dev/vda1 20263528 973360 19273784 5% /
tmpfs 1024172 0 1024172 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 1024172 0 1024172 0% /sys/fs/cgroup
tmpfs 204836 0 204836 0% /run/user/1000

$ grep [sv]db /proc/partitions
 253 16 41943040 vdb
 253 17 27682406 vdb1
 253 18 14259200 vdb2

$ cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 0
/dev/vdb1 /mnt2 auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2
/dev/vdb2 none swap sw,comment=cloudconfig 0 0

$ sudo mount -a

$ df -h /mnt2
Filesystem Size Used Avail Use% Mounted on
/dev/vdb1 26G 44M 25G 1% /mnt2
$ free
              total used free shared buff/cache available
Mem: 2048348 48596 1857652 3208 142100 1844252
Swap: 14259196 0 14259196
$ grep [vs]db2 /proc/swaps
/dev/vdb2 partition 14259196 0 -1

[Scott Moser (smoser)]

The above is verification done on an openstack instance after the 'clean' and reboot as seen in the bug description.

[Scott Moser (smoser)]

heres the same basic thing on azure.

$ df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 321736 0 321736 0% /dev
tmpfs 68328 3216 65112 5% /run
/dev/sda1 29711408 1416484 28278540 5% /
tmpfs 341636 0 341636 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 341636 0 341636 0% /sys/fs/cgroup
tmpfs 68328 0 68328 0% /run/user/1000

$ dpkg-query --show cloud-init
cloud-init 0.7.8-49-g9e904bb-0ubuntu1~16.04.2

$ cat /etc/cloud/build.info
build_name: server
serial: 20161213

$ df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 321736 0 321736 0% /dev
tmpfs 68328 3216 65112 5% /run
/dev/sda1 29711408 1416484 28278540 5% /
tmpfs 341636 0 341636 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 341636 0 341636 0% /sys/fs/cgroup
tmpfs 68328 0 68328 0% /run/user/1000

$ grep [sv]db /proc/partitions
   8 16 20971520 sdb
   8 17 13841203 sdb1
   8 18 7129088 sdb2

$ cat /etc/fstab
# CLOUD_IMG: This file was created/modified by the Cloud Image build process
UUID=c50d1651-3fd4-4fae-9d8d-1713e9bb191e / ext4 defaults,discard0 0
/dev/disk/cloud/azure_resource-part1 /mnt2 auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2
/dev/disk/cloud/azure_resource-part2 none swap sw,comment=cloudconfig 00

$ sudo mount -a

$ df -h /mnt2
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 13G 33M 13G 1% /mnt2

$ free
              total used free shared buff/cache available
Mem: 683276 109488 410920 3244 162868 445812
Swap: 7129084 0 7129084

$ grep [vs]db2 /proc/swaps
/dev/sdb2 partition 7129084 0 -1

[Barry Warsaw (barry)]

This probably affects u-i too.

[Steve Langasek (vorlon)]

We don't use the -u option to sfdisk.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.8-49-g9e904bb-0ubuntu1~16.04.2

---------------
cloud-init (0.7.8-49-g9e904bb-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * cherry-pick 18203bf: disk_setup: Use sectors as unit when formatting
    MBR disks with sfdisk. (LP: #1460715)
  * cherry-pick 6e92c5f: net/cmdline: Consider ip= or ip6= on command
    line not only ip= (LP: #1639930)
  * cherry-pick 8c6878a: tests: fix assumptions that expected no eth0 in
    system. (LP: #1644043)
  * cherry-pick 2d2ec70: OpenStack: extend physical types to include
    hyperv, hw_veb, vhost_user. (LP: #1642679)

 -- Scott Moser <email address hidden> Thu, 01 Dec 2016 16:57:39 -0500

[Robie Basak (racb)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Scott Moser (smoser)]

This is fixed in cloud-init 0.7.9.

[Brian Murray (brian-murray)]

Hello Dan, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-68-gca3ae67-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

Marking this verification-done for yakkety.
## fresh instance openstack kvm instance of yakkety.
$ dpkg-query --show cloud-init
cloud-init 0.7.8-49-g9e904bb-0ubuntu1~16.10.1

$ lsb_release -sc
yakkety

$ cat /etc/cloud/build.info
build_name: server
serial: 20170114

$ m=http://archive.ubuntu.com/ubuntu
$ echo deb $m $(lsb_release -sc)-proposed main |
  sudo tee /etc/apt/sources.list.d/proposed.list
$ sudo apt-get -qy update && sudo apt-get -qy install cloud-init </dev/null

$ dpkg-query --show cloud-init
cloud-init 0.7.8-68-gca3ae67-0ubuntu1~16.10.1

# clear out state for cloud-init, wipe partition table, clean fstab, reboot
$ sudo rm -Rf /var/lib/cloud /var/log/cloud-init*
$ sudo dd if=/dev/zero of=/dev/[sv]db bs=1M count=10
$ sudo sed -i '/cloudconfig/d' /etc/fstab

## ssh back in look around.
# see that vdb is partitioned
$ grep [sv]db /proc/partitions
 252 16 41943040 vdb
 252 17 27682406 vdb1
 252 18 14259200 vdb2

# There should entries in /etc/fstab for the configured devices
$ cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0
/dev/vdb1 /mnt2 auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2
/dev/vdb2 none swap sw,comment=cloudconfig 0 0

## vdb1 is mounted at /mnt2 and swap is used
$ df | grep -v tmpfs
Filesystem 1K-blocks Used Available Use% Mounted on
udev 1016700 0 1016700 0% /dev
/dev/vda1 20145724 1069040 19060300 6% /
/dev/vda15 106858 3607 103251 4% /boot/efi
/dev/vdb1 27116088 45080 25670504 1% /mnt2

$ grep /dev/[sv]db1 /proc/mounts
/dev/vdb1 /mnt2 ext4 rw,relatime,data=ordered 0 0
$ grep /dev/[sv]db2 /proc/swaps
/dev/vdb2 partition 14259196 0 -1

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.8-68-gca3ae67-0ubuntu1~16.10.1

---------------
cloud-init (0.7.8-68-gca3ae67-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/cherry-pick: add utility for cherry picking commits from upstream
    into patches in debian/patches.
  * New upstream snapshot.
    - mounts: use mount -a again to accomplish mounts (LP: #1647708)
    - CloudSigma: Fix bug where datasource was not loaded in local search.
      (LP: #1648380)
    - when adding a user, strip whitespace from group list
      [Lars Kellogg-Stedman] (LP: #1354694)
    - fix decoding of utf-8 chars in yaml test
    - Replace usage of sys_netdev_info with read_sys_net (LP: #1625766)
    - fix problems found in python2.6 test.
    - OpenStack: extend physical types to include hyperv, hw_veb, vhost_user.
      (LP: #1642679)
    - tests: fix assumptions that expected no eth0 in system. (LP: #1644043)
    - net/cmdline: Consider ip= or ip6= on command line not only ip=
      (LP: #1639930)
    - Just use file logging by default [Joshua Harlow] (LP: #1643990)
    - Improve formatting for ProcessExecutionError [Wesley Wiedenmeier]
    - flake8: fix trailing white space
    - Doc: various documentation fixes [Sean Bright]
    - cloudinit/config/cc_rh_subscription.py: Remove repos before adding
      [Brent Baude]
    - packages/redhat: fix rpm spec file.
    - main: set TZ in environment if not already set. [Ryan Harper]
    - disk_setup: Use sectors as unit when formatting MBR disks with sfdisk.
      [Daniel Watkins] (LP: #1460715)

 -- Scott Moser <email address hidden> Mon, 19 Dec 2016 15:07:12 -0500

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2556