Ephemeral disk on xenial is not mounted at boot

Bug #1647708 reported by Dan Watkins on 2016-12-06
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
cloud-init
High
Unassigned
cloud-init (Ubuntu)
Critical
Unassigned
Xenial
Critical
Unassigned
Yakkety
High
Unassigned

Bug Description

=== Begin SRU Template ===
[Impact]
An ephemeral disk will not correctly be mounted on /mnt.
This affects Azure and other instances where an ephemeral device is
mounted by default on /mnt. It is recreated in Azure and on OpenStack.

This can be mitigated by either:
 1.) mount -a
 2.) reboot

[Test Case]
In a correctly functioning image on OpenStack or Azure, you should be
able to:

1.) Launch an instance
2.) ssh into instance and look around
   $ awk '$2 == "/mnt" { print $0 }' /etc/fstab
   /dev/vdb /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2
   $ awk '$2 == "/mnt" { print $0 }' /proc/mounts
/dev/vdb /mnt vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0

   $ df -h /mnt
   Filesystem Size Used Avail Use% Mounted on
   /dev/vdb 40G 32K 40G 1% /mnt

To check that this is functional with -proposed, after you've seen it failed

3.) enable -proposed and apt-get install cloud-init
4.) clean up and reboot as if fresh:
    sudo rm -Rf /var/lib/cloud /var/log/cloud-init
    sudo sed -i '/cloudconfig/d' /etc/fstab
    sudo reboot

[Regression Potential]
This is a regression caused by bug 1611074, so in addition to the above
test case, we should go through the test cases shown there to see that
those also work.
=== End SRU Template ===

When I boot the latest xenial Azure image (containing cloud-init 0.7.8-49-g9e904bb-0ubuntu1~16.04.1), the ephemeral disk does not end up mounted (though it is formatted appropriately). Restarting the mnt.mount service does mount it, which suggests there is an issue in the ordering of the services at boot.

$ mount | grep mnt
$ sudo systemctl status mnt.mount
● mnt.mount - /mnt
   Loaded: loaded (/etc/fstab; bad; vendor preset: enabled)
   Active: inactive (dead)
    Where: /mnt
     What: /dev/disk/cloud/azure_resource-part1
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
$ sudo journalctl -u mnt.mount
-- No entries --
$ cat /etc/fstab
# CLOUD_IMG: This file was created/modified by the Cloud Image build process
UUID=6a8554fa-8e1d-4916-ba03-4ca3837feb34 / ext4 defaults,discard 0 0
/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2
$ sudo systemctl restart mnt.mount
$ mount | grep mnt
/dev/sdb1 on /mnt type ext4 (rw,relatime,data=ordered)
$ ls /mnt/
lost+found
$ sudo systemctl status mnt.mount
● mnt.mount - /mnt
   Loaded: loaded (/etc/fstab; bad; vendor preset: enabled)
   Active: active (mounted) since Tue 2016-12-06 12:49:06 UTC; 6s ago
    Where: /mnt
     What: /dev/sdb1
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
  Process: 1916 ExecMount=/bin/mount /dev/disk/cloud/azure_resource-part1 /mnt -o defaults,x-systemd.requires=cloud-init.service,comment=cloudconfig (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 88.0K
      CPU: 15ms

Dec 06 12:49:06 xenial-161206-1345 systemd[1]: Mounting /mnt...
Dec 06 12:49:06 xenial-161206-1345 systemd[1]: Mounted /mnt.

Related bugs:
 * bug 1611074: Reformatting of ephemeral drive fails on resize of Azure VM
 * bug 1642383: Unable to configure swap space on ephemeral disk in Azure

Related branches

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cloud-init (Ubuntu Xenial):
status: New → Confirmed
Changed in cloud-init (Ubuntu):
status: New → Confirmed
Andrew Spiers (3-andrew) wrote :

seems like maybe exchanging 'mount -a' for daemon-reload might've triggered this:

https://git.launchpad.net/cloud-init/commit/cloudinit/config/cc_mounts.py?id=9e904bbc3336b96475bfd00fb3bf1262ae4de49f

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.8-67-gc9c9197-0ubuntu1

---------------
cloud-init (0.7.8-67-gc9c9197-0ubuntu1) zesty; urgency=medium

  * debian/cherry-pick: add utility for cherry picking commits from upstream
    into patches in debian/patches.
  * New upstream snapshot.
    - mounts: use mount -a again to accomplish mounts (LP: #1647708)
    - CloudSigma: Fix bug where datasource was not loaded in local search. (LP: #1648380)
    - when adding a user, strip whitespace from group list (LP: #1354694)
    - fix decoding of utf-8 chars in yaml test
    - Replace usage of sys_netdev_info with read_sys_net (LP: #1625766)
    - fix problems found in python2.6 test.

 -- Scott Moser <email address hidden> Sun, 11 Dec 2016 21:22:57 -0500

Changed in cloud-init (Ubuntu):
status: Confirmed → Fix Released
Scott Moser (smoser) on 2016-12-13
Changed in cloud-init (Ubuntu Yakkety):
status: New → Confirmed
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Xenial):
importance: Undecided → Critical
Changed in cloud-init (Ubuntu Yakkety):
importance: Medium → High
description: updated
Scott Moser (smoser) on 2016-12-13
description: updated
Robie Basak (racb) wrote :

The SRU to Xenial here is blocked by 0.7.8-49-g9e904bb-0ubuntu1~16.04.2 that is currently in xenial-proposed.

Can't we merge the two uploads so as not to delay things until one or the other is out of proposed?

Jon Grimm (jgrimm) wrote :

Discussion on IRC was to not disrupt the current testing of xenial-proposed. Robie has already reviewed this SRU and will process it as soon as 0.7.8-49-g9e904bb-0ubuntu1~16.04.2 clears (should be Monday assuming all verification/testing is done.

Jon Grimm (jgrimm) on 2016-12-15
Changed in cloud-init (Ubuntu Xenial):
status: Confirmed → In Progress

Hello Dan, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-49-g9e904bb-0ubuntu1~16.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
Scott Moser (smoser) on 2016-12-20
tags: added: verification-done-xenial
removed: verification-needed
Scott Moser (smoser) wrote :

I've verified this as described in the test case.

## fresh instance
$ dpkg-query --show cloud-init
0.7.8-49-g9e904bb-0ubuntu1~16.04.1

$ cat /etc/cloud/build.info
build_name: server
serial: 20161213

## /proc/mountsn does not have the /mnt mount, but /etc/fstab does
$ awk '$2 == "/mnt" { print $0 }' /proc/mounts
$ awk '$2 == "/mnt" { print $0 }' /etc/fstab
/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2

$ m=http://azure.archive.ubuntu.com/ubuntu/
$ echo deb $m xenial-proposed main universe |
   sudo tee /etc/apt/sources.list.d/proposed.list

$ sudo apt-get update && sudo apt-get install -qy cloud-init

$ dpkg-query --show cloud-init
cloud-init 0.7.8-49-g9e904bb-0ubuntu1~16.04.3

## Clean up for new reboot
# remove old entry in /etc/fstab
$ sudo sudo sed -i '/cloudconfig/d' /etc/fstab
$ sudo umount /mnt

## reformat to ntfs to look like first boot
$ disk=/dev/disk/cloud/azure_resource
$ echo "2048,1024000,7" | sudo sfdisk --force "$disk"
$ sudo udevadm settle
$ sudo umount /mnt
$ time sudo mkfs.ntfs --quick ${disk}-part1

## remove state
$ sudo rm -Rf /var/lib/cloud/ /var/log/cloud-init*
$ sudo reboot

## ssh back in ..
$ awk '$2 == "/mnt" { print $0 }' /proc/mounts
/dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0
$ awk '$2 == "/mnt" { print $0 }' /etc/fstab
/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2

$ grep Activ /var/log/cloud-init.log
2016-12-20 18:52:41,270 - cc_mounts.py[DEBUG]: Activate mounts: PASS:mount -a
2016-12-20 18:52:41,507 - cc_mounts.py[DEBUG]: Activate mounts: PASS:systemctl daemon-reload

## for good measure, reboot again (to similate non-first boot).
$ sudo reboot

## ssh back in, look at /mnt is mounted
$ awk '$2 == "/mnt" { print $0 }' /proc/mounts
/dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0

Patricia Gaughen (gaughen) wrote :

I've validated that after installing the cloud-init from -proposed, and rebooting that the ephemeral disk was mounted at /mnt.

tags: added: verification-done
removed: verification-done-xenial
Patricia Gaughen (gaughen) wrote :

(and this validation was done on the latest daily xenial image)

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.8-49-g9e904bb-0ubuntu1~16.04.3

---------------
cloud-init (0.7.8-49-g9e904bb-0ubuntu1~16.04.3) xenial-proposed; urgency=medium

  * debian/cherry-pick: use git format-patch rather than git show
  * cherry-pick a9d41de: CloudSigma: Fix bug where datasource was not
    loaded in local (LP: #1648380)
  * cherry-pick c9c9197: mounts: use mount -a again to accomplish mounts
    (LP: #1647708)

 -- Scott Moser <email address hidden> Tue, 13 Dec 2016 16:02:50 -0500

Changed in cloud-init (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Scott Moser (smoser) wrote :

This is fixed in cloud-init 0.7.9.

Changed in cloud-init:
importance: Undecided → High
status: New → Fix Released

Hello Dan, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-68-gca3ae67-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Yakkety):
status: Confirmed → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Scott Moser (smoser) wrote :

## fresh instance
$ dpkg-query --show cloud-init
cloud-init 0.7.8-49-g9e904bb-0ubuntu1~16.10.1

$ cat /etc/cloud/build.info
build_name: server
serial: 20170114

## /proc/mounts does not have the /mnt mount, but /etc/fstab does
$ awk '$2 == "/mnt" { print $0 }' /proc/mounts
$ awk '$2 == "/mnt" { print $0 }' /etc/fstab
/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2

## enable proposed update
$ m=http://azure.archive.ubuntu.com/ubuntu/
$ echo deb $m $(lsb_release -sc)-proposed main universe |
   sudo tee /etc/apt/sources.list.d/proposed.list

$ sudo apt-get update && sudo apt-get install -qy cloud-init

$ dpkg-query --show cloud-init
cloud-init 0.7.8-68-gca3ae67-0ubuntu1~16.10.1

## Clean up for new reboot
# remove old entry in /etc/fstab
$ sudo sudo sed -i '/cloudconfig/d' /etc/fstab
$ sudo umount /mnt
umount: /mnt: not mounted

## reformat to ntfs to look like first boot
$ disk=/dev/disk/cloud/azure_resource
$ echo "2048,1024000,7" | sudo sfdisk --force "$disk"
$ sudo udevadm settle
$ sudo umount /mnt
$ blockdev --rereadpt "$disk"
$ time sudo mkfs.ntfs --quick ${disk}-part1

## remove state
$ sudo rm -Rf /var/lib/cloud/ /var/log/cloud-init*
$ sudo reboot

## ssh back in
$ awk '$2 == "/mnt" { print $0 }' /proc/mounts
/dev/sdb1 /mnt ext4 rw,relatime,stripe=4,data=ordered 0 0
$ awk '$2 == "/mnt" { print $0 }' /etc/fstab
/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2

$ grep Activ /var/log/cloud-init.log
2017-01-20 17:16:51,201 - cc_mounts.py[DEBUG]: Activate mounts: PASS:mount -a
2017-01-20 17:16:51,262 - cc_mounts.py[DEBUG]: Activate mounts: PASS:systemctl daemon-reload

# for good measure
$ sudo reboot

## ssh back in
$ awk '$2 == "/mnt" { print $0 }' /proc/mounts
/dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.8-68-gca3ae67-0ubuntu1~16.10.1

---------------
cloud-init (0.7.8-68-gca3ae67-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/cherry-pick: add utility for cherry picking commits from upstream
    into patches in debian/patches.
  * New upstream snapshot.
    - mounts: use mount -a again to accomplish mounts (LP: #1647708)
    - CloudSigma: Fix bug where datasource was not loaded in local search.
      (LP: #1648380)
    - when adding a user, strip whitespace from group list
      [Lars Kellogg-Stedman] (LP: #1354694)
    - fix decoding of utf-8 chars in yaml test
    - Replace usage of sys_netdev_info with read_sys_net (LP: #1625766)
    - fix problems found in python2.6 test.
    - OpenStack: extend physical types to include hyperv, hw_veb, vhost_user.
      (LP: #1642679)
    - tests: fix assumptions that expected no eth0 in system. (LP: #1644043)
    - net/cmdline: Consider ip= or ip6= on command line not only ip=
      (LP: #1639930)
    - Just use file logging by default [Joshua Harlow] (LP: #1643990)
    - Improve formatting for ProcessExecutionError [Wesley Wiedenmeier]
    - flake8: fix trailing white space
    - Doc: various documentation fixes [Sean Bright]
    - cloudinit/config/cc_rh_subscription.py: Remove repos before adding
      [Brent Baude]
    - packages/redhat: fix rpm spec file.
    - main: set TZ in environment if not already set. [Ryan Harper]
    - disk_setup: Use sectors as unit when formatting MBR disks with sfdisk.
      [Daniel Watkins] (LP: #1460715)

 -- Scott Moser <email address hidden> Mon, 19 Dec 2016 15:07:12 -0500

Changed in cloud-init (Ubuntu Yakkety):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers