os-prober will double mount multipath partitions

[Ryan Harper]

==== Begin SRU Template ========
[Impact]
During an update-grub in the target system when using multipath
the 'os-prober' tool used to discover other operating systems
to add to the grub menu does not handle duplicate paths and
mounts the same disk up twice causing unknown behavior which
sometimes results in timeouts during installation.

[Test Case]
This failure was transient, but the test cases that showed the
failure were
 tests/vmtests/test_multipath.py

So to verify fix, best path is to just run vmtest successfully.

[Regression Potential]
Likeliest failure path would be for multipath installation to
not boot, or fail to install.

[Other Info]
==== End SRU Template ========

1. % lsb_release -rd
Description: Ubuntu 16.04.1 LTS
Release: 16.04

2. % apt-cache policy os-prober
os-prober:
  Installed: 1.70ubuntu3
  Candidate: 1.70ubuntu3
  Version table:
 *** 1.70ubuntu3 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status

3. os-prober should skip over multipath partitions

4. os-prober only skips currently mounted partitions and ends up mounting the same filesystem a second time with grub-mount; sometimes this results in corrupt reads from /var/lib/os-prober/<mount>/boot/grub/grub.cfg which results in undefined but broken behavior

On multipath systems, you have two block devices pointing to the same disk; when we create and mount a filesytem on one (/dev/sda1) we also see the results on a second device (/dev/sdb1)

grub hooks to calling os-prober which checks /proc/mounts for mounted partitions; it skips /dev/sda1, but when checking if /dev/sdb1 is mounted, its not listed in /proc/mounts and then continues to call grub-mount on /dev/sdb1; this uses grub-fuse to mount the disk a second time.

Even if multipath-tools and the module (dm-multipath) are not loaded, it is possible to detect multipath by examining the FSUUID on the block devices, blkid for example, will show that /dev/sda1 and /dev/sdb1 have the same UUID.

There is a patch for skipping multipath but requires multipathd and the dm-multipath module to be loaded.

https://build-test.opensuse.org/package/view_file/home:coolo:test/os-prober/os-prober-skip-part-on-multipath.patch?expand=1

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: os-prober 1.70ubuntu3
ProcVersionSignature: Ubuntu 4.4.0-59.80-generic 4.4.35
Uname: Linux 4.4.0-59-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
ApportVersion: 2.20.1-0ubuntu2.4
Architecture: amd64
CurrentDesktop: GNOME-Flashback:GNOME
Date: Fri Jan 13 11:17:25 2017
Dependencies:
 gcc-6-base 6.0.1-0ubuntu1
 libc6 2.23-0ubuntu5
 libgcc1 1:6.0.1-0ubuntu1
InstallationDate: Installed on 2016-01-01 (378 days ago)
InstallationMedia: Ubuntu 16.04 LTS "Xenial Xerus" - Alpha amd64 (20151209)
SourcePackage: os-prober
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 0.1.0~bzr459-0ubuntu1

---------------
curtin (0.1.0~bzr459-0ubuntu1) zesty; urgency=medium

  * New upstream snapshot.
    - Remove style checking during build and add latest style checks to tox
    - avoid UnicodeDecode error on passing non-utf8 into shlex
    - adjustments to version string handling, improved pack unit tests.
    - helpers/common: Add grub install debugging output
    - curtin: add version module and display in output and logs
    - content decoding in load_file, apply_net raise exception on errors
    - gpg: retry when recv'ing gpg keys fail (LP: #1661337)
    - Add clear_holders checks to disk and partition handlers (LP: #1659509)
    - net: add new lines after rendered static routes. (LP: #1649652)
    - multipath: don't run update-grub; setup_grub will handle this better.
      (LP: #1656369)
    - test related changes:
      - vmtest: Add tests for zesty and Trusty HWE-X kernels.
      - tests: fix tox tip-pycodestyle complaints
      - image-sync: add debugging output to help diagnose errors
      - vmtest: change get_curtin_version to use version subcommand.
      - subp doc an unit test improvements.
      - vmtest: Add the ability to add extra config files to test execution.
      - vmtest: overhaul image sync
      - vmtest: skip apt-proxy test if not set
      - vmtest: add 'webserv' helper
      - vmtest: add CURTIN_VMTEST_CURTIN_EXE variable.

 -- Scott Moser <email address hidden> Wed, 08 Feb 2017 15:29:57 -0500

[Brian Murray (brian-murray)]

Hello Ryan, or anyone else affected,

Accepted curtin into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr460-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Ryan, or anyone else affected,

Accepted curtin into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr460-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

I've run curtin's vmtest testsuite with the test provided in the
description by doing the following.

# get curtin for it tests at revision 460
$ bzr branch lp:curtin curtin.dist
$ bzr branch -r 460 curtin.dist curtin-r460
$ cd curtin-r460
## need to get 'curtainer' and 'curtin-in-container' from trunk
$ cp ../trunk.dist/tools/{curtainer,curtin-from-container} tools/
$ ./tools/vmtest-system-setup
$ ./tools/curtainer --proposed xenial sm-curtin-x1
...
Unpacking curtin (0.1.0~bzr460-0ubuntu1~16.04.1) ...
Setting up curtin-common (0.1.0~bzr460-0ubuntu1~16.04.1) ...
Setting up python3-curtin (0.1.0~bzr460-0ubuntu1~16.04.1) ...
Setting up curtin (0.1.0~bzr460-0ubuntu1~16.04.1) ...

# bug 1656369 tests/vmtests/test_multipath.py
# bug 1659509 tests/unittests/test_commands_block_meta.py
# bug 1661337 tests/vmtests/test_apt_source.py
$ name=sm-curtin-x1
$ CURTIN_VMTEST_TOPDIR=$PWD/$name CURTIN_VMTEST_CURTIN_EXE="./tools/curtin-from-container $name curtin" \
   ./tools/jenkins-runner \
     tests/vmtests/test_multipath.py \
     tests/unittests/test_commands_block_meta.py \
     tests/vmtests/test_apt_source.py

I'm attaching a tarball of the output directory sm-curtin-x1

Note, due to packaging bug 1666986, we see in the installation logs:
   curtin: Installation started. (0.1.0)
when that is fixed, we'll start seeing 0.1.0~bzr460-0ubuntu1~16.04.1.

[Scott Moser (smoser)]

I've run curtin's vmtest testsuite with the test provided in the
description by doing the following.

# get curtin for it tests at revision 460
$ bzr branch lp:curtin curtin.dist
$ bzr branch -r 460 curtin.dist curtin-r460
$ cd curtin-r460
## need to get 'curtainer' and 'curtin-in-container' from trunk
$ cp ../trunk.dist/tools/{curtainer,curtin-from-container} tools/
$ ./tools/vmtest-system-setup
$ ./tools/curtainer images:ubuntu/yakkety --proposed sm-curtin-y1
# note, used images: due to bug 1668710
$ ./tools/curtainer --proposed yakkety sm-curtin-y1
....
Setting up curtin (0.1.0~bzr460-0ubuntu1~16.10.1) ...

# bug 1656369 tests/vmtests/test_multipath.py
# bug 1659509 tests/unittests/test_commands_block_meta.py
# bug 1661337 tests/vmtests/test_apt_source.py
$ name=sm-curtin-y1
$ CURTIN_VMTEST_TOPDIR=$PWD/$name CURTIN_VMTEST_CURTIN_EXE="./tools/curtin-from-container $name curtin" \
   ./tools/jenkins-runner \
     tests/vmtests/test_multipath.py \
     tests/unittests/test_commands_block_meta.py \
     tests/vmtests/test_apt_source.py

Note, due to packaging bug 1666986, we see in the installation logs:
   curtin: Installation started. (0.1.0)
when that is fixed, we'll start seeing 0.1.0~bzr460-0ubuntu1~16.10.1.

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 0.1.0~bzr460-0ubuntu1~16.10.1

---------------
curtin (0.1.0~bzr460-0ubuntu1~16.10.1) yakkety-proposed; urgency=medium

  * New upstream snapshot.
    - Install zipl in target on s390x arch. (LP: #1662346)
    - avoid UnicodeDecode error on passing non-utf8 into shlex
    - adjustments to version string handling, improved pack unit tests.
    - helpers/common: Add grub install debugging output
    - curtin: add version module and display in output and logs
    - content decoding in load_file, apply_net raise exception on errors
    - gpg: retry when recv'ing gpg keys fail (LP: #1661337)
    - Add clear_holders checks to disk and partition handlers (LP: #1659509)
    - net: add new lines after rendered static routes. (LP: #1649652)
    - multipath: don't run update-grub; setup_grub will handle this better.
      (LP: #1656369)
    - Test changes:
      - vmtest: Add tests for zesty and Trusty HWE-X kernels.
      - tests: fix tox tip-pycodestyle complaints
      - image-sync: add debugging output to help diagnose errors
      - vmtest: change get_curtin_version to use version subcommand.
      - Remove style checking during build and add latest style checks to tox
      - subp doc an unit test improvements.
      - flake8: remove unused variable.
      - vmtest: Add the ability to add extra config files to test execution.
      - vmtest: overhaul image sync
      - vmtest: skip apt-proxy test if not set
      - vmtest: add 'webserv' helper
      - vmtest: add CURTIN_VMTEST_CURTIN_EXE variable.

 -- Scott Moser <email address hidden> Thu, 16 Feb 2017 22:30:13 -0500

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for curtin has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 0.1.0~bzr460-0ubuntu1~16.04.1

---------------
curtin (0.1.0~bzr460-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream snapshot.
    - Install zipl in target on s390x arch. (LP: #1662346)
    - avoid UnicodeDecode error on passing non-utf8 into shlex
    - adjustments to version string handling, improved pack unit tests.
    - helpers/common: Add grub install debugging output
    - curtin: add version module and display in output and logs
    - content decoding in load_file, apply_net raise exception on errors
    - gpg: retry when recv'ing gpg keys fail (LP: #1661337)
    - Add clear_holders checks to disk and partition handlers (LP: #1659509)
    - net: add new lines after rendered static routes. (LP: #1649652)
    - multipath: don't run update-grub; setup_grub will handle this better.
      (LP: #1656369)
    - Test changes:
      - vmtest: Add tests for zesty and Trusty HWE-X kernels.
      - tests: fix tox tip-pycodestyle complaints
      - image-sync: add debugging output to help diagnose errors
      - vmtest: change get_curtin_version to use version subcommand.
      - Remove style checking during build and add latest style checks to tox
      - subp doc an unit test improvements.
      - flake8: remove unused variable.
      - vmtest: Add the ability to add extra config files to test execution.
      - vmtest: overhaul image sync
      - vmtest: skip apt-proxy test if not set
      - vmtest: add 'webserv' helper
      - vmtest: add CURTIN_VMTEST_CURTIN_EXE variable.

 -- Scott Moser <email address hidden> Wed, 08 Feb 2017 19:40:38 -0500