block devices appear twice [install does not use multipath]
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | curtin |
High
|
Unassigned | ||
| | curtin (Ubuntu) |
Critical
|
Unassigned | ||
| | Trusty |
High
|
Unassigned | ||
| | Utopic |
Undecided
|
Unassigned | ||
| | Vivid |
High
|
Unassigned | ||
Bug Description
=== Begin SRU Information ===
[Description]
When curtin installs to a system that has multipath devices, it does not recognize this and enable them.
The result is that a system that has multipath devices installed will have Ubuntu installed to one of the non-multipath paths. The end user is then confused because they see root installed on /dev/sda1 and another device /dev/sdg1 (or other name) that has the same content as /dev/sda1. In addition to this confusion, if the user decides to use /dev/sdg1 and format it, they'll have shot themselves in the foot by destroying their root partition.
The general solution is to install multipath-
curtin currently does:
a.) installs multipath-boot *only* when multipath devices are found, avoiding bug 1463046.
b.) writes /etc/multipath.conf to use user_friendly_
c.) writes /etc/multipath/
[Impact]
Without this fix, most power 8 systems will install to /dev/sda1 and let the user modify /dev/sdg1 (or whatever device name that multipath device appears as). This is not very user friendly, and quite confusing.
[Test Case]
Positive case:
a.) install system that has multipath
b.) verify that multipath-
c.) verify that system is booted with root=/dev/
Negative case:
a.) install system with 2 disks without multipath
b.) verify that no multipath-
[Regression Potential]
The regression potential falls into 2 places:
a.) a system with multipath that installed without multipath fails to install or work after this
this is really a bug in multipath in kenrel or the package if we get here. We've done fairly significant testing to show that we're reliably installing.
b.) a system without multipath is identified as multipath
The detection of multipath basically consists of looking for 2 devices that have a filesystem with the same UUID on them after creating a filesystem with that UUID. This is essentially not more than a check for a random sequence of bytes on all disks rather than a fully determinable identification of multipath. So, the failure case is:
1.) we install to system and 'mkfs' the root partition
2.) we find 2 disks that have the same UUID as the root device (the root device and one other)
In all likelyhood, that would mean that a filesytem with this UUID already existed on the system. That seems quite unlikley.
In the event that the user does not have multipath systems and just wants to disable multipath to avoid that scenario, they can do so by config in maas setting the following in /etc/maas/
multipath:
mode: false
[Other]
Related bugs:
* bug 1432062 : multipath-
* bug 1463043 : some SM15K systems fail to boot after deployment - drop into initramfs shell
* bug 1462530 : multipath errors on vivid, wily kernel
* bug 1463046 : installation of multipath-
* bug 1447167 : [d-i] multipath-install: Installing with multipath enabled does not install multipath-tools
* bug 1429327 : Boot from an unique, stable, multipath-dependent symlink
=== End SRU Information ===
=== Original bug report ===
$ sudo blkid
/dev/sr0: LABEL="
/dev/sda2: UUID="795a6e14-
/dev/sda3: UUID="0a91d81f-
/dev/sdb2: UUID="1d14c1f3-
/dev/sdb3: UUID="9c228177-
/dev/sdg2: UUID="795a6e14-
/dev/sdg3: UUID="0a91d81f-
/dev/sdh2: UUID="1d14c1f3-
/dev/sdh3: UUID="9c228177-
I'm not sure what exactly those block devices are (as in if they're raided in hardware or they actually represent physical spinning disks). But I do know that writing data to sda causes that data to be readable from sdg.
The same is true:
sda -> sdg
sdb -> sdh
That is what causes those UUIDs to be similar. Everything is functional, you just have to know that if your root device is /dev/sdg that you should probably not write data to /dev/sda thinking you can use it as a disk.
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-
ProcVersionSign
Uname: Linux 3.13.0-35-generic ppc64le
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access /dev/snd/: No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3.4
Architecture: ppc64el
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
CRDA: Error: [Errno 2] No such file or directory: 'iw'
CurrentDmesg: [ 88.736220] init: plymouth-
Date: Fri Sep 19 14:31:20 2014
HibernationDevice: RESUME=
Lsusb:
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
ProcEnviron:
TERM=screen
PATH=(custom, no user)
XDG_RUNTIME_
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 radeondrmfb
ProcKernelCmdLine: root=UUID=
RelatedPackageV
linux-
linux-
linux-firmware 1.127.6
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
Related branches
- curtin developers: Pending requested 2015-06-04
-
Diff: 141 lines (+95/-0)3 files modifiedcurtin/block/__init__.py (+53/-0)
curtin/commands/block_meta.py (+5/-0)
curtin/commands/curthooks.py (+37/-0)
| Scott Moser (smoser) wrote : | #1 |
| Changed in linux (Ubuntu): | |
| status: | New → Confirmed |
| Changed in linux (Ubuntu): | |
| importance: | Undecided → Medium |
It looks like you have a couple of RAID controllers in the machine. Do you have them configured with and volumes, etc?
0003:04:00.0 RAID bus controller [0104]: IBM PCI-E IPR SAS Adapter (ASIC) [1014:034a] (rev 01)
Subsystem: IBM PCIe3 x8 Cache SAS RAID Internal Adapter 6Gb (57D8) [1014:03fe]
Subsystem: IBM PCIe3 x8 Cache SAS RAID Internal Adapter 6Gb (57D8) [1014:03fe]
| tags: | added: kernel-da-key |
| Scott Moser (smoser) wrote : | #4 |
Joseph,
I honestly don't know how the systems are set up. I suspect you're right, that there are 2 physical disks raided together.
I don't think that changes the bug at all though.
The kernel should only show me one of those devices, right? If I configured hardware raid to show 2 disks as 1 in raid 1 I certainly wouldn't expect to have to know that from user-space (I'm kind of surprised the *kernel* ever see the individual disks there).
| Anton Blanchard (anton-samba) wrote : | #5 |
This is a multipath issue. The box has two cards with redundant paths to each disk.
Multipath is meant to create /dev/mapper/mpath* devices, and the installer should use them instead of the underlying /dev/sd* devices.
| Changed in linux (Ubuntu): | |
| status: | Confirmed → Invalid |
| Stefan Bader (smb) wrote : | #6 |
Quickly reading over the report I would think this is a result of the installer bug I reported as bug #1447167. The problem is that if you configure a system for multipath, it will present each path as an independent drive. And that causes a lot of confusion when a system has data on those and comes up without the multipath tools installed. To the OS this looks like multiple disks having the same filesystem UUIDs which conflict when creating disk-by/... links in /dev. Even worse LVM finds two PVs with the same ID and bails (iirc).
I theory (with a lot knowledge and care) it is possible to install single-path and convert the whole system to multipath later. But frankly I think when the installer was told to enable multipath detect, it should automatically install the tools.
| Scott Moser (smoser) wrote : | #7 |
Related/possibly duplicate bug is bug 1447167.
| Launchpad Janitor (janitor) wrote : | #8 |
Status changed to 'Confirmed' because the bug affects multiple users.
| Changed in curtin (Ubuntu): | |
| status: | New → Confirmed |
| Changed in multipath-tools (Ubuntu): | |
| status: | New → Confirmed |
| Jeff Lane (bladernr) wrote : | #10 |
I'm setting this to High for now... I really think this is critical as it is gating the PowerNV certification work.
| Changed in multipath-tools (Ubuntu): | |
| importance: | Undecided → High |
| Changed in curtin (Ubuntu): | |
| importance: | Undecided → High |
| importance: | High → Critical |
| Changed in multipath-tools (Ubuntu): | |
| importance: | High → Critical |
| Jeff Lane (bladernr) wrote : | #11 |
Actually, on advice of Andy C, lets go ahead and consider this critical... this cert needs to be completed as soon as possible and we can't do that without multipathing working properly.
| Scott Moser (smoser) wrote : | #12 |
Ok, so Oleg and I were looking at a solution, and figured we'd give the "maybe it just works" path a try.
I installed a system, it installed onto /dev/sda (as this bug shows).
I then 'apt-get install multipath-
I rebooted, with the 'root=LABEL=
The multipath-
| if [ -x /sbin/kpartx -a -x /sbin/dmsetup ]; then
| /sbin/dmsetup ls --target multipath --exec "/sbin/kpartx -a -p -part" >/dev/null
| fi
That looks all well and good.
In the initramfs I did some debugging:
(initramfs) /sbin/dmsetup ls --target multipath
1IBM IPR-0 5EC2A900000000A0 (252, 1)
1IBM IPR-0 5EC2A90000000020 (252, 2)
1IBM IPR-0 5EC2A90000000080 (252, 5)
1IBM IPR-0 5EC2A90000000060 (252, 3)
1IBM IPR-0 5EC2A900000000C0 (252, 4)
1IBM IPR-0 5EC2A90000000040 (252, 0)
And then further debug, I find that dmsetup is calling kpartx seemingly correctly, but kpartx is I think messing up.
I was able to fix this by booting with:
root=
Then doing:
(initramfs) cat /tmp/myprog
#!/bin/sh
tgt="$1"
fixed=$(echo "$tgt" | sed 's, ,\\x20,g')
{
echo "input: '$tgt'"
echo "fixed: '$fixed'"
echo /sbin/kpartx -a -v -p -part "$fixed"
/sbin/kpartx -a -v -p -part "$fixed"
ret=$?
echo $ret
} >> /tmp/log 2>&1
exit $ret
(initramfs) rm /tmp/log
(initramfs) /sbin/dmsetup ls --target multipath --exec "/bin/myprog"
(initramfs) cat /tmp/log
input: '/dev/mapper/1IBM IPR-0 5EC2A900000000A0'
fixed: '/dev/mapper/
/sbin/kpartx -a -v -p -part /dev/mapper/
0
input: '/dev/mapper/1IBM IPR-0 5EC2A90000000020'
fixed: '/dev/mapper/
/sbin/kpartx -a -v -p -part /dev/mapper/
0
input: '/dev/mapper/1IBM IPR-0 5EC2A90000000080'
fixed: '/dev/mapper/
/sbin/kpartx -a -v -p -part /dev/mapper/
add map 1IBM IPR-0 5EC2A9000000008
add map 1IBM IPR-0 5EC2A9000000008
0
input: '/dev/mapper/1IBM IPR-0 5EC2A90000000060'
fixed: '/dev/mapper/
/sbin/kpartx -a -v -p -part /dev/mapper/
0
input: '/dev/mapper/1IBM IPR-0 5EC2A900000000C0'
fixed: '/dev/mapper/
/sbin/kpartx -a -v -p -part /dev/mapper/
| Scott Moser (smoser) wrote : | #13 |
this script attempts to do the following for
img=working.img device=
img=broken.img device='/dev/broken 0'
- create a file $img
- partition it with sfdisk
- losetup a device pointing to $img
- kpartx -a -v -p -part $LODEV
run the script and you can see, it will work for 'working0' and fail for 'broken 0'
$ sudo ./show
=== img=working.img devname=
add map working0-part1 (252:0): 0 202752 linear /dev/working0 2048
=== 0 ==
brw-rw---- 3 root disk 7, 1 Jun 3 20:46 /dev/working0
=== img=broken.img devname=/dev/broken 0 lodev=/dev/loop0 ===
device-mapper: reload ioctl on broken\x200-part1 failed: Invalid argument
create/reload failed on broken 0-part1
add map broken 0-part1 (0:0): 0 202752 linear /dev/broken 0 2048
=== 1 ==
brw-rw---- 2 root disk 7, 0 Jun 3 20:46 /dev/broken 0
failed kpartx to /dev/broken 0
that is basically the same way the multipath-boot setup fails.
| Scott Moser (smoser) wrote : | #14 |
verified the attached script also fails on both trusty and wily
trusty: 0.4.9-3ubuntu7.2
wily: 0.4.9-3ubuntu12
| Scott Moser (smoser) wrote : | #15 |
also just now verified that it is broken in debian's unstable version also (0.5.0-7)
| Jeff Lane (bladernr) wrote : | #16 |
Sorry for the vague comment earlier. What I meant was that, from a certification point of view, I can't certify a system that ships by default with dual RAID cards in a multipath configuration when Multipathing in Ubuntu doesn't work. That leads to a scenario where, out of the box on a fresh install you can destroy your own root filesystem by formatting a drive that you don't realize is actually your root fs...
In other words, it is very easy to destroy data accidentally if you're unaware that sde and sda are the same physical device, as you would reasonably expect those to be separate devices.
| description: | updated |
| Scott Moser (smoser) wrote : | #17 |
OK, so for trivial workaround fix, you can do:
a.) install with curtin as you have
b.) sudo apt-get install multipath-
c.) printf "defaults {\n\tuser_
d.) sudo update-initramfs -u -k all
e.) sudo reboot
You'll come back up with multipath as the root device.
Oleg is working on a merge proposal for curtin that will do that by default.
When the fix for bug 1432062 lands, then we wont even specifically need the user_friendly_names patch.
| Mike Rushton (leftyfb) wrote : | #18 |
I tried following the above workaround on the IBM Power 8 twice on new deployments. Attached is the log.
| Launchpad Janitor (janitor) wrote : | #19 |
This bug was fixed in the package curtin - 0.1.0~bzr213-
---------------
curtin (0.1.0~
* New upstream snapshot.
* retry apt-get update to avoid transient failures (LP: #1403133)
* detect and handle multipath devices (LP: #1371634)
* udevadm settle before unmounting target's /dev (LP: #1462139)
* doc/ improved developer doc and tools using maas images for test
* use --no-nvram option to grub-install if available (LP: #1311827)
-- Scott Moser <email address hidden> Fri, 05 Jun 2015 15:06:31 -0400
| Changed in curtin (Ubuntu): | |
| status: | Confirmed → Fix Released |
| Scott Moser (smoser) wrote : | #20 |
Mike, Oleg, others.
I've opened bug 1462530 to address the errors I'm seeing like those Mike pointed at.
| description: | updated |
| Mike Rushton (leftyfb) wrote : | #21 |
I have tried the latest curtin-common and python-curtin from the wily repositories running on 14.04.2:
#leftyfb@
curtin-common:
Installed: 0.1.0~bzr214-
Candidate: 0.1.0~bzr214-
Version table:
*** 0.1.0~bzr214-
500 http://
100 /var/lib/
0.
500 http://
0.
500 http://
0.
500 http://
#leftyfb@
python-curtin:
Installed: 0.1.0~bzr214-
Candidate: 0.1.0~bzr214-
Version table:
*** 0.1.0~bzr214-
500 http://
100 /var/lib/
0.
500 http://
0.
500 http://
0.
500 http://
#leftyfb@
maas:
Installed: 1.7.5+bzr3369-
Candidate: 1.7.5+bzr3369-
Version table:
1.
500 http://
*** 1.7.5+bzr3369-
500 http://
100 /var/lib/
1.
500 http://
1.
500 http://
1.
500 http://
The deployment never fully boots up. It never gets to a console or allows an ssh connection. Attached is a log of the deployment and first boot process to debug.
| Scott Moser (smoser) wrote : | #22 |
Mike,
Please move issues with multipath (like are shown in that log) to bug 1462530.
Stefan had actually asked for almost exactly what you captured.
This bug is "fixed" as we're now booting into system correctly with multipath enabled from installation under curtin.
I'll copy your comment and attachment there.
| no longer affects: | linux (Ubuntu Trusty) |
| summary: |
- block devices appear twice + block devices appear twice [install does not use multipath] |
| no longer affects: | multipath-tools (Ubuntu Trusty) |
| no longer affects: | multipath-tools (Ubuntu Vivid) |
| Changed in curtin: | |
| status: | New → Fix Committed |
| importance: | Undecided → Medium |
| importance: | Medium → High |
| no longer affects: | linux (Ubuntu) |
| Scott Moser (smoser) wrote : | #23 |
this is a set of logs collected by
https:/
I'm posting them here to show that this seems pretty functional and reliable at the moment. It installed trusty-hwe-u, trusty-hwe-v, vivid, wily in a loop 15 times each. It did fail on runs 16-20, but I believe the ipmi serial died, and since check for success was based on that, it marked them as failed and I did not get logs.
| Changed in curtin (Ubuntu Trusty): | |
| status: | New → Confirmed |
| Changed in curtin (Ubuntu Vivid): | |
| status: | New → Confirmed |
| Changed in curtin (Ubuntu Trusty): | |
| importance: | Undecided → High |
| Changed in curtin (Ubuntu Vivid): | |
| importance: | Undecided → High |
| no longer affects: | linux (Ubuntu Vivid) |
| no longer affects: | multipath-tools (Ubuntu) |
| description: | updated |
| Changed in curtin (Ubuntu Utopic): | |
| status: | New → Won't Fix |
| description: | updated |
| Scott Moser (smoser) wrote : | #24 |
Some status here.
I believe this is fixed in curtin trunk. Curtin should properly identify systems with multipath devices and install multipath-
curtin at > revision 220 should be good.
That is available in:
wily archive: https:/
maas experimental ppa for trusty utopic vivid: https:/
maas testing ppa for trusty utopic vivid: https:/
The things left to get this functional in all supported places are:
a.) copy to maas stable ppa
b.) SRU to trusty and vivid (i'm not planning on bothering with utopic).
| description: | updated |
Hello Scott, or anyone else affected,
Accepted curtin into trusty-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
| Changed in curtin (Ubuntu Trusty): | |
| status: | Confirmed → Fix Committed |
| tags: | added: verification-needed |
| Chris J Arges (arges) wrote : | #26 |
Hello Scott, or anyone else affected,
Accepted curtin into vivid-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
| Changed in curtin (Ubuntu Vivid): | |
| status: | Confirmed → Fix Committed |
| Scott Moser (smoser) wrote : | #27 |
I've tested curtin installations on both multipath systems and non-multipath systems.
marking this as verfication-done.
| tags: |
added: verification-done removed: verification-needed |
| Launchpad Janitor (janitor) wrote : | #28 |
This bug was fixed in the package curtin - 0.1.0~bzr221-
---------------
curtin (0.1.0~
* New upstream snapshot.
- support installation to multipath devices. (LP: #1371634)
- know that kernel version 4.2.0 maps to linux-generic-
- support install to arm64 systems that use UEFI for boot (LP: #1447834)
- fix remaining usage of 'lsblk --out' rather than 'lsblk --output'
(LP: #1386275)
- retry 'apt-get update' on failure to avoid transient failures
(LP: #1403133)
- run udevadm settle before unmounting /dev in a target to avoid transient
failures (LP: #1462139)
- fixes and additions to tools used in development.
- Add --no-nvram to the grub-install command for UEFI. (LP: #1311827)
- avoid race condition and transient failure due busy device in mkfs
(LP: #1443542)
- improvements to device and partition naming code which allow installation
devices with HP cciss smart array drives(LP: #1401190, #1263181)
- do not consider devices < 1G as installable targets
* debian/
-- Scott Moser <email address hidden> Wed, 24 Jun 2015 14:31:14 -0400
| Changed in curtin (Ubuntu Trusty): | |
| status: | Fix Committed → Fix Released |
| Chris J Arges (arges) wrote : Update Released | #29 |
The verification of the Stable Release Update for curtin has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.
| Launchpad Janitor (janitor) wrote : | #30 |
This bug was fixed in the package curtin - 0.1.0~bzr221-
---------------
curtin (0.1.0~
* New upstream snapshot.
- support installation to multipath devices. (LP: #1371634)
- know that kernel version 4.2.0 maps to linux-generic-
- support install to arm64 systems that use UEFI for boot (LP: #1447834)
- fix remaining usage of 'lsblk --out' rather than 'lsblk --output'
(LP: #1386275)
- retry 'apt-get update' on failure to avoid transient failures
(LP: #1403133)
- run udevadm settle before unmounting /dev in a target to avoid transient
failures (LP: #1462139)
- fixes and additions to tools used in development.
- Add --no-nvram to the grub-install command for UEFI. (LP: #1311827)
- avoid race condition and transient failure due busy device in mkfs
(LP: #1443542)
- improvements to device and partition naming code which allow installation
devices with HP cciss smart array drives(LP: #1401190, #1263181)
- do not consider devices < 1G as installable targets
* debian/
-- Scott Moser <email address hidden> Wed, 24 Jun 2015 16:12:59 -0400
| Changed in curtin (Ubuntu Vivid): | |
| status: | Fix Committed → Fix Released |
This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New
Thank you.
| Changed in curtin: | |
| status: | Fix Committed → Fix Released |
This change was made by a bot.