[PowerVM] Ubuntu 16.04 does not install bootloader on multiple PReP partitions in Software RAID1 configuration

Bug #1550448 reported by bugproxy on 2016-02-26
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
grub-installer (Ubuntu)
High
Mathieu Trudel-Lapierre
Trusty
High
Unassigned

Bug Description

== Comment: #0 - Heitor Ricardo Alves de Siqueira - 2016-02-19 11:37:33 ==
---Problem Description---
When installing Ubuntu 16.04 to a RAID1 powerVM configuration, grub only installs to the first disk. The second disk will not have a valid PReP partition, rendering the system unbootable if the first disk fails in any way.

We are currently seeing this on a RAID1 install to two NVMe adapters.

---uname output---
Linux tul131p1 4.4.0-6-generic #21-Ubuntu SMP Tue Feb 16 20:31:37 UTC 2016 ppc64le GNU/Linux

Machine Type = 8286-42A lpar

---boot type---
kexec vmlinuz/initrd

---Kernel cmdline used to launch install---
default cmdline

---Install repository type---
Internet repository

---Install repository Location---
ports.ubuntu.com

---Point of failure---
Problem during post-install (stage 2) configuration or other problem seen after reboot

== Comment: #1 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 05:31:43 ==
Currently:
1) grub-install lists all PReP partitions in the system
2) and prefers a PReP partition on the same disk as the root/boot filesystem.
3) wipe it and install the bootloader code in it.

Idea:
0) keep that as-is / backward compatible, as it works OK so far, but slightly change a particular case
1) grub-install lists all PReP partitions in the system
2) and prefers a PReP partition on the same disk as the root/boot filesystem.
3) but if the root/boot filesystem is a MD device,
4) find its component disks
5) check which ones have a PReP partition
6) wipe it and install bootloader code in it.

== Comment: #4 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 08:54:04 ==
Hi Canonical,

This bug is a follow on to LP bug 1487365 (LTC BZ bug 129040) in the area of PReP partitions and Software RAID installations.

@taco-screen-team
I'd suggest for the bug assignee/owner to be @mathieu-tl on this bug too.

== Comment: #5 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 09:00:23 ==
This patch for Xenial adds support in grub-installer for handling multiple PReP partitions on component devices of MD/dmadm devices.
It's backward compatible w/ the existing handling of other device types.

== Comment: #6 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 09:01:01 ==
Debugging logs from the installation:

Syslog:
 ~ # grep grub-install /var/log/syslog
 Feb 26 11:09:15 anna[7389]: DEBUG: retrieving grub-installer 1.128ubuntu2
 Feb 26 11:57:48 main-menu[1776]: INFO: Menu item 'grub-installer' selected
 Feb 26 11:57:48 grub-installer: info: architecture: ppc64el/chrp_ibm
 Feb 26 11:57:52 grub-installer: info: Identified partition label for /dev/md0p2: gpt
 Feb 26 11:57:52 grub-installer: info: Wiping PReP partition /dev/sda1
 Feb 26 11:57:52 grub-installer: info: Wiping PReP partition /dev/sdb1
 Feb 26 11:57:52 grub-installer: info: Wiping PReP partition /dev/sdc1
 Feb 26 11:57:53 grub-installer: info: Installing grub on '/dev/sdc1'
 Feb 26 11:57:53 grub-installer: info: grub-install does not support --no-floppy
 Feb 26 11:57:53 grub-installer: info: Running chroot /target grub-install --force "/dev/sda1"
 Feb 26 11:57:54 grub-installer: Installing for powerpc-ieee1275 platform.
 Feb 26 11:58:02 grub-installer: Installation finished. No error reported.
 Feb 26 11:58:02 grub-installer: info: grub-install ran successfully
 Feb 26 11:58:02 grub-installer: info: grub-install does not support --no-floppy
 Feb 26 11:58:02 grub-installer: info: Running chroot /target grub-install --force "/dev/sdb1"
 Feb 26 11:58:02 grub-installer: Installing for powerpc-ieee1275 platform.
 Feb 26 11:58:11 grub-installer: Installation finished. No error reported.
 Feb 26 11:58:11 grub-installer: info: grub-install ran successfully
 Feb 26 11:58:11 grub-installer: info: grub-install does not support --no-floppy
 Feb 26 11:58:11 grub-installer: info: Running chroot /target grub-install --force "/dev/sdc1"
 Feb 26 11:58:11 grub-installer: Installing for powerpc-ieee1275 platform.
 Feb 26 11:58:20 grub-installer: Installation finished. No error reported.
 Feb 26 11:58:20 grub-installer: info: grub-install ran successfully

Shell script tracing:

For the shell tracing, add the following to /usr/bin/grub-install:
  exec 2>/grub.debug
  set -x

 ~ # cat /grub.debug
 <...>
 + wipe_bootdevs=
 + /usr/lib/grub-installer/prep-bootdev
 + wipe_bootdev=/dev/sda1
 + /usr/lib/grub-installer/prep-bootdev -l
 + [ /dev/sda = /dev/md ]
 + grep -q /dev/md[0-9]\+
 + echo /dev/md0
 + grep /dev/sda
 + grep -o /dev/[sv]d[a-z]\+
 + mdadm --detail --verbose /dev/md0
 + wipe_bootdevs= /dev/sda1
 + [ /dev/sdb = /dev/md ]
 + grep -q /dev/md[0-9]\+
 + echo /dev/md0
 + grep /dev/sdb
 + grep -o /dev/[sv]d[a-z]\+
 + mdadm --detail --verbose /dev/md0
 + wipe_bootdevs= /dev/sda1 /dev/sdb1
 + [ /dev/sdc = /dev/md ]
 + grep -q /dev/md[0-9]\+
 + echo /dev/md0
 + grep /dev/sdc
 + grep -o /dev/[sv]d[a-z]\+
 + mdadm --detail --verbose /dev/md0
 + wipe_bootdevs= /dev/sda1 /dev/sdb1 /dev/sdc1
 + unset prep_p
 + [ -z /dev/sda1 /dev/sdb1 /dev/sdc1 ]
 + [ -n /dev/sda1 ]
 + info Wiping PReP partition /dev/sda1
 + log info: Wiping PReP partition /dev/sda1
 + logger -t grub-installer info: Wiping PReP partition /dev/sda1
 + blockdev --getsz /dev/sda1
 + log-output -t grub-installer dd if=/dev/zero of=/dev/sda1 bs=512 count=14336
 + [ -n /dev/sdb1 ]
 + info Wiping PReP partition /dev/sdb1
 + log info: Wiping PReP partition /dev/sdb1
 + logger -t grub-installer info: Wiping PReP partition /dev/sdb1
 + blockdev --getsz /dev/sdb1
 + log-output -t grub-installer dd if=/dev/zero of=/dev/sdb1 bs=512 count=14336
 + [ -n /dev/sdc1 ]
 + info Wiping PReP partition /dev/sdc1
 + log info: Wiping PReP partition /dev/sdc1
 + logger -t grub-installer info: Wiping PReP partition /dev/sdc1
 + blockdev --getsz /dev/sdc1
 + log-output -t grub-installer dd if=/dev/zero of=/dev/sdc1 bs=512 count=14336
 <...>
 + bootdevs=/dev/sdc1
 + bootdevs= /dev/sda1 /dev/sdb1 /dev/sdc1
 <...>
 + info Running chroot /target grub-install --force "/dev/sda1"
 + log info: Running chroot /target grub-install --force "/dev/sda1"
 + logger -t grub-installer info: Running chroot /target grub-install --force "/dev/sda1"
 + log-output -t grub-installer chroot /target grub-install --force /dev/sda1
 + [ 0 = 0 ]
 + info grub-install ran successfully
 + log info: grub-install ran successfully
 + logger -t grub-installer info: grub-install ran successfully
 <...>
 + info Running chroot /target grub-install --force "/dev/sdb1"
 + log info: Running chroot /target grub-install --force "/dev/sdb1"
 + logger -t grub-installer info: Running chroot /target grub-install --force "/dev/sdb1"
 + log-output -t grub-installer chroot /target grub-install --force /dev/sdb1
 + [ 0 = 0 ]
 + info grub-install ran successfully
 + log info: grub-install ran successfully
 + logger -t grub-installer info: grub-install ran successfully
 <...>
 + info Running chroot /target grub-install --force "/dev/sdc1"
 + log info: Running chroot /target grub-install --force "/dev/sdc1"
 + logger -t grub-installer info: Running chroot /target grub-install --force "/dev/sdc1"
 + log-output -t grub-installer chroot /target grub-install --force /dev/sdc1
 + [ 0 = 0 ]
 + info grub-install ran successfully
 + log info: grub-install ran successfully
 + logger -t grub-installer info: grub-install ran successfully
 <...>

== Comment: #7 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 09:01:29 ==
Test scenario:

3 individual disks (2 active, 1 spare)

$ for i in 1 2 3; do qemu-img create -f qcow2 disk$i.qcow2 8G; done
Formatting 'disk1.qcow2', fmt=qcow2 size=8589934592 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
Formatting 'disk2.qcow2', fmt=qcow2 size=8589934592 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
Formatting 'disk3.qcow2', fmt=qcow2 size=8589934592 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

$ qemu-system-ppc64 -enable-kvm \
    -nographic -nodefaults \
    -monitor stdio -serial pty \
    -M pseries -smp 1,cores=1,threads=1 -m 8G \
    \
    -device spapr-vscsi \
    -drive file=disk1.qcow2 \
    -drive file=disk2.qcow2 \
    -drive file=disk3.qcow2 \
    -drive media=cdrom,file=xenial-server-ppc64el.iso \
    \
    -device spapr-vlan,netdev=net0,mac=4C:45:42:45:02:03 \
    -netdev bridge,id=net0,br=br0 \
    -S

1) On the 'Partition disks' dialog, select 'Go back', and 'Execute a shell'

2) Check the 'grub-installer' script is present

 ~ # ls -l /usr/bin/grub-installer
 -rwxr-xr-x 1 root root 44452 Jan 4 14:05 /usr/bin/grub-installer

3) Download and install the test package

 ~ # wget http://ausgsa.ibm.com/~mauricfo/public/bugs/bz137572/v1/grub-installer_1.128ubuntu2multiprep1_ppc64el.udeb
 ~ # udpkg -i *.udeb
 <... ignore the messages ...>

4) Verify the patch is present

 ~ # grep MD/mdadm /usr/bin/grub-installer
  # On MD/mdadm devices, each component device may have a PReP partition.
          # On MD/mdadm devices, each component device may have a PReP partition.

5) Return to the installation.

 ~ # exit

6) Proceed normally with a Software RAID based partitioning
   *with a PReP partition per component device.*

7) On the 'Installation finished' prompt (no errors expected), select 'Go Back', and 'Execute a shell'

8) Verify all the PReP partitions on the component devices
   were covered by grub-installer.
   (e.g., notice the 'Wiping PReP partition' and 'chroot /target grub-install' messages)

 ~ # grep -i grub-install /var/log/syslog
 Feb 26 13:19:45 anna[7383]: DEBUG: retrieving grub-installer 1.128ubuntu2
 Feb 26 13:22:22 grub-installer: info: architecture: ppc64el/chrp_ibm
 Feb 26 13:34:36 main-menu[1769]: INFO: Menu item 'grub-installer' selected
 Feb 26 13:34:36 grub-installer: info: architecture: ppc64el/chrp_ibm
 Feb 26 13:34:38 grub-installer: info: Identified partition label for /dev/md0p2: gpt
 Feb 26 13:34:39 grub-installer: info: Wiping PReP partition /dev/sda1
 Feb 26 13:34:39 grub-installer: info: Wiping PReP partition /dev/sdb1
 Feb 26 13:34:39 grub-installer: info: Wiping PReP partition /dev/sdc1
 Feb 26 13:34:55 grub-installer: info: Installing grub on '/dev/sdc1'
 Feb 26 13:34:55 grub-installer: info: grub-install does not support --no-floppy
 Feb 26 13:34:55 grub-installer: info: Running chroot /target grub-install --force "/dev/sda1"
 Feb 26 13:34:55 grub-installer: Installing for powerpc-ieee1275 platform.
 Feb 26 13:35:04 grub-installer: Installation finished. No error reported.
 Feb 26 13:35:04 grub-installer: info: grub-install ran successfully
 Feb 26 13:35:04 grub-installer: info: grub-install does not support --no-floppy
 Feb 26 13:35:04 grub-installer: info: Running chroot /target grub-install --force "/dev/sdb1"
 Feb 26 13:35:04 grub-installer: Installing for powerpc-ieee1275 platform.
 Feb 26 13:35:13 grub-installer: Installation finished. No error reported.
 Feb 26 13:35:13 grub-installer: info: grub-install ran successfully
 Feb 26 13:35:13 grub-installer: info: grub-install does not support --no-floppy
 Feb 26 13:35:13 grub-installer: info: Running chroot /target grub-install --force "/dev/sdc1"
 Feb 26 13:35:13 grub-installer: Installing for powerpc-ieee1275 platform.
 Feb 26 13:35:23 grub-installer: Installation finished. No error reported.
 Feb 26 13:35:23 grub-installer: info: grub-install ran successfully

9) Return to the installation, and select 'Finish the installation'.

 ~ # exit

Boot tests:

During boot, the SLOF firmware could successfully detect and load the GRUB2 code from
the PReP partition of *each individual disk* (test: run guest with either disk1, disk2, or disk3.qcow2).

 Trying to load: from: disk ...
 No DOS disk-label found.
   Successfully loaded
 error: no suitable video mode found.
 <...>
                  GNU GRUB version 2.02~beta2-36
 <...>

The only boot failure case is with only the spare disk (disk3.qcow2), since GRUB2 can't load
the remaining of its code from one of the active devices.
(notice it's a GRUB2 error message, so PReP was correctly detected and GRUB2 loaded from it.)

 Trying to load: from: disk ...
 No DOS disk-label found.
   Successfully loaded
 error: disk `mduuid/b3069040df508c48c7c8ddc6a19a6bc0,2' not found.
 Entering rescue mode...
 grub rescue>

== Comment: #8 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 09:03:00 ==
Hi Heitor,

Can you please check it works correctly on your PowerVM environment?

I verified this works correctly on a qemu-kvm guest.

Thanks

== Comment: #9 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 10:41:01 ==
Hi Canonical,

The patch is verified for 14.04 too. Please apply on 14.04 as well.

Thanks

== Comment: #10 - Mauricio Faria De Oliveira <email address hidden> - 2016-02-26 12:30:05 ==
(In reply to comment #9)
> The patch is verified for 14.04 too. Please apply on 14.04 as well.

Feb 26 14:17:48 anna[6955]: DEBUG: retrieving grub-installer 1.78ubuntu20.3
Feb 26 15:31:21 main-menu[1787]: INFO: Menu item 'grub-installer' selected
Feb 26 15:31:21 grub-installer: info: architecture: ppc64el/chrp_ibm
Feb 26 15:31:34 grub-installer: info: Identified partition label for /dev/md0p2: loop
Feb 26 15:31:34 grub-installer: info: Wiping PReP partition /dev/sda1
Feb 26 15:31:34 grub-installer: info: Wiping PReP partition /dev/sdb1
Feb 26 15:31:34 grub-installer: info: Wiping PReP partition /dev/sdc1
Feb 26 15:31:50 grub-installer: info: Installing grub on '/dev/sdc1'
Feb 26 15:31:50 grub-installer: info: grub-install does not support --no-floppy
Feb 26 15:31:50 grub-installer: info: Wiping PReP partition /dev/sda1
Feb 26 15:31:50 grub-installer: info: Running chroot /target grub-install --force "/dev/sda1"
Feb 26 15:31:50 grub-installer: Installing for powerpc-ieee1275 platform.
Feb 26 15:31:59 grub-installer: Installation finished. No error reported.
Feb 26 15:31:59 grub-installer: info: grub-install ran successfully
Feb 26 15:31:59 grub-installer: info: grub-install does not support --no-floppy
Feb 26 15:31:59 grub-installer: info: Wiping PReP partition /dev/sdb1
Feb 26 15:31:59 grub-installer: info: Running chroot /target grub-install --force "/dev/sdb1"
Feb 26 15:31:59 grub-installer: Installing for powerpc-ieee1275 platform.
Feb 26 15:32:08 grub-installer: Installation finished. No error reported.
Feb 26 15:32:08 grub-installer: info: grub-install ran successfully
Feb 26 15:32:08 grub-installer: info: grub-install does not support --no-floppy
Feb 26 15:32:08 grub-installer: info: Wiping PReP partition /dev/sdc1
Feb 26 15:32:08 grub-installer: info: Running chroot /target grub-install --force "/dev/sdc1"
Feb 26 15:32:08 grub-installer: Installing for powerpc-ieee1275 platform.
Feb 26 15:32:16 grub-installer: Installation finished. No error reported.
Feb 26 15:32:16 grub-installer: info: grub-install ran successfully

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-137572 severity-high targetmilestone-inin1604
Changed in ubuntu:
assignee: nobody → Taco Screen team (taco-screen-team)
Kevin W. Rudd (kevinr) on 2016-02-26
affects: ubuntu → debian-installer (Ubuntu)
affects: debian-installer (Ubuntu) → grub-installer (Ubuntu)
Changed in grub-installer (Ubuntu):
assignee: Taco Screen team (taco-screen-team) → Mathieu Trudel-Lapierre (mathieu-tl)
importance: Undecided → High
status: New → In Progress

One slight change to the patch.
It worked for sd/vd disks, but not for NVMe :)
Just fixed the regex and tested on a system w/ 2 NVMe disks.

Change to this line only:
  grep -o '/dev/\([sv]d[a-z]\+\|nvme[0-9]\+n[0-9]\+p\)'

bugproxy (bugproxy) wrote :

Default Comment by Bridge

The attachment "grub-installer_multi-prep.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub-installer - 1.128ubuntu3

---------------
grub-installer (1.128ubuntu3) xenial; urgency=medium

  * grub-installer [ppc64el]: support multiple PReP partitions on component
    devices of a MD/mdadm device. (LP: #1550448)

 -- Mauricio Faria de Oliveira <email address hidden> Fri, 26 Feb 2016 15:14:21 -0500

Changed in grub-installer (Ubuntu):
status: In Progress → Fix Released
Changed in grub-installer (Ubuntu Trusty):
importance: Undecided → High

Hi @mathieu-tl

Unfortunately the patch that made the package is that w/out the NVMe regex (comment #2).
I see that bugproxy reposted the old patch.. heck (no idea why). Apologies if that caused the confusion.

Can you please adjust the grep regex for that in comment #2?

Thanks!

bugproxy (bugproxy) wrote :

Default Comment by Bridge

------- Comment on attachment From <email address hidden> 2016-03-04 16:28 EDT-------

Hi @mathieu-tl

Er.. this bug needs another fix.. :) Patch attached.
Problem is.. the NVMe expression needs a 'p' suffix.

Apologies. Should I have I tested this on the real hardware before submitting, I wouldn't have give you 3x the trouble.

To make up for it, I cleaned the regex into something more general, which closely matches the compared string (only a removal of trailing partition numbers -- "${prep_p%[0-9]*}").

The changelog version is "1.128ubuntu4multiprepv31" -- in case bugproxy messes up again (I'm trying a few things to keep it quiet this time.. let's see if it works).

Sorry. Thanks.

------- Comment From <email address hidden> 2016-03-04 16:29 EDT-------
Here's the validation on the real hardware this time.

~ # mdadm --detail --verbose /dev/md0 | grep -o '/dev/.*'
/dev/md0:
/dev/nvme0n1p2
/dev/nvme1n1p2

~ # mdadm --detail --verbose /dev/md0 | sed -n 's:.*\(/dev/.*\)[0-9]\+$:\1:p'
/dev/nvme0n1p
/dev/nvme1n1p

Mar 4 20:56:10 main-menu[301]: INFO: Menu item 'grub-installer' selected
Mar 4 20:56:10 grub-installer: info: architecture: ppc64el/chrp_ibm
Mar 4 20:56:11 grub-installer: info: Identified partition label for /dev/md0: loop
Mar 4 20:56:11 grub-installer: info: Wiping PReP partition /dev/nvme0n1p1
Mar 4 20:56:11 grub-installer: info: Wiping PReP partition /dev/nvme1n1p1
Mar 4 20:56:12 grub-installer: info: Installing grub on '/dev/nvme1n1p1'
Mar 4 20:56:12 grub-installer: info: grub-install does not support --no-floppy
Mar 4 20:56:12 grub-installer: info: Running chroot /target grub-install --force "/dev/nvme0n1p1"
Mar 4 20:56:12 grub-installer: Installing for powerpc-ieee1275 platform.
Mar 4 20:56:12 grub-installer: Installation finished. No error reported.
Mar 4 20:56:12 grub-installer: info: grub-install ran successfully
Mar 4 20:56:12 grub-installer: info: grub-install does not support --no-floppy
Mar 4 20:56:12 grub-installer: info: Running chroot /target grub-install --force "/dev/nvme1n1p1"
Mar 4 20:56:12 grub-installer: Installing for powerpc-ieee1275 platform.
Mar 4 20:56:13 grub-installer: Installation finished. No error reported.
Mar 4 20:56:13 grub-installer: info: grub-install ran successfully

if echo $disc_offered | grep -q '/dev/md[0-9]\+'; then
if mdadm --detail --verbose $disc_offered \
| sed -n 's:.*\(/dev/.*\)[0-9]\+$:\1:p' \
| grep "${prep_p%[0-9]*}"; then
wipe_bootdevs="$wipe_bootdevs $prep_p"
fi
fi

tags: removed: bugnameltc-137572 patch severity-high

Default Comment by Bridge

------- Comment on attachment From <email address hidden> 2016-03-04 16:28 EDT-------

Hi @mathieu-tl

Er.. this bug needs another fix.. :) Patch attached.
Problem is.. the NVMe expression needs a 'p' suffix.

Apologies. Should I have I tested this on the real hardware before submitting, I wouldn't have give you 3x the trouble.

To make up for it, I cleaned the regex into something more general, which closely matches the compared string (only a removal of trailing partition numbers -- "${prep_p%[0-9]*}").

The changelog version is "1.128ubuntu4multiprepv31" -- in case bugproxy messes up again (I'm trying a few things to keep it quiet this time.. let's see if it works).

Sorry. Thanks.

------- Comment From <email address hidden> 2016-03-16 13:05 EDT-------
The (hopefully) last fix is available on grub-installer_1.128ubuntu5:

if mdadm --detail --verbose $disc_offered \
| sed -n 's:.*\(/dev/.*[^0-9]\)[0-9]\+$:\1:p' \

tags: added: bugnameltc-137572 severity-high
Changed in grub-installer (Ubuntu Trusty):
status: New → Triaged
tags: added: patch
Robie Basak (racb) wrote :

Unsubscribing ~ubuntu-sponsors because:

1) I don't see any patch ready for upload to Ubuntu here.

2) Assuming that all that is left is Trusty (from the bug task), there isn't any SRU information provided for an SRU to Trusty, which is required for any SRU. Please see https://wiki.ubuntu.com/StableReleaseUpdates#Procedure for details.

If you're still looking to have something uploaded to Trusty, please do resubscribe ~ubuntu-sponsors when you have these two items ready. Thanks!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers