Grub install failed on disks with software RAID

[Helge Meinhard]

I've got two 2TB drives set up with identical partitions, which are combined into RAID1 arrays (five mirrors across the two disks).
On a free RAID1 I've installed Trusty 64bit from the Desktop installer ("try Ubuntu" first, install mdadm, run mdadm --assemble --scan, then start installer). I then wanted grub to be installed onto /dev/sda and /dev/sdb, but both fail.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: ubiquity 2.18.7
ProcVersionSignature: Ubuntu 3.13.0-24.46-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
CasperVersion: 1.340
Date: Fri May 2 13:05:29 2014
InstallCmdLine: noprompt cdrom-detect/try-usb=true file=/cdrom/preseed/ubuntu.seed boot=casper initrd=/casper/initrd.lz quiet splash -- maybe-ubiquity
LiveMediaBuild: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
ProcEnviron:
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: grub-installer
UpgradeStatus: No upgrade log present (probably fresh install)

[Phillip Susi (psusi)]

Can you post the output of sudo mdadm -D /dev/md1, and sudo grub-probe -v -d /dev/md1?

[Helge Meinhard (helge-meinhard)]

Thanks for your swift reply. The machine is in my office; it is
currently running Saucy, I can't reboot Trusty before Monday. Are these
commands run under Saucy (with its root on md0) useful?

Cheers,

Helge

----------------------------------------------------------------------------
   Helge MEINHARD CERN, IT Department, CH-1211 Geneve 23, Switzerland
                       Phone: [phone], Fax: [phone]
                       E-mail: <email address hidden>

On 02/05/14 19:36, Phillip Susi wrote:
> Can you post the output of sudo mdadm -D /dev/md1, and sudo grub-probe
> -v -d /dev/md1?
>

[Helge Meinhard (helge-meinhard)]

Here is the output of the command (run under Ubuntu 13.10 booted from
/dev/md0):

$ sudo mdadm -D /dev/md1
/dev/md1:
         Version : 1.2
   Creation Time : Wed Nov 6 16:18:54 2013
      Raid Level : raid1
      Array Size : 19514240 (18.61 GiB 19.98 GB)
   Used Dev Size : 19514240 (18.61 GiB 19.98 GB)
    Raid Devices : 2
   Total Devices : 2
     Persistence : Superblock is persistent

     Update Time : Mon May 5 07:37:31 2014
           State : clean
  Active Devices : 2
Working Devices : 2
  Failed Devices : 0
   Spare Devices : 0

            Name : <hostname>:1 (local to host <hostname>)
            UUID : 7d38852a:136668be:8d9f540b:7cfc4bcb
          Events : 33

     Number Major Minor RaidDevice State
        0 8 2 0 active sync /dev/sda2
        1 8 18 1 active sync /dev/sdb2

$ sudo grub-probe -v -d /dev/md1
grub-probe: info: cannot open `/boot/grub/device.map': No such file or directory.
grub-probe: info: changing current directory to /dev.
grub-probe: info: changing current directory to vboxusb.
grub-probe: info: changing current directory to 001.
grub-probe: info: changing current directory to 003.
grub-probe: info: changing current directory to dri.
grub-probe: info: changing current directory to snd.
grub-probe: info: changing current directory to by-path.
grub-probe: info: changing current directory to cpu.
grub-probe: info: changing current directory to md.
grub-probe: info: changing current directory to disk.
grub-probe: info: changing current directory to by-uuid.
grub-probe: info: changing current directory to by-id.
grub-probe: info: changing current directory to /dev.
grub-probe: info: changing current directory to vboxusb.
grub-probe: info: changing current directory to 001.
grub-probe: info: changing current directory to 003.
grub-probe: info: changing current directory to dri.
grub-probe: info: changing current directory to snd.
grub-probe: info: changing current directory to by-path.
grub-probe: info: changing current directory to cpu.
grub-probe: info: changing current directory to md.
grub-probe: info: changing current directory to disk.
grub-probe: info: changing current directory to by-uuid.
grub-probe: info: changing current directory to by-id.
grub-probe: info: Looking for /dev/sda2.
grub-probe: info: /dev/sda is a parent of /dev/sda2.
grub-probe: info: /dev/sda2 starts from 39063552.
grub-probe: info: opening the device hostdisk//dev/sda.
grub-probe: info: the size of hostdisk//dev/sda is 3907029168.
grub-probe: info: the size of hostdisk//dev/sda is 3907029168.
grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/sda.
grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/sda.
grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/sda.
grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/sda.
grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/sda.
grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/sda.
grub-probe: info: scanning hostdisk//dev/sda for LDM.
grub-probe: info: no LDM signature found.
grub-probe: info: Scanning for lvm devices on d...

[Phillip Susi (psusi)]

Try this:

sudo -s
mount /dev/md1 /target
for f in proc sys dev dev/pts ; do mount --bind /$f /target/$f ; done
chroot /target
grub-install /dev/sda

I assumed here that /dev/md1 is your root partition; if it is not, change it accordingly.

[Helge Meinhard (helge-meinhard)]

Thanks, did all that, but it doesn't resolve the problem:

 > ...
 > /# grub-install /dev/sda
Installing for i386-pc platform.
(after a few seconds wait)
grub-install: error: disk `md1' not found.

I looked at /target/boot/grub - no grub.cfg. /target/etc/default/grub
and /target/etc/grub.d/* all look unsuspicious. Trying to run
update-grub results in the same 'disk md1 not found' error message
from grub-probe.

Helge

[Phillip Susi (psusi)]

So sudo grub-probe -d /dev/md1 works ( prints "ext2 ) when run from the live cd. Does it work if you run it after the chroot?

[Helge Meinhard (helge-meinhard)]

Yes, indeed - grub-probe -d /dev/md1 worked fine when running from my
previous 13.10 system as well as from the USB stick (Trusty 64bit desktop,
mdadm installed), but not after chroot to the /dev/md1 target.

I found that mdadm was not installed on /dev/md1, I installed it (and
Postfix) by hand, after which grub-install worked apparently fine.

There is one remaining oddity - even though the previous 13.10
installation on /dev/md0 is mounted (on /A), update-grub does not find
its kernel and hence does not add a boot entry for the 13.10 install.
Likewise, when the former was booted, and /dev/md1 was mounted as /B,
the 14.04 entry was not added. This behaviour is different from what I've
seen on other machines using physical partitions rather than md devices.
Is this expected? How can I fix this? Should I submit a dedicated bug
report?

Thanks for all your help!

Cheers,

Helge

----------------------------------------------------------------------------
   Helge MEINHARD CERN, IT Department, CH-1211 Geneve 23, Switzerland
                       Phone: [phone], Fax: [phone]
                       E-mail: <email address hidden>

On 07/05/14 17:39, Phillip Susi wrote:
> So sudo grub-probe -d /dev/md1 works ( prints "ext2 ) when run from the
> live cd. Does it work if you run it after the chroot?
>

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[shane freeder (shane-af-yahoo)]

so, I just hit this issue using a custom install script running from a custom live CD, however I've managed to find where my issue was, just somehow happened to end up finding this page.

The issue was when I ran the grub install command, it would fail with the message that it can't find the md1 device.

I eventually noticed that for some strange reason, mdadm wasn't installed inside the chroot even though I'd told it too. Running apt-get install mdadm fixed the issue for me, so chances are mdadm isn't being installed into the chroot, so this is technically an issue with the ubiquity, I'd say, and not grub.

[Matthew Hall (mhall-9)]

In my case it also failed. I had a precreated series of /dev/md devices I set up using gdisk.

/dev/md0 for swap, /dev/md1 for root (/), and /dev/md2 for home (/home).

The install died during the grub installation. Now, there is no longer an Alternate Ubuntu CD with the Debian Installer (which, sadly, after all these years and all these shiny UIs, still works better than the Ubuntu one does). So it's pretty tough to work around the problem either.

I am not sure where to get all of the error logs from the install because the location of them and most of the behavior of the Ubiquity installed is undocumented as far as I can tell. It seems like it segfaulted on me after the failure. I attached the stuff I could find in /var/log/syslog. But given there's not a lot of decent workarounds I think this is a pretty severe bug... installers shouldn't segfault...

Jul 28 06:33:02 kubuntu kernel: [ 3143.574987] ubiquity[29163]: segfault at 70c0000000c ip 00007f04d7379e88 sp 00007fff1862fa08 error 4 in libQtCore.so.4.8.6[7f04d71eb000+2d8000]

[Matthew Hall (mhall-9)]

Adding the grub probe output requested from other reporters.

[Robert Hrovat (robi-hipnos)]

After so long time this bug is still present. I gave up with Ubuntu server 14.04 and in 16.04 I encountered the same bug. You can't install Ubuntu server with RAID setup from installer.
After spending hours and tried to find what was the difference I made with Gentoo setup that worked perfectly well and finaly managed to install Ubuntu Server on RAID1. The main thing was Ubuntu create RAID with 1.2 version metadata which should be fine, but it auto detects RAID setups wrong. So in installer I escaped to shell and manually built array with command:

mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 --metadata=0.90 /dev/sda1 /dev/sdb1

to force creation of 0.90 metadata which works properly. After that I escaped shell and countinued setup from point where I just had to select filesystem and mount point for RAID1 partition.

Can you please fix this bug? It really does not look serious in server edition which should be installed with high security and availability in mind.

[Christian Ehrhardt  (paelzer)]

Hi,
I wondered about this as there are docs about it and it is not like "everybody complains".
Docs:
- https://help.ubuntu.com/lts/serverguide/advanced-installation.html
- https://help.ubuntu.com/community/Installation/SoftwareRAID

I tested that with a virtual guest for that I set up 3x20GB Disks.
I wanted to configure them the following way:
Partitions on Each
p1 = 1G
p2 = 19G

The small ones for boot on raid 1 (remember boot can only be raid 1, not 5,6,...) as outlined in the links above.
disk1p1 - raid 1 boot
disk2p1 - raid 1 boot
disk3p1 - swap
The bigger part in a raid 5
disk1p2 - raid 5 /
disk2p2 - raid 5 /
disk3p2 - raid 5 /

Following the guides linked above with the described target in mind (to prove that not only the most default setup works) I ended up in the installer with this view:

RAID1 device #0 - 998.7 MB Software RAID device
                  998.7 MB f ext4 /boot
RAID5 device #1 - 40.9 GB Software RAID device
                   40.9 GB f ext4 /
SCSI1 (0,0,0) (sda) - 21.5 GB ATA QEMU HARDDISK
     #1 primary 999.3 MB B K raid
     #2 primary 20.5 GB K raid
SCSI2 (0,0,0) (sdb) - 21.5 GB ATA QEMU HARDDISK
     #1 primary 999.3 MB B K raid
     #2 primary 20.5 GB K raid
SCSI2 (0,1,0) (sdc) - 21.5 GB ATA QEMU HARDDISK
     #1 primary 999.3 MB F swap swap
     #2 primary 20.5 GB K raid

From there I let the installer finish his work and it was booting and running right into the raid devices as it should.

So I wonder what exactly is the issue that is left these days with Ubuntu 16.04?

[Kevin Lyda (lyda)]

I can report the exact same issue with a fresh 16.04 install.

I did manual partitioning, left the requisite 1MB+1MB free for grub and it now boots to grub-rescue with a message about an unknown filesystem. I booted into rescue mode and ran grub-install /dev/sda which annoyingly installed on my usb install fob (that will teach me not to check device names).

I'm now running grub-install on the booted system and will see if it reboots.

Nope. Still broken.

[Kevin Lyda (lyda)]

Wonder if the issue is installing off a usb drive. Is something confused by device order?

[Kevin Lyda (lyda)]

Booted again off of fob - which a rescue boot had installed grub onto. I had removed it the last time when it booted and I ran grub-install /dev/sda. Will now try to run grub-install /dev/sdb.

[Kevin Lyda (lyda)]

Doh! In my case the issue was that I had replaced one of the disks on the server. In the process the server had decided it was going to boot from disk #3... Changed that and it all seems to have worked. Trying a full RAID install now.

[Phillip Susi (psusi)]

You might want to install grub to all of the drives in the server ( dpkg-reconfigure grub-pc ). That way if one fails and it boots from another, it can boot too.

[Launchpad Janitor (janitor)]

[Expired for grub-installer (Ubuntu) because there has been no activity for 60 days.]

[Jost Menke (jmenke)]

I can confirm this bug for Ubuntu server 16.04.3 AMD64. The problem seems to be related to the drive size or/and the interface. grub-install fails with two 2 TB Samsung 960 Pro m2 SSDs. With Intel 180 GB SATA SSDs, it works just fine. Mainboard: ASUS Prime X399-A

[Phillip Susi (psusi)]

I'm not sure why but it seems that a number of people have piled onto this bug report with unrelated issues. The original issue here was simply that grub won't install to a raid without mdadm installed. Since that isn't a bug, I'm closing this report.