IBP: wrong variant of GBUB installed => booting from a huge hard drive (10+ TB) failed

[Oleg S. Gelbukh]

In the following hardware configuration, grub can't boot into image after reboot:

Trying to load : pxelinux.cfg/<MAC> ok
Booting...
error: no such device: <some uuid>.
Entering rescue mode...
grub rescue>

Grub can see 2 devices (hd0), (hd1) and GPT labels on them:
grub rescue> ls
(hd0) (hd0,gpt3) (hd0,gpt2) (hd0,gpt1) (hd1) (hd1,gpt3) (hd1,gpt2) (hd1,gpt1)
grub rescue> insmod ext2
grub rescue> ls (hd0,gpt1)/boot
error: unknown filesystem

No partition contains /boot filesystem.

Output of lspci from bootstrap on the node is in attachment.

[Bogdan Dobrelya (bogdando)]

Won't fix for the 6.0.x milestone as IBP has an experimental status there

[Dmitry Pyzhov (dpyzhov)]

Please add diagnostic snapshot.

[Oleg S. Gelbukh (gelbuhos)]

Diagnostic snapshot in attachment.

[Sergii Golovatiuk (sgolovatiuk)]

According to logs, IBP didn't try to roll out the image. It was not able to find the proper block device.

[Alexander Gordeev (a-gordeev)]

It's not clear what happens.

fuel-agent log stops at
2015-06-03 17:08:14.734 9132 DEBUG fuel_agent.utils.utils [-] Trying to execute command: lvcreate -L 32768m -n swap os

any 'lvs'/'pvs'/another lvm related command gets stuck.

[root@bootstrap ~]# pvs
^C Interrupted...
  Giving up waiting for lock.
  /var/lock/lvm/V_os: flock failed: Interrupted system call
  Can't get lock for os
  Skipping volume group os
  PV VG Fmt Attr PSize PFree
  /dev/sda5 vm lvm2 a-- 10.82t 10.82t
[root@bootstrap ~]# pvs
^C Interrupted...
  Giving up waiting for lock.
  /var/lock/lvm/V_os: flock failed: Interrupted system call
  Can't get lock for os
  Skipping volume group os
  PV VG Fmt Attr PSize PFree
  /dev/sda5 vm lvm2 a-- 10.82t 10.82t
[root@bootstrap ~]# lvs

Also noticed that if 'provision' is executed in background just as 'provision &' it will fihish proreply.

I'm not sure how to properly reproduce lvm deadlock and how to avoid it.

[Alexander Gordeev (a-gordeev)]

> root 17655 0.0 0.0 25596 4612 pts/0 SN+ 17:17 0:00 lvcreate -L 32768m -n swap os

[Andrey Maximov (maximov)]

moving to mos-linux team to fix lvm or at least to understand why it happened

[Alexander Gordeev (a-gordeev)]

Many thanks Sergii Golovatuik for help.

we were able to strace lvcreate.

https://bugzilla.redhat.com/show_bug.cgi?id=997223

lvm is waiting for user's prompt when trying to wipe signature.

"WARNING: <signature name> signature detected on <device name>. Wipe it? [y/n]"

$ lvm dumpconfig |grep wipe
    wipe_signatures_when_zeroing_new_lvs=1

so in order to fix lvm locking, we should set it to 0 in /etc/lvm.conf.

[Aleksander Mogylchenko (amogylchenko)]

LVM supports several command line options to override this behavior: --yes or --wipesignatures. Please use one of them in order to ensure needed behavior.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/188356

[Alexander Gordeev (a-gordeev)]

lvcreate wasn't a root cause. it was entirely new bug.

with lvcreate patch applied, provisioning scripts was succeeded and was completed without any errors//traces/hangs.

it looks like it's GRUB2 issue. It only recognized 2 disks out of 3.

/boot partition was installed in /dev/sda (which is 11.5TB h/w array). GRUB didn't see /dev/sda at all.

As for ugly workaround, I can add some logic into agent to prevent choosing very large disks (2TB+) to be chosen for placing /boot on them.

[Alexander Gordeev (a-gordeev)]

Let mos-linux team take a look at it. Looks like completely GRUB related issue.

[Alexei Sheplyakov (asheplyakov)]

fuel-agent should install grub-efi instead of grub-pc on hard drives larger than 2 TB.

At the moment fuel-agent always installs grub-pc [1]. That version of grub uses BIOS INT13h to access hard drives.
In theory current BIOSes should support LBA48 INT 13h extension so virtually any drive should be readable.
However in practice all firmware suck so quite a number of BIOSes support at most LBA32 (which yields 2 TB size limit).
In order to avoid fighting the crappy firmware fuel-agent should install EFI based grub on hard drives larger than 2 TB.

[1] https://github.com/stackforge/fuel-web/blob/master/fuel_agent/fuel_agent/drivers/nailgun.py#L480

[Stanislaw Bogatkin (sbogatkin)]

Just my 2 cents about how it all works.

If you need just a solution - they are written at the end of comment.

TL;DR

There are BIOS or UEFI boot shcemes and MBR or GPT disk schemes. You CANNOT use UEFI with MBR - it just will not work as long as GPT is mandatory UEFI part.
Also we have bootloader - in our case, grub (when I said 'grub', I do not mean grub legacy).
So, we can have next cases when user want to install operating system:

1. BIOS+MBR
In such case you cannot use (primarily) hard disks larger than 2TB - you can't address that with MBR partitioning scheme. Usually disk partitioning looks like:
- MBR
- MBR gap (your partitions will start from sector 63 due to old DOS restriction about partitioning on cylinder boundaries, so we have about 32 free unused kilobytes there)
- Your partitions (usually, /boot then /)

That thing - MBR gap is used by grub for store stage 1.5 bootloader (core.img in grub2, if I remember it right). So, your MBR point with grub stage 1 point to grub stage 1.5 and then grub stage 1.5 point to your /boot partition with grub stage 2.

2. BIOS+GPT
In this case you can use much larger disks - up to some insane numbers in PB. Usually disk partitioning looks like:
- Protected MBR
- GPT (mbr is part of it, actually)
- Special bios boot partition (usually about 1Mb in size)
- Your partitions (/boot and /)

As long as GPT doesn't have such requirement as partitioning by cylinder boundaries, we don't have 32 kilobytes space there - partitions start just after GPT itself. But we need to store out grub stage 1.5 loader somewhere. For such case we create SPECIAL partition with code ef02, named bios_boot. It replaces old MBR gap and grub stored it's stage 1.5 loader there. But it is an old good grub there, nothing actually new.

3. UEFI+GPT
As long as UEFI is absolutely new standard, old grub cannot be loaded by UEFI firmware. There are especial new one (actually, not - it just ported to meet UEFI specifications) grub-efi that should be installed. Usually disk partitioning looks like:
- Protected MBR (don't used in most of cases)
- GPT (mbr is part of it, actually)
- Analog of /boot partition for UEFI standard, code ef00 (EFI partition, some FAT-related fs with bootloader on it)
- Your partitions (/boot and /, you can place /boot in efi partition)

In this case there are no such thing as stage 1.5 for grub. GPT point right to your loader in EFI partition, nothing else. But it is new grub, incompatible with old one. Also you need to know that your /boot partition should be placed in first 2TB of your hard disk.

So, you can use next solutions there:
1. Don't support GPT and UEFI. No need to rewrite something, this bug will not closed.
2. Support GPT, don't support UEFI. You just need to rewrite your partition scheme, it's all. Fully compatible with old one.
3. Support GPT and UEFI. You will need to do some rather complex logic to understand what you need to do. Actually, we already have bug with UEFI support for 7.0, so I rather don't recommend do it now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/189698

[Vladimir Kozhukalov (kozhukalov)]

Stanislaw,

Rewriting partition scheme won't help because grub does not see huge disk no matter how you make partitions.

[Alexei Sheplyakov (asheplyakov)]

> 2. BIOS+GPT
> In this case you can use much larger disks - up to some insane numbers in PB.

Unless you've got a buggy (UEFI) firmware which does not support LBA48 in BIOS emulation mode.

> Usually disk partitioning looks like:
> - Protected MBR
> - GPT (mbr is part of it, actually)
> - Special bios boot partition (usually about 1Mb in size)
> - Your partitions (/boot and /)

Actually fuel-agent does not create bios boot partition (ef02), and certain firmwares refuse to boot from such a drive in a BIOS emulation mode.

[Alexei Sheplyakov (asheplyakov)]

> That thing - MBR gap is used by grub for store stage 1.5 bootloader (core.img in grub2, if I remember it right).
> So, your MBR point with grub stage 1 point to grub stage 1.5 and then grub stage 1.5 point to your /boot partition with grub stage 2.

There's no such thing as stage 1.5 in grub2, see http://www.gnu.org/software/grub/manual/grub.html#Images

[Vladimir Kozhukalov (kozhukalov)]

> Actually fuel-agent does not create bios boot partition (ef02), and certain firmwares refuse to boot from such a drive in a BIOS emulation mode.

As far as I understand, we are talking about this small partition https://github.com/stackforge/fuel-web/blob/master/fuel_agent/fuel_agent/drivers/nailgun.py#L172-L175 So, Fuel agent does create this partition, but it's size 24 MB (for some reasons).

[Alexei Sheplyakov (asheplyakov)]

> Actually fuel-agent does not create bios boot partition (ef02)

I've filed a bug on this issue: https://bugs.launchpad.net/fuel/+bug/1463708

[Alexei Sheplyakov (asheplyakov)]

> As long as UEFI is absolutely new standard

UEFI 2.0 spec [1] is 9 years old, EFI version of grub2 is almost 7 years old.

[1] http://www.uefi.org/sites/default/files/resources/UEFI_Specification_2_and_Errata_Sept16_08.pdf

[Alexei Sheplyakov (asheplyakov)]

> I've filed a bug on this issue: https://bugs.launchpad.net/fuel/+bug/1463708

Apparently BIOS boot partition is here. However EFI system partition is missing (which might cause firmware misbehavior)

[Alexei Sheplyakov (asheplyakov)]

> grub rescue> insmod ext2
> grub rescue> ls (hd0,gpt1)/boot
> error: unknown filesystem

Actually the 1st GPT partition created by provisioning script is a BIOS boot partition (which is kind of wrong, but that's a different problem). That partition is not supposed to hold any filesystem (grub2 stores its core image into that partition). Therefore...

> No partition contains /boot filesystem.

... this conclusion is wrong.

> Trying to load : pxelinux.cfg/<MAC> ok
> Booting...
> error: no such device: <some uuid>.
> Entering rescue mode...
> grub rescue>

Cobbler might have sent a wrong configuration (root=UUID which does not actually exists, or something).
We need to investigate this bug more carefully. That is,
- check if another partition contains /boot filesystem
- if yes, check if grub is able to boot from that (with manually passed root= argument)

[Alexander Gordeev (a-gordeev)]

> However EFI system partition is missing (which might cause firmware misbehavior)

Alexei Sheplyakov, are you sure that absence of EFI partition could somehow interfere with not found disk by grub2? Could you clarify how it could help?

[Alexei Sheplyakov (asheplyakov)]

Some [U]EFI firmware in BIOS emulation mode refuses to boot from a hard drive which 1) has GPT, 2) has no EFI system partition
(such a hard drive is not available via BIOS int13h).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/189698
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=059831145c657a5eaba563b18652351a459de566
Submitter: Jenkins
Branch: master

commit 059831145c657a5eaba563b18652351a459de566
Author: Vladimir Kozhukalov <email address hidden>
Date: Tue Jun 9 15:56:26 2015 +0300

    IBP: /boot on small hard drive if possible

    On some hardware GRUB can not see huge hard drives (>2T)
    due to firmware bugs, so we'd better avoid placing
    /boot on such drives if it is possible.

    Change-Id: I71e0255820edef4725208db777bf5e0bf8d3958c
    Partial-Bug: #1461126

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/190642

[Vladimir Kozhukalov (kozhukalov)]

Release note:

Currently, /boot partition is created on a disk which is smaller than 2T, if there is such a disk on a node. If a node has only huge disks (>2T) and if BIOS does not support huge disks (LBA48), the node will not be able to boot. If it is a bug of UEFI BIOS emulation mode, than it will be addressed in the future by implementing of UEFI support.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/190642
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=fa44199924c630ce0356cfb74bcf4e94f6d9242e
Submitter: Jenkins
Branch: stable/6.1

commit fa44199924c630ce0356cfb74bcf4e94f6d9242e
Author: Vladimir Kozhukalov <email address hidden>
Date: Tue Jun 9 15:56:26 2015 +0300

    IBP: /boot on small hard drive if possible

    On some hardware GRUB can not see huge hard drives (>2T)
    due to firmware bugs, so we'd better avoid placing
    /boot on such drives if it is possible.

    Change-Id: I71e0255820edef4725208db777bf5e0bf8d3958c
    Partial-Bug: #1461126

[Mike Scherbakov (mihgen)]

Vladimir, can you please update bug status? I see commits merged, is it right to assume that the bug is closed?

[Vladimir Kozhukalov (kozhukalov)]

This commit fixes only this particular case when we have more than one disk on a node and at least one of them is smaller that 2T. The root cause of the issue is UEFI's BIOS emulation bug. It does not support LBA48. The correct way to address this issue is to implement uefi support.