[feature] Add ability to select the BIOS boot method for IPMI - To address issues with newer firmware.

Bug #1750622 reported by Jeff Hillman on 2018-02-20
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Medium
Andres Rodriguez
2.3
Undecided
Unassigned

Bug Description

Using HP DL380 Gen9 server with iLO FW 2.55 Aug 16 2017

Sending a commission statement from MAAS works fine. However when a deploy issued from MAAS, the ipmipower command being sent is causing HP to force a 1 time Legacy Boot Mode over PXE.

On the Remote Console the message ... appears

"298-IMPORTANT: The Boot Mode has been changed to Legacy Boot Mode for this boot only. On the next reboot, the Boot Mode will return to UEFI Boot Mode. Action: No action required."

This is causing nodes to not deploy that have a GPT partition table set with /boot/efi configured. Specifically it gives the following message ...

cloud-init[3081]: Unexpected error while running command.
cloud-init[3081]: Command: ['sh', '-c', 'exec "$0" "$@" 2>&1', 'install-grub', '/tmp/tmpsbfxj9fy/target', '/dev/sda']
cloud-init[3081]: Exit code: 1
cloud-init[3081]: Reason: -
cloud-init[3081]: Stdout: + pkg=grub-pc
cloud-init[3081]: + shift
cloud-init[3081]: + dpkg-reconfigure grub-pc
cloud-init[3081]: Replacing config file /etc/default/grub with new version
cloud-init[3081]: Generating grub configuration file ...
cloud-init[3081]: Found linux image: /boot/vmlinuz-4.4.0-112-generic
cloud-init[3081]: Found initrd image: /boot/initrd.img-4.4.0-112-generic
cloud-init[3081]: done
cloud-init[3081]: + update-grub
cloud-init[3081]: Generating grub configuration file ...
cloud-init[3081]: Found linux image: /boot/vmlinuz-4.4.0-112-generic
cloud-init[3081]: Found initrd image: /boot/initrd.img-4.4.0-112-generic
cloud-init[3081]: done
cloud-init[3081]: + grub-install /dev/sda
cloud-init[3081]: Installing for i386-pc platform.
cloud-init[3081]: grub-install: warning: this GPT partition label contains no BIOS Boot Partition; embedding won't be possible.
cloud-init[3081]: grub-install: warning: Embedding is not possible. GRUB can only be installed in this setup by using blocklists. However, blocklists are UNRELIABLE and their use is discouraged..
cloud-init[3081]: grub-install: error: will not proceed with blocklists.
cloud-init[3081]: + exit
cloud-init[3081]: failed to install grub!
cloud-init[3081]:
cloud-init[3081]: Stderr: ''
cloud-init[3081]: curtin: Installation failed with exception: Unexpected error while running command.
cloud-init[3081]: Command: ['curtin', 'curthooks']
cloud-init[3081]: Exit code: 3

We believe this is the same situation as bug https://bugs.launchpad.net/hp/+bug/1733899

To fix this, editing /usr/lib/python3/dist-packages/provisioningserver/drivers/power/ipmi.py and adding ...

        BIOS_Boot_Type EFI

in the Chassis_Boot_Flags sections and then restarting maas-rackd resolves this issue.

diff --git a/src/provisioningserver/drivers/power/ipmi.py b/src/provisioningserver/drivers/power/ipmi.py
index 9ef5ba4..e531dd4 100644
--- a/src/provisioningserver/drivers/power/ipmi.py
+++ b/src/provisioningserver/drivers/power/ipmi.py
@@ -35,6 +35,7 @@ IPMI_CONFIG = """\
 Section Chassis_Boot_Flags
         Boot_Flags_Persistent No
         Boot_Device PXE
+ BIOS_Boot_Type EFI
 EndSection
 """

Related branches

Jason Hobbs (jason-hobbs) wrote :

The IPMI spec for boot options has two choices - either a system boots from EFI, or from PC Compatible (BIOS). See 'boot flags' in Table 28 from [1]. There is no way via IPMI to say "boot from BIOS/EFI according to how to the system's firmware is configured".

MAAS doesn't set anything, so freeipmi uses the default which is to tell the system to boot from BIOS.

The fix for this is to make MAAS aware of whether systems are BIOS or EFI, and to tell EFI systems to boot from EFI using the BIOS_Boot_Type field Jeff posted in the description.

[1] https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/ipmi-second-gen-interface-spec-v2-rev1-1.pdf

Changed in maas:
milestone: none → 2.4.x
status: New → Triaged
summary: - PXE boot on HP DL380 Gen9 is forced into a one time Legacy Boot Mode
+ [feature] PXE boot on HP DL380 Gen9 is forced into a one time Legacy
+ Boot Mode
Andres Rodriguez (andreserl) wrote :

So i think there are different variables here:

1. boot order (e.g. pxe, disk, usb, cdrom, etc)
2. boot method (legacy or efi).

So, with this issue, what we are saying is that when using a tool to manage the BMC remotely, it will *NOT* respect the settings that have been manually set by the administrator in the BIOS itself. For example, if the administrator sets:

1. boot order - disk first, pxe second
2. boot method - efi
3. ipmitool -I lanplus -H <iLO ip> -U admin -P admin123 chassis bootdev pxe

And then the administrator uses ipmitool/freeipmi-tools to tell the machine to PXE boot the next time instead of boot from disk, then it will *NOT* respect the fact that the boot method is EFI, and it will boot in legacy mode instead.

To me, this seems like a critical issue that the firmware *automatically* decides to overwrite the setting in the *bios*, without the user specifically requesting to do so.

I would think this is something that needs to be fixed in the IPMI spec.

summary: - [feature] PXE boot on HP DL380 Gen9 is forced into a one time Legacy
- Boot Mode
+ [feature] Add ability to select the BIOS boot method for IPMI - To
+ address issues with newer firmware.
Jason Hobbs (jason-hobbs) wrote :

The problem isn't in the firmware, it's in the IPMI specification. It doesn't have a way to respect the firmware's boot settings - it requires either EFI or BIOS to be set in the IPMI boot options. The firmware could ignore that field, but that would break compatibility with IPMI and would leave no way to specify via IPMI that the machine should boot in BIOS or UEFI.

Andres Rodriguez (andreserl) wrote :

The firmware could gain the ability to, if the option is not present, then *respect* what's currently set in the BIOS. That would provide backwards compatibility.

Changed in maas:
milestone: 2.4.x → none
tags: added: feature internal wishlist
Jason Hobbs (jason-hobbs) wrote :

The firmware can't gain that ability. The field is set via a bit in an parameter value that has been there all along in the IPMI spec. Its original reserved value was 0, and the value for BIOS in the spec is now 0. The firmware can't distinguish between '0' being reserved and not set purposefully, and '0' being set to indicate BIOS boot.

Andres Rodriguez (andreserl) wrote :

@Jeff,

Can you gather the output of for the machine with the newer firmware:

ipmi-chassis-config -D LAN_2_0 -u XX -p XX -h XX --checkout -e Chassis_Boot_Flags:BIOS_Boot_Type

Jeff Hillman (jhillman) wrote :

Certainly.

$ ipmi-chassis-config -D LAN_2_0 -u xxxx-p xxxx-h x.x.x.x--checkout -e Chassis_Boot_Flags:BIOS_Boot_Type
Section Chassis_Boot_Flags
        ## Possible values: PC-COMPATIBLE/EFI
        BIOS_Boot_Type PC-COMPATIBLE
EndSection

Andres Rodriguez (andreserl) wrote :

Sucks. It reports the system is set to Legacy even though it is actually set to EFI.

Jeff Hillman (jhillman) wrote :

That is basically what Narinder found. That the default posture is to PXE boot Legacy unless explicitly told otherwise. This will likely have broader ramifications ultimately.

Jason Hobbs (jason-hobbs) wrote :

The value reported in that IPMI command is not what the firmware is configured to do; it is just what was written to it by previous IPMI commands. The IPMI spec calls this "semi-volatile".

Also note that boot flags get reset after ~60 seconds if the Persistent option is not set, which MAAS does not set.

for Lenovo there's an option to force UEFI, and ignore legacy.
eg. ./OneCli config set BootModes.IgnoreLegacyFlagOnIpmi enabled --override --imm USERID:PASSW0RD@<ipaddress>

Jeff Hillman (jhillman) wrote :

I am gonna ask that this be changed back to a bug like I originally had it. This is not a feature request.

I say this because I have seen it in 2 other environments now.

to put it another way, since this was discovered, I have had to modify /usr/lib/python3/dist-packages/provisioningserver/drivers/power/ipmi.py 100% of the time.

Hi Jeff,

This is not a bug in MAAS provided that it is not MAAS that has changed. It
is the firmware that has changed and requiring new behavior which requires
MAAS to add a new capability (feature) that was not previously supported.

On Mon, Mar 19, 2018 at 7:27 AM Jeff Hillman <email address hidden>
wrote:

> I am gonna ask that this be changed back to a bug like I originally had
> it. This is not a feature request.
>
> I say this because I have seen it in 2 other environments now.
>
>
> to put it another way, since this was discovered, I have had to modify
> /usr/lib/python3/dist-packages/provisioningserver/drivers/power/ipmi.py
> 100% of the time.
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1750622
>
> Title:
> [feature] Add ability to select the BIOS boot method for IPMI - To
> address issues with newer firmware.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1750622/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas; status=Triaged; importance=Undecided;
> assignee=None;
> Launchpad-Bug-Tags: cpe-onsite feature internal wishlist
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl gabor.meszaros jason-hobbs jhillman
> Launchpad-Bug-Reporter: Jeff Hillman (jhillman)
> Launchpad-Bug-Modifier: Jeff Hillman (jhillman)
> Launchpad-Message-Rationale: Subscriber (MAAS)
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

Changed in maas:
assignee: nobody → Andres Rodriguez (andreserl)
Changed in maas:
status: Triaged → In Progress
milestone: none → 2.4.0beta1
Changed in maas:
importance: Undecided → Medium
Changed in maas:
status: In Progress → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
tags: added: 4010
Luca Cervigni (cervigni) wrote :

I have installed maas 2.4.0-6981 on ubuntu 18.04 and the issue still persist. The flag is not in the file. Isn't the fix released from the beta?

Applying manually the patch fixes the issue but it would be better if fixed in the packages.
Could you please have a look?

Here the MAAS version

ii maas 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS client and command-line interface
ii maas-common 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server common files
ii maas-dhcp 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS DHCP server
ii maas-proxy 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Rack Controller for MAAS
ii maas-region-api 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Region controller API service for MAAS
ii maas-region-controller 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Region Controller for MAAS
ii python3-django-maas 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server provisioning libraries (Python 3)

Andres Rodriguez (andreserl) wrote :

Hi Luca,

The fix is available in the release. I’ve double verified it and we also
have this under CI.

That said, don’t understand your statement of “the flag is not in the
file”. Can you clarify?

On Mon, Jul 30, 2018 at 11:30 PM Luca Cervigni <email address hidden>
wrote:

> I have installed maas 2.4.0-6981 on ubuntu 18.04 and the issue still
> persist. The flag is not in the file. Isn't the fix released from the
> beta?
>
>
> Applying manually the patch fixes the issue but it would be better if
> fixed in the packages.
> Could you please have a look?
>
> Here the MAAS version
>
> ii maas
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all "Metal as a Service" is
> a physical cloud and IPAM
> ii maas-cli
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS client and
> command-line interface
> ii maas-common
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server common
> files
> ii maas-dhcp
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS DHCP server
> ii maas-proxy
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS Caching Proxy
> ii maas-rack-controller
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Rack Controller for MAAS
> ii maas-region-api
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Region controller API
> service for MAAS
> ii maas-region-controller
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Region Controller for
> MAAS
> ii python3-django-maas
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server Django web
> framework (Python 3)
> ii python3-maas-client
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS python API client
> (Python 3)
> ii python3-maas-provisioningserver
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server
> provisioning libraries (Python 3)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1750622
>
> Title:
> [feature] Add ability to select the BIOS boot method for IPMI - To
> address issues with newer firmware.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1750622/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas; milestone=2.4.0beta1; status=Fix Released;
> importance=Medium; <email address hidden>;
> Launchpad-Bug: product=maas; productseries=2.3; milestone=2.3.2;
> status=Fix Released; importance=Undecided; assignee=None;
> Launchpad-Bug-Tags: 4010 cpe-onsite feature internal wishlist
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl cervigni gabor.meszaros jason-hobbs
> jhillman
> Launchpad-Bug-Reporter: Jeff Hillman (jhillman)
> Launchpad-Bug-Modifier: Luca Cervigni (cervigni)
> Launchpad-Message-Rationale: Assignee
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

Luca Cervigni (cervigni) wrote :

Yes sorry. Around line 35:

Original file:
https://paste.ubuntu.com/p/Bpzh2TxrY2/

File with patch applied ( line 38):
https://paste.ubuntu.com/p/NJPTQvC3ZQ/

in the original file the line in the
   BIOS_Boot_Type EFI
is not present.

I added it manually and the problem is fixed.

Andres Rodriguez (andreserl) wrote :

Hi Luca,

What you are describing is not the solution its available on the product. See lines 42-47 on the original file above:

IPMI_CONFIG_WITH_BOOT_TYPE = """\
Section Chassis_Boot_Flags
        Boot_Flags_Persistent No
        BIOS_Boot_Type %s
        Boot_Device PXE
EndSection
"""

The section:
        BIOS_Boot_Type %s

Is used when either forcing auto/legacy/EFI. That is an option on the configuration of the power settings for the machine. YOu can select between:

'auto' will let the BMC do its own thing, and this is done for backwards compatibility with older BMC versions.
'legacy' - self explanatory.
'efi' - self explanatory.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers