Failed trusty deployment on Lenovo X3650 M5 in UEFI mode

[Larry Michel]

We are no longer able to deploy trusty on the X3650 M5 in UEFI mode.

In curtin environment, it shows as booted in BIOS mode even though BMC says that BMC is running in UEFI mode.

$ [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
BIOS

So the <device>15 partition is never setup and system fails to boot to local disk even though deployment was completed.

I tried in BIOS mode to see whether this is a case of toggling values but BIOS deployment succeeds.

This stops working about a week or so ago. Initially, I thought that perhaps some settings got resetted somehow but after investigation, it's looking like system is in fact running UEFI mode per BMC, but that's not getting propagated to curtin.

When OS boots, it says booted OS or unsupported OS.

[Larry Michel (lmic)]

While chasing another issue, I observed these errors below in the maas log for one of the X3650 that's failing to deploy. I think it's worth investigating why maas is complaining about this server:

2015-08-05 15:42:33 [-] 127.0.0.1 - - [05/Aug/2015:15:42:32 +0000] "GET /MAAS/api/1.0/nodes/?agent_name=ed388ed1-b62c-4489-85c2-47dfe387d63b&id=node-08d6458c-3f4d-11e4-b38a-00163eca07b6&op=list HTTP/1.1" 200 795 "-" "Go 1.1 package http"
2015-08-05 15:42:33 [maasserver] ERROR: Unable to determine purpose for node: 'drapion.oil'
2015-08-05 15:42:33 [-] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the RequestNodeInfoByMACAddress command. [maas-trusty-back-may22:pid=53399:cmd=RequestNodeInfoByMACAddress:ask=2b8c4]
        Traceback (most recent call last):
          File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
            self.__bootstrap_inner()
          File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
            self.run()
          File "/usr/lib/python2.7/threading.py", line 763, in run
            self.__target(*self.__args, **self.__kwargs)
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
            result = context.call(ctx, function, *args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 154, in wrapper
            return func(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 404, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 300, in retrier
            return func(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 339, in inner
            return func(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/maasserver/rpc/nodes.py", line 194, in request_node_info_by_mac_address
            return (node, node.get_boot_purpose())
          File "/usr/lib/python2.7/dist-packages/maasserver/models/node.py", line 1856, in get_boot_purpose
            preseed_type = get_deploying_preseed_type_for(self)
          File "/usr/lib/python2.7/dist-packages/maasserver/preseed.py", line 379, in get_deploying_preseed_type_for
            purpose = get_available_purpose_for_node(purpose_order, node)
          File "/usr/lib/python2.7/dist-packages/maasserver/preseed.py", line 348, in get_available_purpose_for_node
            "Unable to determine purpose for node: '%s'", node.fqdn)
        maasserver.exceptions.PreseedError: (u"Unable to determine purpose for node: '%s'", u'drapion.oil')

2015-08-05 15:42:33 [maasserve...

[Rod Smith (rodsmith)]

The x3550 M5 and x3650 M5 that sparked this report have been temporarily moved to the certification VLAN for regression testing, and the x3550 exhibits this same problem when using the certification MAAS server. (The x3650 had been reconfigured to boot in BIOS/CSM/legacy mode prior to the transfer, and booted OK in that mode.) Certification's MAAS server boots another computer (an older IBM x3650 M2) in EFI mode without problems, so this appears to be an interaction between MAAS and Lenovo's M5 series specifically.

[Jeff Lane  (bladernr)]

First thing I'd suggest, is ensuring these two (x3650 and x3550 M5) have the latest firmware across the board. If this is a wonkiness in the EFI code on the system, that's the first thing Lenovo will say...

Here's the latest firmware for the x3550 M5:
http://www-933.ibm.com/support/fixcentral/systemx/selectFixes?parent=Lenovo%2BSystem%2Bx3550%2BM5&product=ibm/systemx/5463&&platform=All&function=all#UEFI
http://www-933.ibm.com/support/fixcentral/systemx/selectFixes?parent=Lenovo%2BSystem%2Bx3550%2BM5&product=ibm/systemx/5463&&platform=All&function=all

And x3650 M5:
http://www-933.ibm.com/support/fixcentral/systemx/selectFixes?parent=Lenovo%2BSystem%2Bx3650%2BM5&product=ibm/systemx/5462&&platform=All&function=all#UEFI
http://www-933.ibm.com/support/fixcentral/systemx/selectFixes?parent=Lenovo%2BSystem%2Bx3650%2BM5&product=ibm/systemx/5462&&platform=All&function=all

[Blake Rouse (blake-rouse)]

Curtin identifies a efi system by checking if "/sys/firmware/efi" exists on the system [1]. This should be the case if the machine booted with EFI. Can you verify that this path exists?

[1] http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/view/head:/curtin/util.py#L518

[Launchpad Janitor (janitor)]

[Expired for MAAS because there has been no activity for 60 days.]

[Neo Cui (cuilj2)]

I found Ubuntu MAAS can’t recognize Lenovo SystemX BIOS working mode correctly.
MAAS takes it as Legacy mode even if I set the BIOS to UEFI mode.
Then it can’t create the ESP and cause the deploy error. Following information is the error message in log file.
Mar 30 10:36:25 real-tick cloud-init[3578]:: System not running in EFI mode, not installing to EFI system partition.
Mar 30 10:36:25 real-tick cloud-init[3578]: efibootmgr: EFI variables are not supported on this system.
You can check detail information in the attached messages log.
I wonder how did MAAS server check if the server working in UEFI/Legacy mode.
Does it depend on a variable provide by BIOS firmware? Or something else.
Please help provide the information about the mechanism of MAAS to check BIOS working mode (UEFI/Legacy).