MAAS sets an invalid choice for the IPMI cipher suite ID during commissioning

[Jeff Lane ]

MAAS: 3.4.2

I am enlisting and commissioning a Lenovo SD550 V3 server in our MAAS. I powered the machine on and it completed enlistment and appeared in the UI as a New server.

However, power status was marked as an Error. On further investigation, the Cipher Suite ID in the Configuration for that machine was set to 3. To see if that had anything to do with the power control issues, I set it to 17 and saved the config change. After that, MAAS was able to successfully probe power status and was able to control the power to turn the machine on when I selected "Commission" from the action menu.

Commissioning was successful as well, however, after commissioning, power status checking was again failing. I went back to the machines Configuration page and noted that the Cipher Suite ID had been changed back to 3 again. So once more, changing that to 17 and saving it allowed MAAS to successfully probe power on this machine.

As a check, I then recommissioned the machine. MAAS again was able to power the machine on, which successfully completed commissioning a second time. And just as before, MAAS changed the Cipher Suite ID back to 3, causing power control to fail.

Ref: Yakkey in the Cert Lab

[Alessandro Marcolini (alemar99)]

Hi Jeff,
I think that the problem should be related to the 30-maas-01-bmc-config commissioning script, because the default value for the `ipmi_cipher_suite_id` is set to 3 (source /src/metadataserver/builtin_scripts/commissioning_scripts/bmc_config.py).

To solve this you can verify and change the default value by doing the following:
```
# this should output 3
maas $PROFILE maas get-config name=maas_auto_ipmi_cipher_suite_id
# change the value to 17
maas $PROFILE maas set-config name=maas_auto_ipmi_cipher_suite_id value=17
# verify the change
maas $PROFILE maas get-config name=maas_auto_ipmi_cipher_suite_id
```

[Jeff Lane  (bladernr)]

That is a quick workaround, not a resolution... I know how to manually change the Cipher Suite ID to what works, which is how I got the machine working in the first place.

The problem is that MAAS does not honor my choices and overrides them with an incorrect default. This has now been seen against two Lenovo machines in the lab.

I have two actual issues here from this:

1: MAAS is overriding user settings with a hard coded default in a shell script. If I set that manually in the BMC Config in MAAS, MAAS should not overwrite it the next time the machine is commissioned. User set configuration should be immutable (perhaps MAAS needs an additional flag like "Don't update BMC Config" when commissioning is run.

2: MAAS should do a better job finding one that works. Without the correct ID, the ipmi commands will fail and not every machine/BMC will support ID 3 (as is the case here). MAAS should verify that the Cipher Suite ID it's setting is actually usable by attempting IPMI commands of some sort to see if it works, and if not, should move to the next one and retry until if finds the one that works with that particular BMC.

Extrapolate this out from a small number of systems to a customer with hundreds of them, it is not good experience if they then have to manually fix every one of the machines in MAAS because of a hard coded setting.

[Jeff Lane  (bladernr)]

Additionally per the manpage for ipmitool:

C <ciphersuite>
    The remote server authentication, integrity, and encryption algorithms to use for IPMIv2.0 lanplus connections. See table 22-20 in the IPMI v2.0 specification.
    The default is 17 which specifies RAKP-HMAC-SHA256 authentication, HMAC-SHA256-128 integrity, and AES-CBC-128 encryption algorithms.
    NOTE: In 1.8.18 and earlier the default was 3, which was insecure and was not supported by some more recent BMC implementations.

Doing a quick check with ipmitook this takes very little time. This checks the four IDs that are presented in the MAAS UI in 3.4.2:

bladernr@galactica:~/development/checkflinger$ ipmitool -V
ipmitool version 1.8.19
bladernr@galactica:~/development/checkflinger$ time for csi in 17 3 8 12; do echo "Checking ID $csi"; if ipmitool -U $USER -P $PASSWORD -H $HOST -I lanplus -C $csi power status &> /dev/null; then echo SUCCESS; else echo FAILURE; fi; done
Checking ID 17
SUCCESS
Checking ID 3
FAILURE
Checking ID 8
FAILURE
Checking ID 12
FAILURE

real 0m1.412s
user 0m0.054s
sys 0m0.018s

It is also a pretty fast check using freeipmi-tools as well:

bladernr@galactica:~/development/checkflinger$ ipmipower -V
ipmipower - 1.6.11
Copyright (C) 2007-2015 Lawrence Livermore National Security, LLC.
Copyright (C) 2003-2007 The Regents of the University of California.
This program is free software; you may redistribute it under the terms of
the GNU General Public License. This program has absolutely no warranty.
bladernr@galactica:~/development/checkflinger$ time for csi in 17 3 8 12; do echo "Checking ID $csi"; if ipmipower -u $USER -p $PASSWORD -h $HOST -D LAN_2_0 -I $csi -s &> /dev/null; then echo SUCCESS; else echo FAILURE; fi; done
Checking ID 17
SUCCESS
Checking ID 3
FAILURE
Checking ID 8
FAILURE
Checking ID 12
FAILURE

real 0m1.035s
user 0m0.001s
sys 0m0.021s

[Alessandro Marcolini (alemar99)]

Sorry for being superficial, you're right.

Regarding MAAS overriding the current configuration: you can avoid this by checking the option "Skip configuring supported BMC controllers with a MAAS generated username and password" and MAAS will not override anything (the 30-maas-01-bmc-config will be skipped). When you are doing commissioning without that, you're basically asking MAAS to configure the BMC and, to do so, it will override the actual configuration (with the wrong cipher suite id, unfortunately).

Regarding the cipher suite id we should implement a solution to find the right one (thanks for pointing out possible ways of doing this and the correct default cipher suite id).

[Mauricio Faria de Oliveira (mfo)]

> Regarding the cipher suite id we should implement a solution to find the right one (thanks for pointing out possible ways of doing this and the correct default cipher suite id).

For doc purposes: a related feature (with technical details) has been submitted for product feedback internally as PF-4702 ("Allow users to opt-in for IPMI Cipher Suite auto-detection available with ipmitool in Noble images").

[Jacopo Rota (r00ta)]

So just to recap:

1) if the user commissions the machine without checking the flag "Skip configuring supported BMC controllers with a MAAS generated username and password" it is expected that MAAS will replace the current BMC configuration of the machine

2) it is not expected that MAAS sets an invalid choice of the cipher suite ID during commissioning -> this can be considered a bug from the user's perspective. On the technical side, this is more a missing feature than a bug.

For this reason, we keep this bug open as well as we keep track of it in the product feedback board. I'm changing the bug description accordingly.

Thank you very much for the feedbacks @Jeff!