ThinkSystem SR645 failed commissioning ERROR: Failed to commit `User3:Password': Invalid/Unsupported Config

[Angel Vargas]

Unable to commission ThinkSystem SR645

MAAS: 3.2.6

30-maas-01-bmc-config details

Stderr
```
ERROR: Failed to commit `User3:Password': Invalid/Unsupported Config
ERROR: Failed to commit `User3:Password': Invalid/Unsupported Config
ERROR: Unable to add BMC user!
```

Seen on MAAS 3.2 stable snap channel; later, we upgraded to 3.3.0~beta1, but the same problem is noticed.

Is there any workaround for this problem?

[Angel Vargas (angelvargas)]

This is part of the commissioning log

ERROR: Redfish <urlopen error [Errno 111] Connection refused>
Traceback (most recent call last):
  File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/usr/lib/python3.10/http/client.py", line 1447, in connect
    super().connect()
  File "/usr/lib/python3.10/http/client.py", line 941, in connect
    self.sock = self._create_connection(
  File "/usr/lib/python3.10/socket.py", line 845, in create_connection
    raise err
  File "/usr/lib/python3.10/socket.py", line 833, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/user_data.sh.HCysqt/scripts/commissioning/30-maas-01-bmc-config", line 1103, in detect_and_configure
    if bmc.detected():
  File "/tmp/user_data.sh.HCysqt/scripts/commissioning/30-maas-01-bmc-config", line 1037, in detected
    return self.get_bmc_ip() is not None
  File "/tmp/user_data.sh.HCysqt/scripts/commissioning/30-maas-01-bmc-config", line 1092, in get_bmc_ip
    self._bmc_ip = self._get_bmc_ip()
  File "/tmp/user_data.sh.HCysqt/scripts/commissioning/30-maas-01-bmc-config", line 1055, in _get_bmc_ip
    response = urllib.request.urlopen(
  File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.10/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>

[Angel Vargas (angelvargas)]

By setting the security requirements for the BMC to "compatability mode" and setting TLS 1.1 and newer, the previous error doesn't show anymore but stills fails with:

```
ERROR: Failed to commit `User3:Password': Invalid/Unsupported Config
ERROR: Failed to commit `User3:Password': Invalid/Unsupported Config
ERROR: Unable to add BMC user!
INFO: Loading IPMI kernel modules...
INFO: Checking for HP Moonshot...
INFO: Checking for Redfish...
INFO: Reading current IPMI BMC values...
INFO: Found existing IPMI user "maas"!
INFO: Configuring IPMI BMC user "maas"...
INFO: IPMI user number - User3
INFO: IPMI user privilege level - Administrator
```

[Björn Tillenius (bjornt)]

It doesn't give any details on what's wrong, but most likely the password isn't considered strong enough, since we have seen hints toward that before.

Could you please manually try to set different passwords and see if that's the case, and maybe also check what the rules for your system are?

To work around it, you can set the username and password manually in both the BMC and MAAS, and then check "Skip configuring supported BMC controllers with a MAAS generated username and password" when commissioning the machine.

The traceback you see should be harmless. It means that your machine is advertising that there is a Redfish host interface, but it's probably disabled.

[Angel Vargas (angelvargas)]

@bjornt thanks for the reply, what other details can I share to help to debug this better? if you can provide some instructions I can try to give you the details.

[Björn Tillenius (bjornt)]

I found some info on the password policy for Lenovo here: https://sysmgt.lenovofiles.com/help/index.jsp?topic=%2Fcom.lenovo.systems.management.xcc.doc%2FNN1ia_c_accountsecuritypolicysettings.html

As far as I can see, though, the default policy should work for the passwords that MAAS generates. When generating passwords we generate two; one simple (which doesn't comply) and one with special characters, which should be good enough.

Can you check the settings in XClarity to see if you have enable some additional restrictions?

Also, it would be good if you could upload this file as a commissioning script called 29-debug-01-bmc-config: https://people.canonical.com/~bjorn/bug-1993916/bmc_config.py

If you then re-run the commissioning you should see which password it's trying to use.

[Launchpad Janitor (janitor)]

[Expired for MAAS because there has been no activity for 60 days.]

[laurent wandrebeck (laurentwandrebeck)]

Hello, un-burying this ticket, as I hit a similar problem trying to toy with a Compal SR221-2A (running maas 3.3/edge).
The 30-maas-01-bmc-config errors out with the same error above.
by hand, in rescue mode:
ubuntu@witty-eagle:~$ sudo ipmitool user set password 7 secretpassword
IPMI command failed: Unknown (0x03)
Set User Password command failed (user 7)
but it works with a 15+ characters password.
ubuntu@witty-eagle:~$ sudo ipmitool user set password 7 secretpassword1
Set User Password command successful (user 7)
Would it be possible to change default password creation to 15 chars by default ?

Thanks,
Laurent.

[Jacopo Rota (r00ta)]

We'll add a password of length >= 16 in the set of passwords that we try at that stage

[laurent wandrebeck (laurentwandrebeck)]

Thanks a lot for the quick fix. Is a backport to 3.3 planned ?

[Jacopo Rota (r00ta)]

I'd say no atm. I think for the time being you can disable the `30-maas-01-bmc-config` commissioning script and create a new one like `31-maas-01-bmc-config-custom` with the updated content. Would that work for you?

[laurent wandrebeck (laurentwandrebeck)]

It looks like I can’t disable 30-maas-01-bmc-config as I’m running maas with snap and thus bmc_config.py can’t be edited (to change tags to noauto).
Is there any other way ?

[laurent wandrebeck (laurentwandrebeck)]

something like update metadataserver_script set default=false where name='30-maas-01-bmc-config'; in maas database maybe ?

[Jacopo Rota (r00ta)]

Nope, just use the checkbox "disable BMC config" in the UI when you commission a machine

[laurent wandrebeck (laurentwandrebeck)]

OK, I didn’t think about it at first.
Commissioning fails at 1st try with default script.
Thing is machine is in « Unconfigured power type » state after that, which is quite annoying, we need to get to manual, then commission again and go to the machine to power it up again.
I’ve toyed with database so 30-maas-01-bmc-config is now default=false and 31-maas-01-bmc-config-custom is default=true in metadataserver_script table.
Now it runs directly the custom script and not 30-maas-01-bmc-config, which is a far better user experience.
Given it’s not a so clean solution, I’d say backporting that patch to 3.3/3.4 would be nice for everyone.

Thanks !