Power Action error in web UI even though Power Actions work

[Jeff Lane ]

See attached screen shot.

MAAS 2.6.0 in use.

A system was powered down this afternoon to replace some hardware. During that time, MAAS reported errors while trying to get power status per these messages from maas.log:

2019-07-11T16:33:07.993788+00:00 lxc-maas maas.power: [error] lenovo-sr650: Power state could not be queried: Connection timed out while performing power action. Check BMC configuration and connectivity and try again.
2019-07-11T16:33:08.031904+00:00 lxc-maas maas.power: [error] lenovo-sr650: Could not query power state: Connection timed out while performing power action. Check BMC configuration and connectivity and try again..
2019-07-11T16:38:22.919993+00:00 lxc-maas maas.power: [error] lenovo-sr650: Power state could not be queried: Connection timed out while performing power action. Check BMC configuration and connectivity and try again.
2019-07-11T16:38:22.937029+00:00 lxc-maas maas.power: [error] lenovo-sr650: Could not query power state: Connection timed out while performing power action. Check BMC configuration and connectivity and try again..
2019-07-11T16:43:38.018754+00:00 lxc-maas maas.power: [error] lenovo-sr650: Power state could not be queried: Connection timed out while performing power action. Check BMC configuration and connectivity and try again.
2019-07-11T16:43:38.038695+00:00 lxc-maas maas.power: [error] lenovo-sr650: Could not query power state: Connection timed out while performing power action. Check BMC configuration and connectivity and try again..
2019-07-11T16:48:52.965718+00:00 lxc-maas maas.power: [error] lenovo-sr650: Power state could not be queried: Connection timed out while performing power action. Check BMC configuration and connectivity and try again.
2019-07-11T16:48:52.984173+00:00 lxc-maas maas.power: [error] lenovo-sr650: Could not query power state: Connection timed out while performing power action. Check BMC configuration and connectivity and try again..
2019-07-11T16:54:07.950557+00:00 lxc-maas maas.power: [error] lenovo-sr650: Power state could not be queried: Connection timed out while performing power action. Check BMC configuration and connectivity and try again.
2019-07-11T16:54:07.967037+00:00 lxc-maas maas.power: [error] lenovo-sr650: Could not query power state: Connection timed out while performing power action. Check BMC configuration and connectivity and try again..
2019-07-11T16:59:22.926991+00:00 lxc-maas maas.power: [error] lenovo-sr650: Power state could not be queried: Connection timed out while performing power action. Check BMC configuration and connectivity and try again.
2019-07-11T16:59:22.942393+00:00 lxc-maas maas.power: [error] lenovo-sr650: Could not query power state: Connection timed out while performing power action. Check BMC configuration and connectivity and try again..

Now, once the system was reconnected, the logs no longer complain about power state:
2019-07-11T20:10:31.067405+00:00 lxc-maas maas.interface: [info] enp6s0f1 (physical) on lenovo-sr650: IP address automatically unlinked: None:type=AUTO
2019-07-11T20:10:31.108489+00:00 lxc-maas maas.interface: [info] enp0s20f0u1u6 (physical) on lenovo-sr650: IP address automatically unlinked: None:type=STICKY
2019-07-11T20:10:31.126960+00:00 lxc-maas maas.interface: [info] enp6s0f0 (physical) on lenovo-sr650: IP address automatically unlinked: None:type=STICKY
2019-07-11T20:10:31.161791+00:00 lxc-maas maas.interface: [info] eno1 (physical) on lenovo-sr650: IP address automatically unlinked: None:type=STICKY
2019-07-11T20:10:31.192260+00:00 lxc-maas maas.interface: [info] eno2 (physical) on lenovo-sr650: IP address automatically unlinked: None:type=STICKY
2019-07-11T20:10:31.222243+00:00 lxc-maas maas.interface: [info] eno3 (physical) on lenovo-sr650: IP address automatically unlinked: None:type=STICKY
2019-07-11T20:10:31.251686+00:00 lxc-maas maas.interface: [info] eno4 (physical) on lenovo-sr650: IP address automatically unlinked: None:type=STICKY
2019-07-11T20:10:31.281061+00:00 lxc-maas maas.interface: [info] enp88s0f0 (physical) on lenovo-sr650: IP address automatically unlinked: None:type=STICKY
2019-07-11T20:10:31.307825+00:00 lxc-maas maas.interface: [info] enp88s0f1 (physical) on lenovo-sr650: IP address automatically unlinked: None:type=STICKY
2019-07-11T20:10:31.334676+00:00 lxc-maas maas.node: [info] lenovo-sr650: Status transition from READY to COMMISSIONING
2019-07-11T20:10:31.615850+00:00 lxc-maas maas.power: [info] Changing power state (on) of node: lenovo-sr650 (qyw7c3)
2019-07-11T20:10:31.619525+00:00 lxc-maas maas.node: [info] lenovo-sr650: Commissioning started
2019-07-11T20:10:36.282625+00:00 lxc-maas maas.power: [info] Changed power state (on) of node: lenovo-sr650 (qyw7c3)
2019-07-11T20:30:04.959931+00:00 lxc-maas maas.node: [info] lenovo-sr650: Storage layout was set to flat.
2019-07-11T20:30:05.465162+00:00 lxc-maas maas.node: [info] lenovo-sr650: Status transition from COMMISSIONING to TESTING
2019-07-11T20:30:34.058882+00:00 lxc-maas maas.node: [info] lenovo-sr650: Status transition from TESTING to READY
2019-07-11T20:31:48.755523+00:00 lxc-maas maas.node: [info] lenovo-sr650: Status transition from READY to ALLOCATED
2019-07-11T20:31:48.775061+00:00 lxc-maas maas.node: [info] lenovo-sr650: allocated to user bladernr
2019-07-11T20:31:49.741065+00:00 lxc-maas maas.interface: [info] Allocated automatic IP address 10.193.36.10 for ens3f0 (physical) on lenovo-sr650.
2019-07-11T20:31:49.821469+00:00 lxc-maas maas.node: [info] lenovo-sr650: Status transition from ALLOCATED to DEPLOYING
2019-07-11T20:31:49.925861+00:00 lxc-maas maas.power: [info] Changing power state (on) of node: lenovo-sr650 (qyw7c3)
2019-07-11T20:31:54.664209+00:00 lxc-maas maas.power: [info] Changed power state (on) of node: lenovo-sr650 (qyw7c3)
2019-07-11T20:48:32.129502+00:00 lxc-maas maas.node: [info] lenovo-sr650: Status transition from DEPLOYING to DEPLOYED

And we are able to commission and deploy the node. However, when looking at it in the web UI, the config tab still displays the big "ERROR" banner across the BMC section (See attached screenshot) EVEN THOUGH the information at the time clearly shows the power state as on, and clicking the Check Power button is successful every time. Also, I've done a commission AND a Deployment and in both cases, the power status indicator changed to correctly show the power state every time.

So I think the issue is just that this error banner is not being cleared out once the error is resolved.

[Rod Smith (rodsmith)]

The original bug report was on MAAS 2.6.0-rc2. I've updated to 2.6.0-7802-g59416a869-0ubuntu1~18.04.1 and the problem persists.

[Aurelien Lourot (aurelien-lourot)]

I think it's just that power error messages won't go away once the underlying issue has been solved. A similar issue can easily be reproduced with a MAAS on KVM setup where the error message "Error: Failed to login to virsh console" won't go away although everything is working fine, i.e. MAAS is able to query power state and power on and off machines.

Steps to reproduce:
1. Create a new KVM VM.
2. Power it on and see that MAAS enlisted it.
3. Go to the MAAS machine configuration, set the power type to Virsh and
   make a typo, e.g. `qemu+ssh://myuser@10.17.17.1/systme` instead of
   "system". See that you get an "Error: Failed to login to virsh
   console." and that the machine ends up in "Power error" state.
4. Finally fix the typo. See that the machine ends up in "Power off"
   state.

Expected result: the error message goes away.
Actual result: the error message stays although that machine can be
powered on and off.

Both in steps 3 and 4 /var/log/maas/maas.log shows:

maas.log:2019-12-19T09:30:33.318778+01:00 0-maas-controller0 maas.power: [error] wanted-crab: Power state could not be queried: Failed to login to virsh console.
maas.log:2019-12-19T09:30:33.334934+01:00 0-maas-controller0 maas.power: [error] wanted-crab: Failed to refresh power state: Failed to login to virsh console.

Affected MAAS version: 2.7.0~beta2 (8193-g.ac0dacf56-0ubuntu1~18.04.1)
Also seen on 2.6

[Jerzy Husakowski (jhusakowski)]

Is this issue reproducible in MAAS 3.2 or later? Power error reporting and the UI have undergone changes since the issue was submitted.

[Jeff Lane  (bladernr)]

No, this particular bug was resolved at some point by a MAAS update.