snap deployed maas is not able to use openstack nova power type due to missing python3-novaclient dependency

[Victor Vavan]

This can be easily reproduced with a snap deployed maas environment (tested on versions 3.1 and 3.3). Attempt to add a machine with the `OpenStack Nova` power type, OpenStack instance UUID, tenant name, username, password, and Auth URL. After attaching the machine and clicking "Check Power", the status will change to "Power Error" and you will see the following in /var/snap/maas/common/log/regiond.log:

maasserver.websockets.handlers.machine: [critical] Failed to update power state of machine.
Traceback (most recent call last):
File "/snap/maas/26273/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1475, in gotResult
_inlineCallbacks(r, g, status)
File "/snap/maas/26273/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1464, in _inlineCallbacks
status.deferred.errback()
File "/snap/maas/26273/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in errback
self._startRunCallbacks(fail)
File "/snap/maas/26273/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/snap/maas/26273/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 654, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/snap/maas/26273/lib/python3.8/site-packages/maasserver/websockets/handlers/machine.py", line 1109, in eb_unknown
failure.trap(UnknownPowerType, NotImplementedError)
File "/snap/maas/26273/usr/lib/python3/dist-packages/twisted/python/failure.py", line 439, in trap
self.raiseException()
File "/snap/maas/26273/usr/lib/python3/dist-packages/twisted/python/failure.py", line 467, in raiseException
raise self.value.with_traceback(self.tb)
File "/snap/maas/26273/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "/snap/maas/26273/lib/python3.8/site-packages/maasserver/models/node.py", line 3294, in confirm_power_driver_operable
raise PowerActionFail(
provisioningserver.rpc.exceptions.PowerActionFail: Power control software is missing from the rack controller 'maas.maas'. To proceed, install the python3-novaclient package.

Upon installing the python3-novaclient package on the rack controller (using apt), and rebooting the rack controller, the error persists.

Further, looking into the /snap/maas/current/usr/lib/python3/dist-packages/ directory, the novaclient package is not present.

[Alan Baghumian (alanbach)]

I was able to easily reproduce this. Since this is a snap installing the python3-novaclient package has no effect.

[Björn Tillenius (bjornt)]

Can anyone confirm that the nova power driver works when you run maas 3.3 as a deb (after having installed pythong3-novaclient)?

[Simon Forsayeth (sforsayeth)]

Hi Bjorn,

First up. I am using the snap version of MAAS 3.3, which led me to this bug thread. I am experiencing the same issue.

Second, I can confirm a similar behaviour exists in the deb version of MAAS 3.3. I've created a small deployment on Openstack. There is a separate region controller and separate db host on subnet X, and a single rackd on subnet y. Subnets X / Y are routable between each other. Both subnets can get to our keystone API.

I manually add a new machine to MAAS. It's a test compute node instance on subnet y. The power controller is set to Nova, with all details appropriately set according to my test compute instance / Openstack env.

When attempting to get the power state of my compute test node, I see the following in the Controller UI:

"Failed to query node's BMC - (admin) - No rack controllers can access the BMC of node sf-maas-compute1"

I see the following in /var/log/maas/regiond.log:

2023-03-24 22:40:54 maasserver.websockets.handlers.machine: [critical] Failed to update power state of machine.
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 271, in _onTimer
            self.runUntilCurrent()
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 991, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 700, in errback
            self._startRunCallbacks(fail)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback( # type: ignore[misc]
          File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/machine.py", line 1176, in eb_unknown
            failure.trap(UnknownPowerType, NotImplementedError)
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
            self.raiseException()
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
            result = inContext.theWork() # type: ignore[attr-defined]
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
            inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
            return func(*args, **kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 857, in callInContext
            return func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/pro...

[Jerzy Husakowski (jhusakowski)]

The nova power driver has been contributed to MAAS. It's not part of the CI tests because there's no OpenStack available for such tests to the MAAS team. Therefore, this code is prone to regression and falling behind nova API.

We're happy to accept a fix, but have no way of investigating this further ourselves.

[Jerzy Husakowski (jhusakowski)]

In order to move this forward, we'll add the missing dependency, but don't have sufficient information to ensure that the issue with this driver is completely gone.

[Ioanna Alifieraki (joalif)]

I have an environment setup (thanks to victorvavan) where I can reproduce the issue.
When installing maas via snap, even if you 'apt install python3-novaclient' the problem
won't go away because of snap confinement, in practice maas cannot see the installed package.

I've installed maas via debs. If the python3-novaclient is present, the message
"Power control software is missing from the rack controller 'maas.maas'. To proceed, install the python3-novaclient package."
goes away but it still fails.

It turns out that, maas openstack driver was written a while back and the novaclient api has changed.
The problem is in this line :
https://git.launchpad.net/maas/tree/src/provisioningserver/drivers/power/nova.py?h=3.1#n81
nova.authenticate()
This method has been deprecated.

The authentication will happen automatically at the first call to the server, but the call
to the novaclient.Client() need to be modified like this:

diff --git a/src/provisioningserver/drivers/power/nova.py b/src/provisioningserver/drivers/power/nova.py
index 039195317..12934947c 100644
--- a/src/provisioningserver/drivers/power/nova.py
+++ b/src/provisioningserver/drivers/power/nova.py
@@ -73,7 +73,7 @@ class NovaPowerDriver(PowerDriver):
         if not self.try_novaapi_import():
             raise PowerToolError("Missing the python3-novaclient package.")
         nova = self.nova_api.Client(
- 2, os_username, os_password, os_tenantname, os_authurl
+ version="2", username="os_username", password="os_password", project_id=os_tenantname, auth_url=os_authurl, user_domain_name=<user-domain-name>
         )

Hardcoding the the user_domain_name for my environment, the "check power" does not throw errors.

At this point, I see 2 problems:

1. Authentication related
To authenticate we need the user_domain_name. I am not an openstack expert and I'm not sure if we can avoid it, but if we can't
that would imply further changes to maas (and maas ui) for the user to provide the user_domain_name.

2. Commissioning
Even if we go past the power on errors, when it gets to the commissioning stage it seems it gets stuck.

[Jerzy Husakowski (jhusakowski)]

As discussed - Openstack nova driver is currently unmaintained. Our advice is to use virtualBMC in OpenStack and point MAAS at that. Alternatively, use the webhook driver.