Comment 3 for bug 2009209

Hi Bjorn,

First up. I am using the snap version of MAAS 3.3, which led me to this bug thread. I am experiencing the same issue.

Second, I can confirm a similar behaviour exists in the deb version of MAAS 3.3. I've created a small deployment on Openstack. There is a separate region controller and separate db host on subnet X, and a single rackd on subnet y. Subnets X / Y are routable between each other. Both subnets can get to our keystone API.

I manually add a new machine to MAAS. It's a test compute node instance on subnet y. The power controller is set to Nova, with all details appropriately set according to my test compute instance / Openstack env.

When attempting to get the power state of my compute test node, I see the following in the Controller UI:

"Failed to query node's BMC - (admin) - No rack controllers can access the BMC of node sf-maas-compute1"

I see the following in /var/log/maas/regiond.log:

2023-03-24 22:40:54 maasserver.websockets.handlers.machine: [critical] Failed to update power state of machine.
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 271, in _onTimer
            self.runUntilCurrent()
          File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 991, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 700, in errback
            self._startRunCallbacks(fail)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback( # type: ignore[misc]
          File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/machine.py", line 1176, in eb_unknown
            failure.trap(UnknownPowerType, NotImplementedError)
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
            self.raiseException()
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
            result = inContext.theWork() # type: ignore[attr-defined]
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
            inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
            return func(*args, **kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 857, in callInContext
            return func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
            result = func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 5639, in _get_bmc_client_connection_info
            raise PowerProblem(err_msg)
        maasserver.exceptions.PowerProblem: No rack controllers can access the BMC of node sf-maas-compute1

I first added 'python3-novaclient' to the rackd node. No change. Same error
Then I added 'python3-novaclient' to the region controller. No change. Same error.

So I figured maybe test it from the Region Controller point of view. The next test was that I added in the Nova power settings for my Rackd controller. Then checked the power status from the Controller UI by attempting to power off / power on my Rackd controller.

From /var/log/maas/regiond.log on the Region Controller:

2023-03-24 22:54:10 maasserver.websockets.protocol: [critical] Error on request (15) controller.action: Power control software is missing from the rack controller 'sf-maas-rackd.maas'. To proceed, install the python3-novaclient package.
        Traceback (most recent call last):
          File "/usr/lib/python3.10/threading.py", line 953, in run
            self._target(*self._args, **self._kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 822, in worker
            return target()
          File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 47, in work
            task()
          File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 182, in doWork
            task()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
            result = inContext.theWork() # type: ignore[attr-defined]
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
            inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
            return func(*args, **kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 857, in callInContext
            return func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
            result = func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 770, in call_within_transaction
            with connected(), post_commit_hooks:
          File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 636, in __exit__
            self.fire()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
            result = func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/maasserver/utils/asynchronous.py", line 207, in fire
            self._fire_in_reactor(hook).wait(LONGTIME)
          File "/usr/lib/python3/dist-packages/crochet/_eventloop.py", line 198, in wait
            result.raiseException()
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
            result = current_context.run(gen.send, result)
          File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 3349, in confirm_power_driver_operable
            raise PowerActionFail(
        provisioningserver.rpc.exceptions.PowerActionFail: Power control software is missing from the rack controller 'sf-maas-rackd.maas'. To proceed, install the python3-novaclient package.

Installing 'python3-novaclient' on both the Region Controller and Rackd (just because), this error goes away, but nothing actually happens.

It does not appear that the Nova power control method is working at all in either snap or deb.

Simon