From SolQA test 3.3.7-13241-g.2f0fd752c from https://solutions.qa.canonical.com/testruns/2a19dd1c-e02a-4cfb-a185-1cad0f7497ed
From console log
2024-05-09-06:11:12 root DEBUG [localhost]: maas root rack-controllers read
2024-05-09-06:11:18 foundationcloudengine.layers.configuremaas INFO Controller infra1: Commissioning status is Failed
Traceback (most recent call last):
File "/usr/local/bin/fce", line 33, in <module>
sys.exit(load_entry_point('foundationcloudengine', 'console_scripts', 'fce')())
File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/main.py", line 217, in entry_point
sys.exit(main(sys.argv[1:]))
File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/main.py", line 208, in main
opts.func(opts)
File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/command.py", line 91, in run
self.run_per_layer(layer, args)
File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/commands/build.py", line 46, in run_per_layer
build_and_validate_if_needed(
File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/commands/build.py", line 98, in build_and_validate_if_needed
layer.build_outer(only_steps)
File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/baselayer.py", line 160, in build_outer
self.build(only_steps=only_steps)
File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/maaslayer.py", line 2084, in build
super().run_steps(only_steps)
File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/steppedbaselayer.py", line 110, in run_steps
step.build()
File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/configuremaas.py", line 232, in build
raise Exception("Wrong commissioning status")
Exception: Wrong commissioning status
from rackd.log
2024-05-09 06:09:56 provisioningserver.rackdservices.external: [critical] Failed to get external services configurations.
Traceback (most recent call last):
File "/snap/maas/35360/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 661, in callback
self._startRunCallbacks(result)
File "/snap/maas/35360/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
self._runCallbacks()
File "/snap/maas/35360/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/snap/maas/35360/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 360, in call
return maybeDeferred(func, *args, **kwargs)
--- <exception caught here> ---
File "/snap/maas/35360/lib/python3.10/site-packages/provisioningserver/rackdservices/external.py", line 461, in _tryUpdate
config = yield self._getConfiguration()
File "/snap/maas/35360/lib/python3.10/site-packages/provisioningserver/rackdservices/external.py", line 432, in _getConfiguration
client = yield self._rpc_service.getClientNow()
File "/snap/maas/35360/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 190, in maybeDeferred
result = f(*args, **kwargs)
File "/snap/maas/35360/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1241, in getClient
raise e
File "/snap/maas/35360/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1232, in getClient
self.connections.get_random_free_connection()
File "/snap/maas/35360/lib/python3.10/site-packages/provisioningserver/rpc/connectionpool.py", line 109, in get_random_free_connection
raise exceptions.AllConnectionsBusy()
provisioningserver.rpc.exceptions.AllConnectionsBusy:
Looking at the logs, it seems to be bug 2052958.
The fix was backported to 3.4, but not to 3.3.