[3.3.7 candidate] Failed to get external services configurations from rackd.log

[Jeffrey Chang]

From SolQA test 3.3.7-13241-g.2f0fd752c from https://solutions.qa.canonical.com/testruns/2a19dd1c-e02a-4cfb-a185-1cad0f7497ed

From console log
2024-05-09-06:11:12 root DEBUG [localhost]: maas root rack-controllers read
2024-05-09-06:11:18 foundationcloudengine.layers.configuremaas INFO Controller infra1: Commissioning status is Failed
Traceback (most recent call last):
  File "/usr/local/bin/fce", line 33, in <module>
    sys.exit(load_entry_point('foundationcloudengine', 'console_scripts', 'fce')())
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/main.py", line 217, in entry_point
    sys.exit(main(sys.argv[1:]))
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/main.py", line 208, in main
    opts.func(opts)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/command.py", line 91, in run
    self.run_per_layer(layer, args)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/commands/build.py", line 46, in run_per_layer
    build_and_validate_if_needed(
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/commands/build.py", line 98, in build_and_validate_if_needed
    layer.build_outer(only_steps)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/baselayer.py", line 160, in build_outer
    self.build(only_steps=only_steps)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/maaslayer.py", line 2084, in build
    super().run_steps(only_steps)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/steppedbaselayer.py", line 110, in run_steps
    step.build()
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/configuremaas.py", line 232, in build
    raise Exception("Wrong commissioning status")
Exception: Wrong commissioning status

from rackd.log
2024-05-09 06:09:56 provisioningserver.rackdservices.external: [critical] Failed to get external services configurations.
 Traceback (most recent call last):
   File "/snap/maas/35360/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 661, in callback
     self._startRunCallbacks(result)
   File "/snap/maas/35360/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
     self._runCallbacks()
   File "/snap/maas/35360/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
     current.result = callback( # type: ignore[misc]
   File "/snap/maas/35360/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 360, in call
     return maybeDeferred(func, *args, **kwargs)
 --- <exception caught here> ---
   File "/snap/maas/35360/lib/python3.10/site-packages/provisioningserver/rackdservices/external.py", line 461, in _tryUpdate
     config = yield self._getConfiguration()
   File "/snap/maas/35360/lib/python3.10/site-packages/provisioningserver/rackdservices/external.py", line 432, in _getConfiguration
     client = yield self._rpc_service.getClientNow()
   File "/snap/maas/35360/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 190, in maybeDeferred
     result = f(*args, **kwargs)
   File "/snap/maas/35360/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1241, in getClient
     raise e
   File "/snap/maas/35360/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1232, in getClient
     self.connections.get_random_free_connection()
   File "/snap/maas/35360/lib/python3.10/site-packages/provisioningserver/rpc/connectionpool.py", line 109, in get_random_free_connection
     raise exceptions.AllConnectionsBusy()
 provisioningserver.rpc.exceptions.AllConnectionsBusy:

[Björn Tillenius (bjornt)]

Looking at the logs, it seems to be bug 2052958.

The fix was backported to 3.4, but not to 3.3.