configure-resources actions ends in error

[Marian Gasparovic]

Post deploy configure-resources action failed. It passed on at least two other runs.

2020-08-18-13:14:31 root DEBUG [localhost]: juju run-action -m foundations-maas:openstack --format=yaml octavia/leader configure-resources
2020-08-18-13:14:32 root DEBUG [localhost]: juju show-action-output -m foundations-maas:openstack --format=yaml --wait 600 34
2020-08-18-13:15:26 root DEBUG action output: {'UnitId': 'octavia/2', 'id': '34', 'message': "Unbable to set quotas to unlimited: ('nova', 'quotas', BadRequest())", 'results': {'Stderr': 'Warning: Stopping systemd-networkd.service, but it can still be activated by:\n systemd-networkd.socket\n', 'Stdout': 'active\nactive\nactive\nactive\n'}, 'status': 'failed', 'timing': {'completed': '2020-08-18 13:15:25 +0000 UTC', 'enqueued': '2020-08-18 13:14:32 +0000 UTC', 'started': '2020-08-18 13:14:32 +0000 UTC'}}
2020-08-18-13:15:26 root ERROR action output: {'UnitId': 'octavia/2', 'id': '34', 'message': "Unbable to set quotas to unlimited: ('nova', 'quotas', BadRequest())", 'results': {'Stderr': 'Warning: Stopping systemd-networkd.service, but it can still be activated by:\n systemd-networkd.socket\n', 'Stdout': 'active\nactive\nactive\nactive\n'}, 'status': 'failed', 'timing': {'completed': '2020-08-18 13:15:25 +0000 UTC', 'enqueued': '2020-08-18 13:14:32 +0000 UTC', 'started': '2020-08-18 13:14:32 +0000 UTC'}}

[Michael Skalka (mskalka)]

Seen again in this release test run: https://solutions.qa.canonical.com/testruns/testRun/cefecb46-4fcb-439f-ab43-3c15c1928b28

Crashdump here: https://oil-jenkins.canonical.com/artifacts/cefecb46-4fcb-439f-ab43-3c15c1928b28/generated/generated/openstack/juju-crashdump-openstack-2020-10-10-06.21.08.tar.gz

[Bas de Bruijne (basdbruijne)]

We are seeing this again here: https://solutions.qa.canonical.com/v2/testruns/c26f6331-7387-4938-8921-9c09302d8087/

Lots of python errors in the crashdump, e.g.:
```
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils Traceback (most recent call last):
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/cotyledon/_utils.py", line 95, in exit_on_exception
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils yield
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/cotyledon/_service.py", line 139, in _run
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils self.run()
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/octavia/controller/queue/v1/consumer.py", line 47, in run
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils self.message_listener.start()
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in wrapper
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils states[state].run_once(lambda: fn(self, *args, **kwargs),
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 188, in run_once
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils post_fn = fn()
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in <lambda>
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils states[state].run_once(lambda: fn(self, *args, **kwargs),
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 413, in start
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils self.listener = self._create_listener()
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 150, in _create_listener
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils return self.transport._listen(self._target, 1, None)
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 139, in _listen
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils raise exceptions.InvalidTarget('A server\'s target must have '
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils oslo_messaging.exceptions.InvalidTarget: A server's target must have topic and server names specified:<Target server=juju-c4a296-4-lxd-9>
```

Logs can be found here:
https://oil-jenkins.canonical.com/artifacts/c26f6331-7...

[Felipe Reyes (freyes)]

I was taking a look to the logs available at https://oil-jenkins.canonical.com/artifacts/c26f6331-7387-4938-8921-9c09302d8087/generated/generated/openstack/juju-crashdump-openstack-2022-10-25-14.15.19.tar.gz , and the error posted is misleading, this is being printed before config-changed has generated the configuration, after those errors it can be seen how octavia-worker started running successfully

2022-10-25 13:06:54.321 96392 ERROR cotyledon._utils oslo_messaging.exceptions.InvalidTarget: A server's target must have topic and server names specified:<Target server=juju-c4a296-3-lxd-8>
2022-10-25 13:06:54.321 96392 ERROR cotyledon._utils
2022-10-25 13:07:15.463 96579 INFO octavia.common.config [-] Logging enabled!
2022-10-25 13:07:15.463 96579 INFO octavia.common.config [-] /usr/bin/octavia-worker version 10.0.0
2022-10-25 13:07:15.543 97301 INFO octavia.controller.queue.v1.consumer [-] Starting consumer...
2022-10-25 13:07:15.556 97304 INFO octavia.controller.queue.v2.consumer [-] Starting V2 consumer...
2022-10-25 13:07:26.841 96579 INFO cotyledon._service_manager [-] Caught SIGTERM signal, graceful exiting of master process
2022-10-25 13:07:26.842 97301 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(0) [97301]
2022-10-25 13:07:26.842 97301 INFO octavia.controller.queue.v1.consumer [-] Stopping consumer...
2022-10-25 13:07:26.843 97304 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(0) [97304]
2022-10-25 13:07:26.844 97304 INFO octavia.controller.queue.v2.consumer [-] Stopping V2 consumer...
2022-10-25 13:07:26.908 97304 INFO octavia.controller.queue.v2.consumer [-] V2 Consumer successfully stopped. Waiting for final messages to be processed...
2022-10-25 13:07:26.919 97304 WARNING amqp [-] Received method (60, 30) during closing channel 1. This method will be ignored
2022-10-25 13:07:26.922 97304 INFO octavia.controller.queue.v2.consumer [-] Shutting down V2 endpoint worker executors...
2022-10-25 13:07:26.928 97301 INFO octavia.controller.queue.v1.consumer [-] Consumer successfully stopped. Waiting for final messages to be processed...
2022-10-25 13:07:26.937 97301 WARNING amqp [-] Received method (60, 30) during closing channel 1. This method will be ignored
2022-10-25 13:07:26.939 97301 INFO octavia.controller.queue.v1.consumer [-] Shutting down endpoint worker executors...
2022-10-25 13:07:33.040 97444 INFO octavia.common.config [-] Logging enabled!
2022-10-25 13:07:33.041 97444 INFO octavia.common.config [-] /usr/bin/octavia-worker version 10.0.0
2022-10-25 13:07:33.209 97654 INFO octavia.controller.queue.v1.consumer [-] Starting consumer...
2022-10-25 13:07:33.230 97662 INFO octavia.controller.queue.v2.consumer [-] Starting V2 consumer...
2022-10-25 13:09:01.537 97444 INFO cotyledon._service_manager [-] Caught SIGTERM signal, graceful exiting of master process
2022-10-25 13:09:01.538 97654 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(0) [97654]
2022-10-25 13:09:01.538 97662 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(0) [97662]
2022...

[Abdullah (motjuste)]

We have another run that seems to have the same issue: https://solutions.qa.canonical.com/testruns/8b025c31-8c68-4491-8aa9-568dec286d75/

Other identified occurrences of this bug may be found at:
https://solutions.qa.canonical.com/bugs/1892063