Solutions QA has a failed run in which designate-bind/0 stays in unknown state (agent lost) with the message: agent lost, see 'juju show-status-log designate-bind/0'
The failure occurs during deployment of the openstack layer, on the deploy_bundle step. The test deploys MAAS on Jammy, then bootstraps a Juju 3.1.6 controller, which is then used to deploy openstack yoga.
From the logs, we notice that connection with Juju controller is lost permanently:
2023-09-28 06:04:01 DEBUG juju.worker.dependency engine.go:618 "migration-inactive-flag" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-09-28 06:04:01 DEBUG juju.worker.dependency engine.go:618 "migration-minion" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-09-28 06:04:01 DEBUG juju.worker.dependency engine.go:618 "api-address-updater" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-09-28 06:04:01 DEBUG juju.worker.dependency engine.go:618 "meter-status" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-09-28 06:04:01 DEBUG juju.worker.dependency engine.go:618 "logging-config-updater" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-09-28 06:04:01 DEBUG juju.worker.dependency engine.go:618 "hook-retry-strategy" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-09-28 06:04:01 DEBUG juju.worker.dependency engine.go:618 "secret-drain-worker" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-09-28 06:04:01 DEBUG juju.worker.dependency engine.go:603 "migration-fortress" manifold worker completed successfully
2023-09-28 06:04:01 DEBUG juju.worker.dependency engine.go:618 "uniter" manifold worker stopped: connection is shut down
stack trace:
github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1261:
2023-09-28 06:04:01 DEBUG juju.worker.dependency engine.go:603 "api-caller" manifold worker completed successfully
2023-09-28 06:04:04 INFO juju unit_agent.go:289 Starting unit workers for "designate-bind/0"
2023-09-28 06:04:04 INFO juju.agent.setup agentconf.go:128 setting logging config to "<root>=DEBUG;juju.worker.deployer=TRACE"
2023-09-28 06:04:04 DEBUG juju.worker.dependency engine.go:580 "agent" manifold worker started at 2023-09-28 06:04:04.840867351 +0000 UTC
2023-09-28 06:04:04 DEBUG juju.worker.dependency engine.go:580 "migration-fortress" manifold worker started at 2023-09-28 06:04:04.840934777 +0000 UTC
2023-09-28 06:04:04 DEBUG juju.worker.dependency engine.go:580 "api-config-watcher" manifold worker started at 2023-09-28 06:04:04.850955235 +0000 UTC
2023-09-28 06:04:04 DEBUG juju.worker.apicaller connect.go:116 connecting with current password
2023-09-28 06:04:04 DEBUG juju.worker.apicaller connect.go:129 connecting with old password
2023-09-28 06:04:04 DEBUG juju.worker.apicaller connect.go:160 [7f42c2] failed to connect
2023-09-28 06:04:04 ERROR juju.worker.apicaller connect.go:209 Failed to connect to controller: invalid entity name or password (unauthorized access)
2023-09-28 06:04:04 DEBUG juju.worker.dependency engine.go:618 "api-caller" manifold worker stopped: [7f42c2] "unit-designate-bind-0" cannot open api: connection permanently impossible
stack trace:
github.com/juju/juju/worker/apicaller.init:42: connection permanently impossible
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [7f42c2] "unit-designate-bind-0" cannot open api
Failed run: https://solutions.qa.canonical.com/testruns/7af34bad-8d5f-4bd7-9ddd-af29563965d7
Logs: https://oil-jenkins.canonical.com/artifacts/7af34bad-8d5f-4bd7-9ddd-af29563965d7/index.html
I'm noticing that this can also happen after the initial installation, which isn't tested extensively by SQA yet. I'm working on an OpenStack deployment that installs like it should, but then after a day or so all the services break due to this error.