Controller in HA stays down after upgrade

[Joseph Phillips]

After upgrading a controller (using --build agent) in a running HA set-up, one of the agents remained down.

This ended up being because the API server worker was not able to come up. The error occurring repeatedly was as follows:

2022-12-14 12:58:12 DEBUG juju.worker.dependency engine.go:618 "api-server" manifold worker stopped: duplicate metrics collector registration attempted
stack trace:
duplicate metrics collector registration attempted
github.com/juju/juju/worker/apiserver.ManifoldConfig.start:211:
2022-12-14 12:58:12 ERROR juju.worker.dependency engine.go:695 "api-server" manifold worker returned unexpected error: duplicate metrics collector registration attempted

[Joseph Phillips (manadart)]

Upon restarting the agent, it restored operation.

[Simon Richardson (simonrichardson)]

There was a proposal sometime ago, that we would unregister every time before we did a new register. Although that seems overkill, we could do a pattern where you always defer an unregister and just ignore the error if there was one, similar to the one in worker/modlecache/worker.go:201 (was you that did that), then at least we're always unregistering.