raft metrics missing from some controllers

[Christian Muirhead]

We only see internal raft metrics collected from some controllers. From the engine report I can see that the raft worker has only started once on the controller that we're getting stats from - the others have higher counts (presumably because of state worker restarts due to mongo i/o timeouts).

[Christian Muirhead (2-xtian)]

To get the raft internal metrics we hook our prometheus collector up to the metrics library hashicorp/raft uses internally. It turns out that the armon/go-metrics prometheus integration automatically registers the collector it returns with the default global collector registry, even though we don't use it. That means that metrics registration fails when the worker is started a second time, even though we make sure to unregister it from our Juju-wide registry.

[Christian Muirhead (2-xtian)]

PR for fix: https://github.com/juju/juju/pull/9691