No feedback to client when Juju HA controller lost a quorum

[Nobuto Murata]

According to the actual behavior, Juju stops listening tcp port 17070 when the HA controllers lost a quorum. Then, juju status command just times out.

Expected result is explicit feedback with a message like:

$ juju status
API is disabled due to missing a quorum. For help see <URL>

with a document describing how to promote the remaining controller as the master to continue operation if a user is pretty sure the one has the latest database.

Actual:

$ juju status --debug
15:02:12 INFO juju.cmd supercommand.go:57 running juju [2.5.1 gc go1.10.4]
15:02:12 DEBUG juju.cmd supercommand.go:58 args: []string{"/snap/juju/6629/bin/juju", "status", "--debug"}
15:02:12 INFO juju.juju api.go:67 connecting to API addresses: [10.0.9.157:17070 10.0.9.189:17070 10.0.9.118:17070]
-> timeout

How to reproduce:

$ juju bootstrap localhost
$ juju enable-ha

kill two of three controllers.
$ lxc stop -f juju-<controller_model>-0
$ lxc stop -f juju-<controller_model>-0

$ juju status
-> timeout

[Nobuto Murata (nobuto)]

juju_engine_report.yaml after killing two of three controllers.

[John A Meinel (jameinel)]

If you lose Quorum we probably fail to even bring up the API server at all, to report anything to the user.
We could probably bring up something, though.

[John A Meinel (jameinel)]

(there are a lot of potential reasons why we cannot, like not being able to authorize because the backend database cannot give us strong consistency, etc.) But you could put up some sort of static error response.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.