HA tests fail after the leader is deleted

[Curtis Hovey]

As seen in
    http://reports.vapour.ws/releases/issue/5762cc04749a5666b4caf038

Juju CI started seeing systemic HA failures about Friday Nov 3. The failures were in AWS, Rackspace, Azure, and Prodstack. The failures were across trusty and xenial controllers. After the leader is deleted, juju status fails. We expect juju to fall back to another controller.

The first signs of the problem are in
    http://reports.vapour.ws/releases/4560
    http://reports.vapour.ws/releases/4561

After a change to the test to ensure lingering procs, we see a consistent timeout waiting for a new controller to answer the clients call to status.
    http://reports.vapour.ws/releases/4563

[Curtis Hovey (sinzui)]

While a prodstack test was failing, I was able to confirm on the machine that juju status failed. I see the three controllers that "nova list" show in the api-endpoints in JUJU_DATA dir setup for the test.

$ cat /var/lib/jenkins/cloud-city/jes-homes/functional-ha-recovery-prodstack/controllers.yaml
controllers:
  functional-ha-recovery-prodstack:
    unresolved-api-endpoints: ['10.25.28.67:17070', '10.25.28.7:17070', '10.25.28.70:17070']
    uuid: 60807685-8236-4984-8d91-7be04f1e4732
    api-endpoints: ['10.25.28.67:17070', '10.25.28.7:17070', '10.25.28.70:17070']
    ca-cert: |
      -----BEGIN CERTIFICATE-----
      CERT
      -----END CERTIFICATE-----
    cloud: prodstack45
    region: bootstack-ps45
    agent-version: 2.1-beta1
    controller-machine-count: 1
    active-controller-machine-count: 0
    model-count: 2
    machine-count: 1
current-controller: functional-ha-recovery-prodstack

[Richard Harding (rharding)]

+1 to it not working in GCE. In testing the client does realize the master is gone and rotates pinging the other two nodes. However, on those nodes they don't end up picking up the mongodb primary.

I restarted juju-db on one of the other two controllers and got the following failure to get mongodb happy again:

https://pastebin.canonical.com/170629/

[Menno Finlay-Smits (menno.smits)]

The first thing to establish is whether this is mongodb failing to elect a new master or juju not handling the failover correctly. My guess is that it's a Juju issue, but MongoDB 3.2 does use a new consensus protocol so there could be a problem there too.

In order to check that the remaining nodes elect a new leader once the previous leader is killed, please check the output of "rs.status()" in the mongo shell on each of the remaining nodes.

Up to date details for getting a mongo shell are here: https://github.com/juju/juju/wiki/Login-into-MongoDB

[Horacio Durán (hduran-8)]

The culprit is juju and almost certainly https://github.com/juju/juju/commit/b02339f16e10ae472bcbf646846f566a79dee5e7

[Curtis Hovey (sinzui)]

Horacio has a fix at https://github.com/juju/juju/pull/6652

[Horacio Durán (hduran-8)]

I proposed https://github.com/juju/juju/pull/6652