raft backstop cannot recover from deleted raft directory

[John A Meinel]

If you ever delete the 'raft/' directory, the raft worker is unable to recover. We already have the raft-backstop worker, which can handle when there is only a single machine in the server configuration, and it doesn't have voting rights.
However, that worker *doesn't* handle when the configuration has *no* servers, as can happen if the raft/ directory is completely removed. (Or during a restore operation where the controller that was bootstrapped is being replaced by a new identity controller that didn't exist in the old config.)

It seemed the easiest way would be to handle removal of the raft/ directory, as that is generally going to aid robustness.

The particular issue is here:
localServer := raftServers[w.config.LocalID]
if localServer == nil {
      return nil
}

that doesn't account for len(raftServers) == 0.

[John A Meinel (jameinel)]

this probably won't land for 2.5.4 but we don't have a 2.5.5 target opened up yet.

[John A Meinel (jameinel)]

https://github.com/juju/juju/pull/9987 is a PR for 2.5

[John A Meinel (jameinel)]

https://github.com/juju/juju/pull/9996/ merged this into develop