Comment 1 for bug 2016868

Joe and I dug into this for a bit today. It looks like what is happening is that we start with HA 3, and machine-0,-1,-2 are all happily in the replicaset.
However, when we 'juju remove-machine 0', it goes into dying, and then the replicaset notices that it wants to remove machine 0, so it kicks off the process to have machine 0 lose its primary status (which it does, and in this case it moved to 1).
At which point, we notice that we can remove machine 0 safely, and that ultimately we want to go to HA 2, where we have only 1 voter, and another non-voter backup.

So we end up going from:
{
  0: true,
  1: true,
  2: true,
}

to
{
  1: true,
  2: false,
}

However, Mongo refuses that change, because you are changing 2 voters simultaneously (machine 0 is going from voting to non-voting and removed, and machine-2 is going to non-voting.)
If we logged into the mongo shell, and directly changed only machine-0 so that it was no longer in the replicaset, the rest of the peergrouper worker kicked off and we ended up in the stable desired target. (To prove this, I suppose we could try the opposite and only downgrade machine-2 to non-voting and see if the peergrouper successfully removes machine-0).

I think the logic that we have to 'only do one change at a time' is missing something about handling removals.

The other piece is that when we tried to issue the request for removal and downgrade directly on the mongodb commandline, we got an error from mongo. For whatever reason, the peergrouper is not reporting an error in the logs. So somehow we are suppressing the error.
It may be that we are using 'force' and mongo isn't erroring, but also isn't respecting our request. Or some code path where we fail to notice the response from mongo.