Model Upgrade 2.7.6 -> 2.8.6 required mongodb rebuild

[Adam Dyess]

I upgraded a controller model today using

`juju upgrade-model -m controller`

before running i confirmed that mongodb was healthy:
```
juju:SECONDARY> rs.status()
{
    "set" : "juju",
    "date" : ISODate("2020-11-02T15:18:31.072Z"),
    "myState" : 2,
    "term" : NumberLong(10),
    "syncingTo" : "172.18.247.83:37017",
    "heartbeatIntervalMillis" : NumberLong(2000),
    "members" : [...],
    "ok" : 1
}
```

during the process I saw that mongodb moved its primary to another controller unit and the original primary mongodb went stale. After about an hour, i decided it wasn't going to catch back up and restored by stopping mongo on the unit, removing it's db, then starting mongo again and seeing it replicate in about 5 minutes.

[Adam Dyess (addyess)]

The entire upgrade of ONLY the controller model took about 90 minutes after I dropped mongo on the first unit and re-replicated.

[Pen Gale (pengale)]

@addyess do you have logs from the mongodb instance that went stale? Both mongodb and Juju logs would be useful. (I'm guessing that we'll see some sort of hung worker, but without the logs, it's hard to know where to start looking.)

[Adam Dyess (addyess)]

I've upload juju and syslogs here:
http://private-fileshare.canonical.com/~addyess/1902587.tar.gz

[Canonical Juju QA Bot (juju-qa-bot)]

This Medium-priority bug has not been updated in 60 days, so we're marking it Low importance. If you believe this is incorrect, please update the importance.