juju upgrade-juju failed to configure mongodb replicasets
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
juju-core |
Fix Released
|
High
|
Menno Finlay-Smits | ||
1.24 |
Fix Released
|
High
|
Menno Finlay-Smits |
Bug Description
We have an environment where juju upgrade-juju failed to configure
replicasets on mongodb correctly leaving the environment without a
jujud on machine 0.
This environment was previously running juju 1.18.3.1 on Ubuntu 14.04
and we were in process of upgrading to 1.20.11. machine 0 had run out
of space (because versions of juju available in an LTS don't rotate
logs - LP #1078213) and the full root partition had left apt in a very
confused state. That in turn left the juju-upgrade in an infinite
loop, retrying apt-get (LP #1441904).
Eventually a human came along, freed up space and unconfused apt.
This unblocked upgrade-juju but it didn't proceed as expected and
mongo was left in a state where replicasets are uninitalized.
Unfortunately, I can't trivial reproduce this if I try in with a fresh
environment. It could be because it's a race of some sort or it could
be because of the specific way in which apt-get was failing (which I
also can't easily reproduce). apt-get was causing juju to say this:
juju.utils.apt apt.go:166 apt-get command failed: unexpected error type *errors.errorString
Thankfully this is a staging environment so we can leave it in a
broken state for as long as you need, if that helps.
tags: | added: mongodb upgrade-juju |
Changed in juju-core: | |
status: | New → Triaged |
importance: | Undecided → High |
milestone: | none → 1.24-alpha1 |
tags: | added: canonical-is |
Changed in juju-core: | |
milestone: | 1.24-alpha1 → 1.25.0 |
Changed in juju-core: | |
assignee: | nobody → Menno Smits (menno.smits) |
Changed in juju-core: | |
status: | Triaged → In Progress |
Changed in juju-core: | |
status: | In Progress → Fix Committed |
Changed in juju-core: | |
status: | Fix Committed → Fix Released |
no longer affects: | juju-core/1.23 |
root@juju- stag-ue- summit- machine- 0:~# mongo --ssl -u admin -p $(grep oldpassword /var/lib/ juju/agents/ machine- 0/agent. conf | awk -e '{print $2}') localhost: 37017/admin 37017/admin shell/query. js:128
"startupStatus " : 3, replset config from self or any seed (EMPTYCONFIG)"
MongoDB shell version: 2.4.9
connecting to: localhost:
> show collections
Thu Apr 9 00:00:02.440 error: { "$err" : "not master and slaveOk=false", "code" : 13435 } at src/mongo/
> rs.status()
{
"info" : "run rs.initiate(...) if not yet done for the set",
"ok" : 0,
"errmsg" : "can't get local.system.
}
>