Restore backup failed (unauthorized mongo access)

[Curtis Hovey]

As seen at
    http://qa.jujucharms.com/releases/issue/564c9f52749a564c5f12e153

An ambiguous case where Juju didn't bring up a replacement bootstrap instance. In debug we can see

06:38:51 ERROR juju.api.backups restore.go:136 could not clean up after failed restore attempt: cannot complete restore: <nil>: juju restore is in progress - API is disabled to prevent data loss
06:38:51 ERROR cmd supercommand.go:458 cannot perform restore: <nil>: restore failed: cannot open state: cannot create index: unauthorized mongo access: not authorized on juju to execute command { createIndexes: "actions", indexes: [ { name: "model-uuid_1_name_1", ns: "juju.actions", key: { model-uuid: 1, name: 1 } } ] } (unauthorized access)
06:38:51 DEBUG cmd supercommand.go:459 (error details: [{github.com/juju/juju/cmd/juju/backups/restore.go:419: } {github.com/juju/juju/api/backups/restore.go:137: cannot perform restore: <nil>} {github.com/juju/juju/api/apiclient.go:754: } {github.com/juju/retry/retry.go:187: } {github.com/juju/juju/rpc/client.go:149: } {restore failed: cannot open state: cannot create index: unauthorized mongo access: not authorized on juju to execute command { createIndexes: "actions", indexes: [ { name: "model-uuid_1_name_1", ns: "juju.actions", key: { model-uuid: 1, name: 1 } } ] } (unauthorized access)}])

[John A Meinel (jameinel)]

The first one is odd. Cannot cleanup failed restore because a restore is in
process.

Sounds like the failure path of restore has not been well tested and it
breaks the system. Probably leaves the wrong password saved. (new bootstrap
password, I would guess)
I'd also be curious what the original failure was that broke restore such
that it wanted to cleanup in the first place.

John
=:->

On Sep 6, 2017 22:56, "Nicholas Skaggs" <email address hidden>
wrote:

> ** Description changed:
>
> As seen at
> - http://reports.vapour.ws/releases/issue/564c9f52749a564c5f12e153
> + http://qa.jujucharms.com/releases/issue/564c9f52749a564c5f12e153
>
> An ambiguous case where Juju didn't bring up a replacement bootstrap
> instance. In debug we can see
>
> 06:38:51 ERROR juju.api.backups restore.go:136 could not clean up after
> failed restore attempt: cannot complete restore: <nil>: juju restore is in
> progress - API is disabled to prevent data loss
> 06:38:51 ERROR cmd supercommand.go:458 cannot perform restore: <nil>:
> restore failed: cannot open state: cannot create index: unauthorized mongo
> access: not authorized on juju to execute command { createIndexes:
> "actions", indexes: [ { name: "model-uuid_1_name_1", ns: "juju.actions",
> key: { model-uuid: 1, name: 1 } } ] } (unauthorized access)
> 06:38:51 DEBUG cmd supercommand.go:459 (error details: [{
> github.com/juju/juju/cmd/juju/backups/restore.go:419: } {
> github.com/juju/juju/api/backups/restore.go:137: cannot perform restore:
> <nil>} {github.com/juju/juju/api/apiclient.go:754: } {
> github.com/juju/retry/retry.go:187: } {github.com/juju/juju/rpc/
> client.go:149: } {restore failed: cannot open state: cannot create index:
> unauthorized mongo access: not authorized on juju to execute command {
> createIndexes: "actions", indexes: [ { name: "model-uuid_1_name_1", ns:
> "juju.actions", key: { model-uuid: 1, name: 1 } } ] } (unauthorized
> access)}])
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1671329
>
> Title:
> Restore backup failed (unauthorized mongo access)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1671329/+subscriptions
>

[Nicholas Skaggs (nskaggs)]

Would be interested to compare the shared secrets. Anyways, the test is running the following commands. Probably need to dig in with a manual test to see what's happening:

juju --debug bootstrap --constraints mem=2G google/us-central1 ha-backup-restore-xenial-gce --config /tmp/tmpuu2iwz.yaml --default-model ha-backup-restore-xenial-gce --agent-version 2.2.3 --bootstrap-series xenial
juju --debug list-controllers
juju --debug list-models -c ha-backup-restore-xenial-gce
juju --debug show-status -m ha-backup-restore-xenial-gce:controller --format yaml
juju --debug show-status -m ha-backup-restore-xenial-gce:ha-backup-restore-xenial-gce --format yaml
juju --debug deploy -m ha-backup-restore-xenial-gce:ha-backup-restore-xenial-gce /var/lib/jenkins/repository/charms/dummy-source --series xenial
juju --debug deploy -m ha-backup-restore-xenial-gce:ha-backup-restore-xenial-gce /var/lib/jenkins/repository/charms/dummy-sink --series xenial
juju --debug add-relation -m ha-backup-restore-xenial-gce:ha-backup-restore-xenial-gce dummy-source dummy-sink
juju --debug expose -m ha-backup-restore-xenial-gce:ha-backup-restore-xenial-gce dummy-sink
juju --debug config -m ha-backup-restore-xenial-gce:ha-backup-restore-xenial-gce dummy-source token=One
juju --debug config -m ha-backup-restore-xenial-gce:ha-backup-restore-xenial-gce dummy-source token=
juju --debug enable-ha -n 3 -c ha-backup-restore-xenial-gce
juju --debug restore-backup -m ha-backup-restore-xenial-gce:controller -b --constraints mem=2G --file /mnt/jenkins/workspace/ha-backup-restore-xenial-gce/juju-backup-20170906-231039.tar.gz
juju-restore correctly refused to restore because the state-server was still up.
juju --debug restore-backup -m ha-backup-restore-xenial-gce:controller -b --constraints mem=2G --file /mnt/jenkins/workspace/ha-backup-restore-xenial-gce/juju-backup-20170906-231039.tar.gz

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.