JuJu restore gets the new controller stuck in "juju restore is in progress" state

[Matthew Williams]

I used the following command to restore the backup to another host

```juju restore-backup --id 20190220-002259.1d72c1b2-ddc1-4acb-890f-4848f7b43053 --constraints tags="jujuslave1"```

Which worked except for some permission errors that I troubleshot and discovered that the new host used agent machine-3 to spin up the juju-db and not machine-0 like the previous host and the new agent had no authority inside of mongo. I resolved that by duplicating the permissions from the original agent to the current one. Which resolved all the permission errors I received but started a new one when attempting to contact the controller and get its status.

```juju status --debug
09:19:58 INFO juju.cmd supercommand.go:56 running juju [2.3.7 gc go1.8]
09:19:58 DEBUG juju.cmd supercommand.go:57 args: []string{"juju", "status", "--debug"}
09:19:58 INFO juju.juju api.go:67 connecting to API addresses: [10.54.4.17:17070 10.54.4.2:17070 10.54.4.3:17070]
09:19:58 DEBUG juju.api apiclient.go:843 successfully dialed "wss://10.54.4.17:17070/model/1d72c1b2-ddc1-4acb-890f-4848f7b43053/api"
09:19:58 INFO juju.api apiclient.go:597 connection established to "wss://10.54.4.17:17070/model/1d72c1b2-ddc1-4acb-890f-4848f7b43053/api"
09:19:58 DEBUG juju.api monitor.go:35 RPC connection died
ERROR juju restore is in progress - API is disabled to prevent data loss
09:19:58 DEBUG cmd supercommand.go:459 error stack:
juju restore is in progress - API is disabled to prevent data loss
github.com/juju/juju/rpc/client.go:149:
github.com/juju/juju/api/apiclient.go:924:
github.com/juju/juju/cmd/juju/status/status.go:147:```

I can find no solution for removing this server from its eternal restoring state, I remove the replset, i re-initialized the repleset, I attempted just about everything with the database I could think of. I cannot get the server out of this state and actually usable. This bug is rather frustrating when attempting to do restores. I duplicated the issue with a new controller.

Bootstrap new controller
place controller in HA - juju enable-ha
create backup
restore backup to new host
resolve permission issues
receive API disabled due to restore errors

[John A Meinel (jameinel)]

'juju restore-backup' is currently broken if the original backup was taken from an HA controller. This is due to a variety of reasons, mostly around the new controller isn't going to be HA, so a few of the restored bits bring up a configuration that says "you should be HA, wait for the other agents."

I have started working through a step-by-step to manually restore a controller, which we hope to incorporate into 'juju restore-backup'. I haven't finished polishing all the steps, but it might give you steps that you could follow.

[Matthew Williams (matthew.williams)]

I would be happy to test any steps you have. As at the moment my juju installation is broken and has left me trying to manually maintain over 120 agents and their services. I will note, that everything is working correctly, I just cannot do any architecture changes, as the agents will still overwrite anything I attempt to change in the configs. I was just about to disable all of the agents, but if your steps work then I wont need to, if not I will go ahead with that.

Thank you very much for your response!

[John A Meinel (jameinel)]

https://discourse.jujucharms.com/t/manual-steps-to-restore-a-backup/1330

Should be the manual steps necessary to recover a new controller from a
backup of an HA controller.
There are a couple other steps that you'll eventually want, namely "juju
remove-machine -m controller $OTHERMACHINES" and then "juju enable-ha" when
you are done.

On Thu, Mar 28, 2019 at 7:50 PM Matthew Williams <email address hidden>
wrote:

> I would be happy to test any steps you have. As at the moment my juju
> installation is broken and has left me trying to manually maintain over
> 120 agents and their services. I will note, that everything is working
> correctly, I just cannot do any architecture changes, as the agents will
> still overwrite anything I attempt to change in the configs. I was just
> about to disable all of the agents, but if your steps work then I wont
> need to, if not I will go ahead with that.
>
> Thank you very much for your response!
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1820101
>
> Title:
> JuJu restore gets the new controller stuck in "juju restore is in
> progress" state
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1820101/+subscriptions
>

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.