Restore-backup model is not usable because upgrade in progress

[Curtis Hovey]

As seen in
    http://reports.vapour.ws/releases/issue/5821ede4749a5654901006e5

The controller was restored. The config was set on an app to prove the model was usable. The call failed because Juju thinks an upgrade is in progress. An upgrade cannot be in progress because the juju under-test is the newest version.

ERROR cmd supercommand.go:458 upgrade in progress (upgrade in progress)
2016-11-02 15:08:00 ERROR Command '('juju', '--show-log', 'config', '-m', 'functional-backup-restore-xenial:functional-backup-restore-xenial', 'dummy-source', 'token=Two')' returned non-zero exit status 1
Traceback (most recent call last):
  File "/var/lib/jenkins/juju-ci-tools/deploy_stack.py", line 783, in runtime_context
    yield
  File "/var/lib/jenkins/juju-ci-tools/deploy_stack.py", line 893, in booted_context
    yield machines
  File "/var/lib/jenkins/juju-ci-tools/assess_recovery.py", line 199, in main
    assess_recovery(bs_manager, args.strategy, args.charm_series)
  File "/var/lib/jenkins/juju-ci-tools/assess_recovery.py", line 189, in assess_recovery
    check_controller=check_controller)
  File "/var/lib/jenkins/juju-ci-tools/assess_recovery.py", line 123, in restore_missing_state_server
    client.set_config('dummy-source', {'token': 'Two'})
  File "/var/lib/jenkins/juju-ci-tools/jujupy.py", line 1430, in set_config
    self.juju('config', (service,) + option_strings)
  File "/var/lib/jenkins/juju-ci-tools/jujupy.py", line 1481, in juju
    model, check, timeout, extra_env)
  File "/var/lib/jenkins/juju-ci-tools/jujupy.py", line 797, in juju
    rval = call_func(args)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '('juju', '--show-log', 'config', '-m', 'functional-backup-restore-xenial:functional-backup-restore-xenial', 'dummy-source', 'token=Two')' returned non-zero exit status 1

[Curtis Hovey (sinzui)]

This issue happens 50% of the time is restore-backup testing.

[John A Meinel (jameinel)]

Whenever the Controller starts, it doesn't know yet whether it has to run upgrade steps, so it starts in 'limited API' mode which always reports 'upgrade-in-progress'. Usually this would imply that we should back off and try again after a bit (the controller took a while to start).

I don't know this particular script.

[Curtis Hovey (sinzui)]

We cannot back off because juju wont tell us API is limited or an upgrade is in progress. If juju reported that is was limited in status, scripts could back off. My understanding of https://bugs.launchpad.net/juju/+bug/1645478 from Horacio was that the backup-restore case was not showing up in status.

[Anastasia (anastasia-macmood)]

@John A Meinel,

Are you really working on the fix for this bug now? If not, it probably should not be "In Progress".

[John A Meinel (jameinel)]

I think this was a "In Progress by Horacio" and all of his bugs got assigned to me.

I'm tempted to mark this Incomplete, as its based on failing CI tests, and if we see it fail again, we can reopen.

[John A Meinel (jameinel)]

I think we did add the Model Status in the recent past so the tests could watch for that as part of the "status" output and only go on to the next phase if it is not currently upgrading.

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]