A controller gets stuck upgrading if a model had a missing credential.
In this case, the model for which the credential is missing is dying.
017-06-26 01:41:10 DEBUG juju.worker.dependency engine.go:499 "upgrade-steps-flag" manifold worker stopped: "upgrade-steps-gate" not running: dependency not available
2017-06-26 01:41:10 DEBUG juju.worker.dependency engine.go:499 "upgrade-steps-runner" manifold worker stopped: "agent" not running: dependency not available
2017-06-26 01:41:10 DEBUG juju.worker.dependency engine.go:485 "upgrade-steps-gate" manifold worker started
2017-06-26 01:41:10 DEBUG juju.worker.dependency engine.go:499 "upgrade-steps-runner" manifold worker stopped: <nil>
2017-06-26 01:41:10 DEBUG juju.worker.dependency engine.go:499 "upgrade-steps-runner" manifold worker stopped: "api-caller" not running: dependency not available
2017-06-26 01:41:10 DEBUG juju.worker.dependency engine.go:485 "upgrade-steps-flag" manifold worker started
2017-06-26 01:41:37 DEBUG juju.worker.dependency engine.go:485 "upgrade-steps-runner" manifold worker started
2017-06-26 01:41:37 INFO juju.worker.upgradesteps worker.go:250 checking that upgrade can proceed
2017-06-26 01:41:37 INFO juju.upgrade preupgradesteps.go:32 updating distro-info
2017-06-26 01:41:38 DEBUG juju.worker.dependency engine.go:499 "migration-fortress" manifold worker stopped: "upgrade-steps-flag" not set: dependency not available
2017-06-26 01:41:45 INFO juju.worker.upgradesteps worker.go:259 signalling that this controller is ready for upgrade
2017-06-26 01:41:45 INFO juju.worker.upgradesteps worker.go:267 waiting for other controllers to be ready for upgrade
2017-06-26 01:41:45 INFO juju.worker.upgradesteps worker.go:286 finished waiting - all controllers are ready to run upgrade steps
2017-06-26 01:41:45 INFO juju.worker.upgradesteps worker.go:345 starting upgrade from 2.1.3 to 2.2.1 for "machine-0"
2017-06-26 01:43:49 ERROR juju.worker.upgradesteps worker.go:375 upgrade from 2.1.3 to 2.2.1 for "machine-0" failed (will retry): cloud credential "azure/someuser@external/MicrosoftAzure12345" not found
2017-06-26 01:49:50 DEBUG juju.worker.dependency engine.go:499 "upgrade-steps-runner" manifold worker stopped: <nil>
Having a model that's Alive or Dying but without credentials is a problem: that will prevent destroying the model's machines and such. Also, the Azure provider in particular will not be able to call the "Environ.Destroy" method, and so the model's resource group won't be deleted. It may be that we'll want to allow that, but we'll need to be able to clear that the model is disabled.
The other part is that we probably shouldn't block the entire controller upgrade because of one model. An alternative would be to do the environ upgrades later, when the model workers start. We would have to gate any Environ-related operations on the environ being upgraded.