can no longer deploy to model: 'error': 'shared state watcher was stopped'

[Francis Ginther]

I'm using python-jujuclient via juju-deployer to deploy a bundle to a juju 2 model (backed by openstack).

python-jujuclient: 0.53.2+bzr95+37
juju-deployer: 1:0.9.0~bzr191~58
juju: 2.0.0-trusty-amd64

After deploying and removing applications and machines in this model, I'm no longer able to deploy anything with juju-deployer. I hit this Traceback:

$ juju-deployer -c /srv/mojo/[redacted]/relations
2016-11-03 19:29:58 Using deployment shopify-adapter
2016-11-03 19:29:58 Starting deployment of shopify-adapter
2016-11-03 19:30:00 Deploying applications...
2016-11-03 19:31:15 Config specifies num units for subordinate: canonical-livepatch
2016-11-03 19:32:15 Config specifies num units for subordinate: nrpe
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/jujuclient/watch.py", line 58, in next
    result = self._rpc(self.next_args())
  File "/usr/lib/python3/dist-packages/jujuclient/rpc.py", line 42, in _rpc
    raise EnvError(result)
jujuclient.exc.EnvError: <Env Error - Details:
 {'error': 'shared state watcher was stopped', 'request-id': 2, 'response': {}}
 >

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/juju-dist/2.0/usr/bin/juju-deployer", line 9, in <module>
    load_entry_point('juju-deployer==0.9.0', 'console_scripts', 'juju-deployer')()
  File "/usr/lib/python3/dist-packages/deployer/cli.py", line 140, in main
    run()
  File "/usr/lib/python3/dist-packages/deployer/cli.py", line 250, in run
    importer.Importer(env, deployment, options).run()
  File "/usr/lib/python3/dist-packages/deployer/action/importer.py", line 317, in run
    self.wait_for_units(ignore_errors=ignore_errors)
  File "/usr/lib/python3/dist-packages/deployer/action/importer.py", line 288, in wait_for_units
    services=self.deployment.get_service_names(), on_errors=on_errors)
  File "/usr/lib/python3/dist-packages/deployer/env/go.py", line 279, in wait_for_units
    services=services, on_errors=on_errors).run(callback)
  File "/usr/lib/python3/dist-packages/jujuclient/watch.py", line 162, in run
    for change_set in self.watch:
  File "/usr/lib/python3/dist-packages/jujuclient/watch.py", line 124, in next
    return super(BaseTimeoutWatcher, self).next()
  File "/usr/lib/python3/dist-packages/jujuclient/watch.py", line 63, in next
    if not self.reconnect():
  File "/usr/lib/python3/dist-packages/jujuclient/watch.py", line 75, in reconnect
    return super(BaseWatcher, self).reconnect()
  File "/usr/lib/python3/dist-packages/jujuclient/rpc.py", line 96, in reconnect
    self.conn = self.connector().connect_socket_loop(
AttributeError: 'TimeoutWatcher' object has no attribute 'connector'

This occurs after removing all application and machines, and then trying to deploy again.

[Paul Gear (paulgear)]

We experienced this on a HA controller running 2.0.3 recently; a rolling restart seemed to allow things to proceed.

[James Hebden (ec0)]

Also seeing this on a 2.0.3 environment, using juju-deploy 0.9.2~bzr207~60~ubuntu16.04.1
Restart of the controller's machine 0 seems to have cleared this up for now.

[Drew Freiberger (afreiberger)]

Seeing the same with 2.1.3-xenial juju environment with juju-deployer 0.10.0~bzr212~60~ubuntu16.04.1. James' restart of controller machine-0 did the trick.