Deployer failure upgrading to proposed 1.23.3 from 1.23.2 client

[Adam Collard]

Using the attached YAML, with an OpenStack provider and Juju 1.23.2-vivid-amd64 with agent-stream: proposed in environments.yaml.

2015-05-12 09:17:22 [DEBUG] deployer.import: Adding units...
2015-05-12 09:17:23 [DEBUG] deployer.import: Service 'postgresql' does not need any more units added.
2015-05-12 09:17:23 [WARNING] deployer.import: Config specifies num units for subordinate: storage
2015-05-12 09:17:23 [DEBUG] deployer.import: Service 'block-storage-broker' does not need any more units added.
2015-05-12 09:17:23 [DEBUG] deployer.import: Waiting for units before adding relations
2015-05-12 09:17:25 [DEBUG] deployer.env: Delta unit: block-storage-broker/0 change:allocating
2015-05-12 09:17:25 [DEBUG] deployer.env: Delta unit: block-storage-broker/0 change:installing
Traceback (most recent call last):
  File "/usr/bin/juju-deployer", line 9, in <module>
    load_entry_point('juju-deployer==0.4.4', 'console_scripts', 'juju-deployer')()
  File "/usr/lib/python2.7/dist-packages/deployer/cli.py", line 135, in main
    run()
  File "/usr/lib/python2.7/dist-packages/deployer/cli.py", line 233, in run
    importer.Importer(env, deployment, options).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 206, in run
    self.wait_for_units(ignore_errors=ignore_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 181, in wait_for_units
    services=self.deployment.get_service_names(), on_errors=on_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 227, in wait_for_units
    services=services, on_errors=on_errors).run(callback)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 1918, in run
    for change_set in self.watch:
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 404, in next
    return super(TimeoutWatcher, self).next()
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 346, in next
    'Id': self.watcher_id})
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 281, in _rpc
    raise EnvError(result)
jujuclient.EnvError: <Env Error - Details:
 { u'Error': u'watcher was stopped', u'RequestId': 4, u'Response': { }}
 >

[Adam Collard (adam-collard)]

juju-deployer 0.4.4+bzr137+44~ubuntu14.04.1

[Curtis Hovey (sinzui)]

I assume this is an implicit upgrade during bootstrap. Can you conifrm?

[Curtis Hovey (sinzui)]

Does this happen with 1.23.2? I always get a config-changed error with 1.23.2, so I cannot reproduce the exact error to demonstrate a regression.

[Adam Collard (adam-collard)]

RE: implicit upgrade during bootstrap?

No, it's not "during" bootstrap, bootstrap had finished, services were added, units were being added and then it went boom. However I believe it is the implicit upgrade, my gripe is that this upgrade doesn't occur during the "block til we're done" bootstrap.

RE: Does this happen with 1.23.2?

Yes, it's with 1.23.2 client and proposed agent-stream so that 1.23.3 tools are found.

I'm almost certain that the charms in question aren't important, you will get a config-changed error because those are credentials for an OpenStack deployment that's not publicly available (and has subsequently been torn down)

[Curtis Hovey (sinzui)]

Since I cannot reproduce this, can delay running deployer until the state-server is upgraded to 1.23.3? In my tests, the implicit upgrade happened quick enough that the env was upgraded before I could run deployer. You can also add this to the env config to pin the version
    agent-version: 1.23.2

[David Britton (dpb)]

Use attached file like:

$ juju version
1.23.2-trusty-amd64
$ juju bootstrap -v && juju-deployer -vdWc ubuntu.yaml

I get the error every time.

[David Britton (dpb)]

machine-0 log from same deploy