Race where recently destroyed service is not redeployed causing tracebacks

[Stuart Bishop]

I regular see this failure running an Amulet test suite locally, where a service is is repeatedly setup and torn down. It seems that if I destroy a service and run juju-deployer again too quickly to redeploy it, then juju-deployer will think the recently destroyed service is still there and neglect to deploy it. I have also seen similar tracebacks where juju-deployer thought the old units were still there, which would cause it to fail when it attempted to create the relations. The most odd thing to me is that I'm waiting until the service is no longer listed in 'juju status' before continuing, so either that is broken or juju-deployer is receiving data from the API that is more out of date than juju-status. Or perhaps this is a change in Juju 1.24, where status is no longer listing services in their death throws but juju-deployer is not filtering them out in the same way.

2015-08-27 00:57:16 Starting deployment of charm-testing-joyent
2015-08-27 00:57:18 Deploying services...
2015-08-27 00:57:19 Deploying service postgresql using /tmp/charmjgoqt5af/trusty/postgresql
2015-08-27 00:58:44 Adding 1 more units to client
Traceback (most recent call last):
  File "/usr/bin/juju-deployer", line 9, in <module>
    load_entry_point('juju-deployer==0.5.1', 'console_scripts', 'juju-deployer')()
  File "/usr/lib/python2.7/dist-packages/deployer/cli.py", line 135, in main
    run()
  File "/usr/lib/python2.7/dist-packages/deployer/cli.py", line 233, in run
    importer.Importer(env, deployment, options).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 312, in run
    self.add_units()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 64, in add_units
    self.env.add_units(svc.name, abs(delta))
  File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 48, in add_units
    return self.client.add_units(service_name, num_units)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 1205, in add_units
    "NumUnits": num_units}})
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 281, in _rpc
    raise EnvError(result)
jujuclient.EnvError: <Env Error - Details:
 { u'Error': u'service "client" not found',
    u'ErrorCode': u'not found',
    u'RequestId': 1,
    u'Response': { }}
 >
ERROR

[Stuart Bishop (stub)]

I get the same issue if I I wait for 'juju-deployer -f service' to return an error code for each service before continuing. I get this a lot with the local provider, but the traceback in the description is from a joyent deploy.

[Kapil Thangavelu (hazmat)]

Deployer was using Ali before status was there and used the watch Api.. To
reconstruct status.. Apparently the sla on that has degraded with larger
delta delays to reflect current reality. Deployed should move to the
current status Api to avoid issues. Also would effect the gui i suspect,
but not clear anyone uses that.
On Thu, Aug 27, 2015 at 1:50 AM Stuart Bishop <email address hidden>
wrote:

> I get the same issue if I I wait for 'juju-deployer -f service' to
> return an error code for each service before continuing. I get this a
> lot with the local provider, but the traceback in the description is
> from a joyent deploy.
>
> --
> You received this bug notification because you are subscribed to juju-
> deployer.
> Matching subscriptions: deployer
> https://bugs.launchpad.net/bugs/1489237
>
> Title:
> Race where recently destroyed service is not redeployed causing
> tracebacks
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-deployer/+bug/1489237/+subscriptions
>