pyjuju provision-agent hangs and fails to provision instances

[David Ames]

Filing this bug from an IS event on 2013-06-07 if it is a duplicate please disregard

Juju would not provision an instance
 juju -v add-unit showed the following:
2013-06-08 00:55:56,011:7806(0x7f9e1ab94700):ZOO_ERROR@handle_socket_error_msg@1603: Socket [127.0.0.1:32926] zk retcode=-4, errno=112(Host is down): failed whilereceiving a server response

On the bootnode we found the provision-agent.log was 3GB.
Truncating the log and restarting the juju-provision-agent resolved the problem

Hazmat received a copy of the log for analysis.

[David Ames (thedac)]

Traceback in the provision-agent.log

2013-06-08 01:20:30,213: juju.agents.provision@ERROR: Cannot process machine 179
Traceback (most recent call last):
  File "/usr/lib/juju/juju/juju/agents/provision.py", line 187, in process_machines
    machine_state_id, provider_machines)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/juju/juju/juju/agents/provision.py", line 229, in process_machine
    {"machine-id": machine_state.id, "constraints": constraints})
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/juju/juju/juju/providers/common/launch.py", line 64, in run
    machines = yield self.start_machine(machine_id, zookeepers)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/juju/juju/juju/providers/openstack/launch.py", line 84, in start_machine
    yield self._provider.port_manager.ensure_groups(machine_id))
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/juju/juju/juju/providers/openstack/ports.py", line 132, in ensure_groups
    groups_by_name[machine_group])
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/juju/juju/juju/providers/openstack/client.py", line 319, in _empty
    response.code, body))
ProviderInteractionError: Unexpected 400: '{"badRequest": {"message": "Security group is still in use", "code": 400}}'

[Haw Loeung (hloeung)]

LP:1191651 is also related.