Hello,
Jenkins seems to fail sometime in the check 'check-tempest-dsvm-postgres-full' because a race condition:
2013-12-16 14:46:17.813 | Checking logs...
2013-12-16 14:46:20.831 | Log File: h-eng
2013-12-16 14:46:20.833 | 2013-12-16 14:41:19.669 25868 ERROR heat.openstack.common.rpc.common [req-69ba92a4-4129-401e-93bb-170f7d984475 None] Returning exception Stack heat-tempest-1152838063 already has an action (DELETE) in progress. to caller
2013-12-16 14:46:20.835 |
2013-12-16 14:46:20.836 | 2013-12-16 14:41:19.669 25868 ERROR heat.openstack.common.rpc.common [req-69ba92a4-4129-401e-93bb-170f7d984475 None] ['Traceback (most recent call last):\n', ' File "/opt/stack/new/heat/heat/openstack/common/rpc/amqp.py", line 462, in _process_data\n **args)\n', ' File "/opt/stack/new/heat/heat/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyobj, method)(ctxt, **kwargs)\n', ' File "/opt/stack/new/heat/heat/engine/service.py", line 62, in wrapped\n return func(self, ctx, *args, **kwargs)\n', ' File "/opt/stack/new/heat/heat/engine/service.py", line 511, in delete_stack\n self._start_thread_with_lock(cnxt, stack, stack.delete)\n', ' File "/opt/stack/new/heat/heat/engine/service.py", line 125, in _start_thread_with_lock\n lock.acquire()\n', ' File "/opt/stack/new/heat/heat/engine/stack_lock.py", line 61, in acquire\n action=self.stack.action)\n', 'ActionInProgress: Stack heat-tempest-1152838063 already has an action (DELETE) in progress.\n']
2013-12-16 14:46:20.837 |
2013-12-16 14:46:23.295 | Log File: h-api
2013-12-16 14:46:23.297 | 2013-12-16 14:41:19.672 25874 ERROR root [-] Unexpected error occurred serving API: Stack heat-tempest-1152838063 already has an action (DELETE) in progress.
2013-12-16 14:46:23.300 |
2013-12-16 14:46:23.683 | Logs have errors
2013-12-16 14:46:23.712 | FAILED
http://logs.openstack.org/17/61717/4/check/check-tempest-dsvm-postgres-full/3b776c3/
Currently any user-triggered exception is logged as an ERROR in amqp.py.
We should be decorating our service methods to wrap these exceptions in an ClientException