ResourceGroupTest fails with the below traceback where a db deadlock happens, that includes a line 'Triggering resource None for update'.
2016-10-21 01:57:19.096 21070 ERROR heat.engine.resource [req-9b880eb3-a371-438c-9d0a-6e95575b16fe ba4cfb5ef5fc482a8b5215e9da8bd383 2bce5d67f238417cb34429244cdad30f - - -] DB error (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'INSERT INTO resource (created_at, action, status, status_reason, uuid, name, nova_instance, rsrc_metadata, stack_id, root_stack_id, updated_at, properties_data, properties_data_encrypted, engine_id, atomic_key, needed_by, requires, replaces, replaced_by, current_template_id) VALUES (%(created_at)s, %(action)s, %(status)s, %(status_reason)s, %(uuid)s, %(name)s, %(nova_instance)s, %(rsrc_metadata)s, %(stack_id)s, %(root_stack_id)s, %(updated_at)s, %(properties_data)s, %(properties_data_encrypted)s, %(engine_id)s, %(atomic_key)s, %(needed_by)s, %(requires)s, %(replaces)s, %(replaced_by)s, %(current_template_id)s)'] [parameters: {'status': 'COMPLETE', 'replaces': None, 'current_template_id': 1748, 'name': u'1', 'engine_id': None, 'stack_id': 'dd896b7f-ae4b-42cb-8871-fc94edde4bfe', 'rsrc_metadata': 'null', 'created_at': datetime.datetime(2016, 10, 21, 1, 57, 19, 66833), 'status_reason': '', 'updated_at': None, 'properties_data_encrypted': 0, 'root_stack_id': u'3ef7bb50-4475-45d0-bf70-e4891cee04f9', 'needed_by': '[]', 'properties_data': 'null', 'action': 'INIT', 'atomic_key': None, 'replaced_by': None, 'requires': '[]', 'nova_instance': None, 'uuid': '0457b366-fc61-4715-886a-d994b96b247a'}]
2016-10-21 01:57:19.101 21067 DEBUG heat.engine.scheduler [req-0866e8ec-30f6-4d1c-8b43-eb75024a84b7 ba4cfb5ef5fc482a8b5215e9da8bd383 2bce5d67f238417cb34429244cdad30f - - -] Task create from WaitCondition "WaitCondition" Stack "CfnInitIntegrationTest-111693441" [a9a50a9c-4ccf-4b80-a3bb-358a5258dc56] running step /opt/stack/new/heat/heat/engine/scheduler.py:215
2016-10-21 01:57:19.111 21067 DEBUG heat.engine.scheduler [req-0866e8ec-30f6-4d1c-8b43-eb75024a84b7 ba4cfb5ef5fc482a8b5215e9da8bd383 2bce5d67f238417cb34429244cdad30f - - -] Task create from WaitCondition "WaitCondition" Stack "CfnInitIntegrationTest-111693441" [a9a50a9c-4ccf-4b80-a3bb-358a5258dc56] sleeping _sleep /opt/stack/new/heat/heat/engine/scheduler.py:156
2016-10-21 01:57:19.125 21070 INFO heat.engine.stack [req-9b880eb3-a371-438c-9d0a-6e95575b16fe ba4cfb5ef5fc482a8b5215e9da8bd383 2bce5d67f238417cb34429244cdad30f - - -] convergence_dependencies: (Dependencies([((1024, True), None), ((None, True), None)]),)
2016-10-21 01:57:19.137 21070 INFO heat.engine.stack [req-9b880eb3-a371-438c-9d0a-6e95575b16fe ba4cfb5ef5fc482a8b5215e9da8bd383 2bce5d67f238417cb34429244cdad30f - - -] Triggering resource 1024 for update
2016-10-21 01:57:19.138 21070 DEBUG oslo_messaging._drivers.amqpdriver [req-9b880eb3-a371-438c-9d0a-6e95575b16fe ba4cfb5ef5fc482a8b5215e9da8bd383 2bce5d67f238417cb34429244cdad30f - - -] CAST unique_id: 002b178e05a0447b8ed877f3f94436d3 exchange 'heat' topic 'engine_worker' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:451
2016-10-21 01:57:19.140 21070 INFO heat.engine.stack [req-9b880eb3-a371-438c-9d0a-6e95575b16fe ba4cfb5ef5fc482a8b5215e9da8bd383 2bce5d67f238417cb34429244cdad30f - - -] Triggering resource None for update
2016-10-21 01:57:19.141 21070 DEBUG oslo_messaging._drivers.amqpdriver [req-9b880eb3-a371-438c-9d0a-6e95575b16fe ba4cfb5ef5fc482a8b5215e9da8bd383 2bce5d67f238417cb34429244cdad30f - - -] CAST unique_id: e249ac96938549b88925c10928eda329 exchange 'heat' topic 'engine_worker' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:451
2016-10-21 01:57:19.142 21075 DEBUG oslo_messaging._drivers.amqpdriver [-] received message with unique_id: 002b178e05a0447b8ed877f3f94436d3 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:196
2016-10-21 01:57:19.144 21068 DEBUG oslo_messaging._drivers.amqpdriver [-] received message with unique_id: e249ac96938549b88925c10928eda329 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:196
2016-10-21 01:57:19.144 21070 DEBUG oslo_messaging._drivers.amqpdriver [req-9b880eb3-a371-438c-9d0a-6e95575b16fe ba4cfb5ef5fc482a8b5215e9da8bd383 2bce5d67f238417cb34429244cdad30f - - -] sending reply msg_id: 35fa696729534db5b59417a4149f7eac reply queue: reply_9bc755f0b14d4f1b8f40680c2fbd0c1d time elapsed: 0.163571543s _send_reply /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:73
2016-10-21 01:57:19.147 21070 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 35fa696729534db5b59417a4149f7eac __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:299
2016-10-21 01:57:19.147 21068 ERROR oslo_messaging.rpc.server [req-9b880eb3-a371-438c-9d0a-6e95575b16fe ba4cfb5ef5fc482a8b5215e9da8bd383 2bce5d67f238417cb34429244cdad30f - - -] Exception during message handling
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 225, in dispatch
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 195, in _do_dispatch
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 154, in wrapper
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server return f(*args, **kwargs)
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/opt/stack/new/heat/heat/common/context.py", line 424, in wrapped
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server return func(self, ctx, *args, **kwargs)
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/opt/stack/new/heat/heat/engine/worker.py", line 167, in check_resource
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server cnxt, resource_id, resource_data, is_update)
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/opt/stack/new/heat/heat/engine/check_resource.py", line 292, in load_resource
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server is_update, cache_data)
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/opt/stack/new/heat/heat/engine/resource.py", line 314, in load
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server db_res = resource_objects.Resource.get_obj(context, resource_id)
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/opt/stack/new/heat/heat/objects/resource.py", line 121, in get_obj
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server refresh=refresh)
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/opt/stack/new/heat/heat/db/api.py", line 104, in resource_get
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server return IMPL.resource_get(context, resource_id, refresh=refresh)
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/opt/stack/new/heat/heat/db/sqlalchemy/api.py", line 171, in resource_get
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server result = context.session.query(models.Resource).get(resource_id)
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 831, in get
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server return self._get_impl(ident, loading.load_on_ident)
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 842, in _get_impl
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server if len(ident) != len(mapper.primary_key):
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server TypeError: object of type 'NoneType' has no len()
2016-10-21 01:57:19.147 21068 TRACE oslo_messaging.rpc.server
http://logs.openstack.org/05/389405/1/gate/gate-heat-dsvm-functional-convg-mysql-lbaasv2/634fc61/logs/screen-h-eng.txt.gz?#_2016-10-21_01_57_19_096
