test_stack_update_with_conditions failing with KeyError intermittently

[Rabi Mishra]

Probably after Id438336e5c88dc7c2d168ba01ee703faa17e8b8e test_stack_update_with_conditions has started failing again with KeyError when listing an UPDATE_IN_PROGRESS stack.

traceback:

Jul 19 03:07:24.298403 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: ERROR oslo_messaging.rpc.server [None req-66d49551-02d7-495e-89ac-69fe0250f592 demo None] Exception during message handling: KeyError: u'test2'
Jul 19 03:07:24.298525 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
Jul 19 03:07:24.298635 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
Jul 19 03:07:24.298731 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jul 19 03:07:24.298826 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
Jul 19 03:07:24.298919 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jul 19 03:07:24.299017 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
Jul 19 03:07:24.299118 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jul 19 03:07:24.299223 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
Jul 19 03:07:24.299328 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server return f(*args, **kwargs)
Jul 19 03:07:24.299428 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server File "/opt/stack/new/heat/heat/common/context.py", line 399, in wrapped
Jul 19 03:07:24.299544 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server return func(self, ctx, *args, **kwargs)
Jul 19 03:07:24.299636 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server File "/opt/stack/new/heat/heat/engine/service.py", line 2023, in list_stack_resources
Jul 19 03:07:24.299737 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server for resource in rsrcs]
Jul 19 03:07:24.299827 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server File "/opt/stack/new/heat/heat/engine/stack.py", line 334, in iter_resources
Jul 19 03:07:24.299920 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server for res in self._find_filtered_resources(filters):
Jul 19 03:07:24.300012 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server File "/opt/stack/new/heat/heat/engine/stack.py", line 325, in _find_filtered_resources
Jul 19 03:07:24.300114 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server loaded_res = self._resource_from_db_resource(rsc, rsrc_def_cache)
Jul 19 03:07:24.300207 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server File "/opt/stack/new/heat/heat/engine/stack.py", line 376, in _resource_from_db_resource
Jul 19 03:07:24.300302 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server cur_res = self.resources[db_res.name]
Jul 19 03:07:24.300400 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server KeyError: u'test2'
Jul 19 03:07:24.300498 ubuntu-xenial-osic-cloud1-s3500-9926557 heat-engine[32657]: 2017-07-19 03:07:24.296 2598 TRACE oslo_messaging.rpc.server

engine log:

http://logs.openstack.org/56/482056/3/check/gate-heat-dsvm-functional-orig-mysql-lbaasv2-ubuntu-xenial/87e5228/logs/screen-h-eng.txt.gz?level=ERROR#_Jul_19_03_07_24_300400

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/485364

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/485364
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=65630d4e8bbff2f3ae5c659f8e9396ec7a6f6ae9
Submitter: Jenkins
Branch: master

commit 65630d4e8bbff2f3ae5c659f8e9396ec7a6f6ae9
Author: Zane Bitter <email address hidden>
Date: Wed Jul 19 17:50:58 2017 -0400

    Handle non-existent resources in list during update

    When showing the resource list during a stack update, the patch
    5ce238fb17e38702ceef85a3706e9d2236c43e0e inadvertently raises a KeyError
    when a resource in the new template hasn't been created yet. This resolves
    the problem.

    Change-Id: If0784156211b21d8ef7047a7082d1b48e977252d
    Related-Bug: #1703703
    Closes-Bug: #1705170

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 9.0.0.0b3 development milestone.