Stack validation (stack-update --dry-run) with --show-nested fails, while stack update itself works

[Dmitry Sutyagin]

Issue was previously reported as https://bugs.launchpad.net/mos/+bug/1799006, however it is present in MOS 9.2 and can be reproduced.

Error is:

2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher [req-10e9d8e5-ac94-4bd6-bacc-03a5f740b37c - admin - default default] Exception during message handling: Property private_network_id not assigned
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 117, in wrapper
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/common/context.py", line 329, in wrapped
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher return func(self, ctx, *args, **kwargs)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 1085, in preview_update_stack
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher nested_fmt_actions(current_stack, updated_stack, actions)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 1077, in nested_fmt_actions
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher n_actions = update_task.preview()
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/update.py", line 268, in preview
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher None):
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 1062, in preview_update
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher before_props)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 566, in update_template_diff_properties
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher changed_properties_set = set(k for k in after_props if prop_changed(k))
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 566, in <genexpr>
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher changed_properties_set = set(k for k in after_props if prop_changed(k))
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 563, in prop_changed
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher return before != after_props.get(key)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/_abcoll.py", line 363, in get
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher return self[key]
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/properties.py", line 453, in __getitem__
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher return self._get_property_value(key)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/properties.py", line 450, in _get_property_value
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher raise ValueError(_('Property %s not assigned') % key)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher ValueError: Property private_network_id not assigned

If "--show-nested" is not used, then validation works. Stack update also works, to the issue is only with the --dry-run method when --show-nested is used.

To Mirantis staff - I have a lab environment where I have reproduced this issue, can provide access, please contact me in Slack.

[Dmitry Sutyagin (dsutyagin)]

I have identified that in:

File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 1062, in preview_update

The values for IDs which should be extracted from nested stacks are 'None' in the updated_stack structure:

http://paste.openstack.org/show/734732/

If debug log is enabled, I can see that Heat tries to "GET" these resources via going to respective APIs and passes None as ID, fails several times.

Subsequently in:

File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 563, in prop_changed
return before != after_props.get(key)

Heat raises this error because after_props does not contain some of the keys.

Right now it seems that updated_stack does not get populated with any resource IDs known to current_stack, and I'm not sure how this should work.

[Dmitry Sutyagin (dsutyagin)]

p.s. line numbers in tracebacks might be off a bit due to my debugging code inserts

[Oleksiy Molchanov (omolchanov)]

After the investigation we have found out that it must be a bug in heat, so we have created a bugreport for heat-team

https://storyboard.openstack.org/#!/story/2004385

[Oleksiy Molchanov (omolchanov)]

Waiting for upstream patch to backport it to mitaka.

[Oleksiy Molchanov (omolchanov)]

https://review.fuel-infra.org/#/c/39911/1

[Mikhail Samoylov (msamoylov)]

Steps for verify:
1. Create configs
repro.yaml

heat_template_version: 2015-04-30
resources:
  net:
    type: OS::Neutron::Net
    properties:
      name: repro
  grp:
    type: OS::Heat::ResourceGroup
    properties:
      count: 1
      resource_def:
        type: repro-nested.yaml
        properties:
          private_network_id: { get_resource: net }

repro-nested.yaml

heat_template_version: 2015-04-30
parameters:
  private_network_id:
    type: string
resources:
  unit:
    type: OS::Heat::TestResource
    properties:
      value: { get_param: private_network_id }

openstack stack create repro -t repro.yaml
openstack stack update repro -t repro.yaml --dry-run --show-nested

2. Run verify
heat stack-update repro --template-file repro.yaml --dry-run --show-nested
+-----------+---------------+--------------------------------+-----------------------------------------------------------+
| state | resource_name | resource_type | resource_identity |
+-----------+---------------+--------------------------------+-----------------------------------------------------------+
| unchanged | net | OS::Neutron::Net | { |
| | | | "stack_name": "repro", |
| | | | "stack_id": "4442b7fe-bb2b-4766-a6dd-4947603a213b", |
| | | | "tenant": "6fe6cd742c384eafa3f853dc7ecb995b", |
| | | | "path": "/resources/net" |
| | | | } |
| updated | grp | OS::Heat::ResourceGroup | { |
| | | | "stack_name": "repro", |
| | | | "stack_id": "4442b7fe-bb2b-4766-a6dd-4947603a213b", |
| | | | "tenant": "6fe6cd742c384eafa3f853dc7ecb995b", |
| | | | "path": "/resources/grp" |
| | | | } |
| updated | 0 | file:///root/repro-nested.yaml | { |
| | | | "stack_name": "repro-grp-zpvljfhha3zg", |
| | | | "stack_id": "e233c0dc-e355-4720-95f0-65ae68b30502", |
| | | | "tenant": "6fe6cd742c384eafa3f853dc7ecb995b", |
| | | | "path": "/resources/...