tripleo

Centos9 standalone deployment failing with error: "heat.common.exception.StackValidationFailed: Property error: resources.AnsibleGroupVars.properties.value: Incorrect arguments: Items to merge must be maps."

Bug #1947193 reported by Sandeep Yadav on 2021-10-14

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Critical	Unassigned	tripleo yoga-1

Bug Description

Description:

Centos9 standalone deployment failing with error: "heat.common.exception.StackValidationFailed: Property error: resources.AnsibleGroupVars.properties.value: Incorrect arguments: Items to merge must be maps."

https://logserver.rdoproject.org/29/35229/11/check/periodic-tripleo-ci-centos-9-standalone-master/b9db632/logs/undercloud/home/zuul-worker/tripleo-deploy/heat_launcher/tripleo_deploy-1634230395.359465/heat.log

~~~
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource Traceback (most recent call last):
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/properties.py", line 491, in _resolve_user_value
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource value = self.resolve(unresolved_value, nullable=True)
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/function.py", line 264, in resolve
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource result = snippet.result()
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/hot/functions.py", line 746, in result
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource ret_map.update(ensure_map(m))
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/hot/functions.py", line 742, in ensure_map
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource raise TypeError(msg)
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource TypeError: Incorrect arguments: Items to merge must be maps.
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource During handling of the above exception, another exception occurred:
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource Traceback (most recent call last):
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/properties.py", line 426, in validate
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource self._get_property_value(key, validate=True)
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/properties.py", line 519, in _get_property_value
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource value, found = self._resolve_user_value(key, prop, validate)
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/properties.py", line 512, in _resolve_user_value
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource raise ValueError(str(e))
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource ValueError: Incorrect arguments: Items to merge must be maps.
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource During handling of the above exception, another exception occurred:
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource Traceback (most recent call last):
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/properties.py", line 437, in validate
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource raise exception.StackValidationFailed(
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource heat.common.exception.StackValidationFailed: value: Incorrect arguments: Items to merge must be maps.
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource During handling of the above exception, another exception occurred:
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource Traceback (most recent call last):
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 1889, in validate_template
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource validate = self.properties.validate(
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/properties.py", line 450, in validate
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource raise exception.StackValidationFailed(
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource heat.common.exception.StackValidationFailed: Property error: Properties.value: Incorrect arguments: Items to merge must be maps.
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource During handling of the above exception, another exception occurred:
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource Traceback (most recent call last):
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 916, in _action_recorder
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource yield
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 1028, in _do_action
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource yield from self.action_handler_task(action, args=handler_args)
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 970, in action_handler_task
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource handler_data = handler(*args)
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 303, in handle_create
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource return self.create_with_template(self.child_template(),
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 341, in create_with_template
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource raw_template.RawTemplate.delete(self.context,
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource self.force_reraise()
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource raise self.value
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 336, in create_with_template
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource result = self.rpc_client()._create_stack(self.context,
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/rpc/client.py", line 295, in _create_stack
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource return self.call(
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/rpc/client.py", line 89, in call
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource return client.call(ctxt, method, **kwargs)
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/client.py", line 189, in call
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource result = self.transport._send(
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource return self._driver.send(target, ctxt, message,
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource return self._send(target, ctxt, message, wait_for_reply, timeout,
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/impl_fake.py", line 209, in _send
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource raise failure
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/common/context.py", line 410, in wrapped
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource return func(self, ctx, *args, **kwargs)
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/service.py", line 848, in create_stack
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource stack = self._parse_template_and_validate_stack(
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/service.py", line 756, in _parse_template_and_validate_stack
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource stack.validate()
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource result = f(*args, **kwargs)
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource result = res.validate()
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 1870, in validate
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource return self.validate_template()
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 1896, in validate_template
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource raise exception.StackValidationFailed(
2021-10-14 12:54:04.621 57719 ERROR heat.engine.resource heat.common.exception.StackValidationFailed: Property error: resources.AnsibleGroupVars.properties.value: Incorrect arguments: Items to merge must be maps.
~~~

Tags:

Revision history for this message

Alex Schultz (alex-schultz) wrote on 2021-10-14:

I reproduced it with the rpms from the repos in the job. It loooks like there's an ansible_group_vars being set to [] instead of {} somewhere. Looking to track it down...

2021-10-14 15:40:38.213 18195 ERROR heat.engine.hot.functions [req-d2f5e744-783e-44ca-a5e4-f22ea9ea9a9c admin admin - - -] m was []

Revision history for this message

Alex Schultz (alex-schultz) wrote on 2021-10-14:

I downgraded python3-yaql to 1.1.3-16.el9s and it resolves this. The installed version is 2.0.0-2.el9s. So this is a bug in our yaql with 2.0.0

Revision history for this message

Alex Schultz (alex-schultz) wrote on 2021-10-14:

I added some debugging in heat.engine.hot.functions to print out the evaluated yaql data/results. Here are the differences:

yaql 1.1.3

2021-10-14 16:43:19.409 19398 ERROR heat.engine.hot.functions [req-b7b8b947-a93b-4d1b-925f-8a189a4c443a admin admin - - -] yaql stmt list(coalesce($.data.role_data, []).where($ != null).select($.get('keystone_resources')).where($ != null))
2021-10-14 16:43:19.410 19398 ERROR heat.engine.hot.functions [req-b7b8b947-a93b-4d1b-925f-8a189a4c443a admin admin - - -] yaql data {'role_data': None}
2021-10-14 16:43:19.414 19398 ERROR heat.engine.hot.functions [req-b7b8b947-a93b-4d1b-925f-8a189a4c443a admin admin - - -] yaql res []

yaql 2.0.0

2021-10-14 16:49:48.766 19477 ERROR heat.engine.hot.functions [req-12bf2a17-b0d0-4716-b5ed-96583586818d admin admin - - -] yaql stmt list(coalesce($.data.role_data, []).where($ != null).select($.get('keystone_resources')).where($ != null))
2021-10-14 16:49:48.767 19477 ERROR heat.engine.hot.functions [req-12bf2a17-b0d0-4716-b5ed-96583586818d admin admin - - -] yaql data {'role_data': None}
2021-10-14 16:49:48.770 19477 ERROR heat.engine.hot.functions [req-12bf2a17-b0d0-4716-b5ed-96583586818d admin admin - - -] yaql res [[]]

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-14: Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/814067

Changed in tripleo:
status:	Confirmed → In Progress

Sandeep Yadav (sandeepyadav93) on 2021-10-15

Changed in tripleo:
importance:	High → Critical
tags:	added: promotion-blocker

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-16: Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/814067
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/6b50e162235f083163e22e1f7dd9157e6c859e94
Submitter: "Zuul (22348)"
Branch: master

commit 6b50e162235f083163e22e1f7dd9157e6c859e94
Author: Alex Schultz <email address hidden>
Date: Thu Oct 14 16:06:01 2021 -0600

Flatten yaql list expressions

    In python yaql 2, the expressions are returning a nested list but we
    need a list of dictionaries in order to use map_merge. This change adds
    flatten so it returns [] instead of [[]] when nothing is available.

Change-Id: Ic5144c58ceb9bd146e2c470725ec7f4b65328c4d
Closes-Bug: #1947193

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-16: Fix proposed to tripleo-heat-templates (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/814254

Revision history for this message

Sandeep Yadav (sandeepyadav93) wrote on 2021-10-17:

Thank you Alex.

We have passed the previous error after merge of above patch.

https://logserver.rdoproject.org/29/35229/17/check/periodic-tripleo-ci-centos-9-standalone-master/86c314f/logs/undercloud/home/zuul-worker/standalone_deploy.log

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-17: Fix merged to tripleo-heat-templates (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/814254
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/559838b3c16ee6cc7b8d9ac7fd189f07a8690ab6
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 559838b3c16ee6cc7b8d9ac7fd189f07a8690ab6
Author: Alex Schultz <email address hidden>
Date: Thu Oct 14 16:06:01 2021 -0600

Flatten yaql list expressions

    Change-Id: Ic5144c58ceb9bd146e2c470725ec7f4b65328c4d
    Closes-Bug: #1947193
    (cherry picked from commit 6b50e162235f083163e22e1f7dd9157e6c859e94)