Unable to update a parameter from string to comma_delimited_list

Bug #1538551 reported by Jiří Stránský on 2016-01-27
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Medium
Zane Bitter
Kilo
Fix Released
Undecided
Unassigned

Bug Description

This problem appears with TripleO upgrades. Trying to upgrade the overcloud from Kilo to Liberty, we upgrade from templates which didn't have this patch:

https://github.com/openstack/tripleo-heat-templates/commit/16093c3932545b1a8d1f4572c98d6953c277b3d5

to templates which do have it. This means that the NtpServer parameter changes from string to comma_delimited_list. The upgrade then fails with:

ValueError: resources[0]: "u'clock.redhat.com'" is not a list

However, a string is still a valid value for comma_delimited_list, so i'd kind of expect this to work in an ideal case.

Perhaps a greater problem is that it's not possible to work around this issue (as far as i can tell). I tried specifying a completely new value for the parameter in an environment file, using both parameter_defaults and parameters to make sure that the value gets picked up:

parameter_defaults:
  NtpServer: ['1.2.3.4']

parameters:
  NtpServer: ['1.2.3.4']

But i still get the original error message:

ValueError: resources[0]: "u'clock.redhat.com'" is not a list

At this point i'd expect the stack to try to use the new value (the array with the fake IP) and not try to use/validate the old one.

Jiří Stránský (jistr) wrote :

Checked heat-api log -- Heat did receive the new NtpServer parameter value.

Rabi Mishra (rabi) wrote :

I tried with some simple templates[1] with master and doing and UPDATE by changing a parameter from string to comma_delimited_list and didn't find any issue.

[1] http://paste.openstack.org/show/485254/

Would probably need the engine logs or some reproducer for this.

Changed in heat:
assignee: nobody → Crag Wolfe (cwolfe)
status: New → In Progress
Zane Bitter (zaneb) wrote :

The fact that the error is occurring with the *old* input value suggests to me that this is another manifestation of bug 1543685.

Basically this only occurs after a failure during an upgrade (which is why we've been having so much trouble coming up with a minimal reproducer). Heat will have stored the new nested template, and it proceeds to try to resolve the *old* property values against it. So while comma_delimited_list always returns a Python list, it's the old string that we're trying to validate as a list here and that's why it's failing.

As noted in that bug, there's no completely correct way to fix it until convergence. However, the fix that was merged (https://review.openstack.org/#q,I859ea72b1cc95162b6f498b6af65e4d9fdd9458f,n,z) does catch ValueError specifically, so it should also resolve this issue also. If you could test with that fix in, that would be helpful, and if it's resolved then we can close this as a duplicate.

Zane Bitter (zaneb) wrote :

We're also seeing a TypeError in the same place that we're catching ValueError for bug 1543685, so we need to catch both of those exceptions:

2016-03-01 14:12:14.662 29747 INFO heat.engine.resource [-] UPDATE: TemplateResource "0" [17065956-ece1-4717-a38b-d67a986c312c] Stack "overcloud-Controller-rnd6grj6l3t7" [e7e9a121-50fb-467e-b140-2a2ad0d22b12]
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource Traceback (most recent call last):
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource yield
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 965, in update
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource before_props)
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 512, in update_template_diff_properties
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource changed_properties_set = set(k for k in after_props if prop_changed(k))
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 512, in <genexpr>
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource changed_properties_set = set(k for k in after_props if prop_changed(k))
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 497, in prop_changed
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource before = before_props.get(key)
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource File "/usr/lib64/python2.7/_abcoll.py", line 363, in get
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource return self[key]
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 456, in __getitem__
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource return self._get_property_value(key)
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 451, in _get_property_value
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource return prop.get_value(None, validate)
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 326, in get_value
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource _value = self._get_list(value, validate)
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 296, in _get_list
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource raise TypeError(_('"%s" is not a list') % repr(value))
2016-03-01 14:12:14.662 29747 ERROR heat.engine.resource TypeError: "u'1:1000'" is not a list

Fix proposed to branch: master
Review: https://review.openstack.org/286874

Changed in heat:
assignee: Crag Wolfe (cwolfe) → Zane Bitter (zaneb)
Zane Bitter (zaneb) on 2016-03-01
tags: added: kilo-backport-potential liberty-backport-potential

Reviewed: https://review.openstack.org/286874
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=79c410d92326e63c8bf3752922c35abf6264e211
Submitter: Jenkins
Branch: master

commit 79c410d92326e63c8bf3752922c35abf6264e211
Author: Zane Bitter <email address hidden>
Date: Tue Mar 1 17:56:33 2016 -0500

    Ignore TypeErrors on old properties during update

    Extend the approach from 1cfde62b2ff8a60c8cc3a9ef4926e5f4a7116aee of
    ignoring ValueErrors in the _previous_ properties when doing an update to
    also ignoring TypeErrors.

    Change-Id: Ic08165fe031e1764cfb34eaf782d3e105f04fee3
    Closes-Bug: #1538551
    Related-Bug: #1543685

Changed in heat:
status: In Progress → Fix Released
Changed in heat:
milestone: none → mitaka-3
importance: Undecided → Medium

Reviewed: https://review.openstack.org/286879
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=94a840d486d60523daeaf9c35c9f55351518842e
Submitter: Jenkins
Branch: stable/liberty

commit 94a840d486d60523daeaf9c35c9f55351518842e
Author: Zane Bitter <email address hidden>
Date: Tue Mar 1 17:56:33 2016 -0500

    Ignore TypeErrors on old properties during update

    Extend the approach from 1cfde62b2ff8a60c8cc3a9ef4926e5f4a7116aee of
    ignoring ValueErrors in the _previous_ properties when doing an update to
    also ignoring TypeErrors.

    Change-Id: Ic08165fe031e1764cfb34eaf782d3e105f04fee3
    Closes-Bug: #1538551
    Related-Bug: #1543685
    (cherry picked from commit 79c410d92326e63c8bf3752922c35abf6264e211)

tags: added: in-stable-liberty
Zane Bitter (zaneb) on 2016-03-02
tags: removed: liberty-backport-potential

This issue was fixed in the openstack/heat 6.0.0.0b3 development milestone.

Change abandoned by Crag Wolfe (<email address hidden>) on branch: master
Review: https://review.openstack.org/275544
Reason: Confirmed fixed in Ic08165fe031e1764cfb34eaf782d3e105f04fee3

Reviewed: https://review.openstack.org/287272
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=121f5312185a5ade30889ed975cd2091dc63f288
Submitter: Jenkins
Branch: stable/kilo

commit 121f5312185a5ade30889ed975cd2091dc63f288
Author: Zane Bitter <email address hidden>
Date: Tue Mar 1 17:56:33 2016 -0500

    Ignore TypeErrors on old properties during update

    Extend the approach from 1cfde62b2ff8a60c8cc3a9ef4926e5f4a7116aee of
    ignoring ValueErrors in the _previous_ properties when doing an update to
    also ignoring TypeErrors.

    Change-Id: Ic08165fe031e1764cfb34eaf782d3e105f04fee3
    Closes-Bug: #1538551
    Related-Bug: #1543685
    (cherry picked from commit 79c410d92326e63c8bf3752922c35abf6264e211)

tags: added: in-stable-kilo
Zane Bitter (zaneb) on 2016-03-10
tags: removed: kilo-backport-potential

This issue was fixed in the openstack/heat 2015.1.4 release.

This issue was fixed in the openstack/heat 5.0.2 release.

This issue was fixed in the openstack/heat 2015.1.4 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers