str_replace and list_join only accept strings

Bug #1489028 reported by Steven Hardy
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Medium
Steven Hardy

Bug Description

When trying to work with json parameters, or json/map data returned by e.g SoftwareDeploymentGroup, it's very inconvenient that neither str_replace nor list_join will accept anything other than strings/numbers.

It's possible to pass map/list data via input_values via SoftwareDeployments, but it seems like these end up with the python string __repr__ of the dict, not actually serialized as json, which is an inconvenient format to consume inside the nodes (doesn't work with jq, python json.loads etc)

I'd like to:

1. Allow str_replace to accept map and list values (which are currently rejected with an error), and serialize them as json

2. Same for list_join, which also errors for non-string input, we can serialize map/list values to json, then join that

3. Work out a backwards compatible way to enable transparent conversion of a list/map input_values value to a json string when consumed in the hook. We already have a "Json" and "String" types allowed in the inputs schema, but I can't see if/where there is any explicit type conversion associated with this - need to discuss with stevebaker how this might work.

The motivation behind this is to avoid the sort of mess demonstrated here:

https://review.openstack.org/#/c/215694/3/puppet/extraconfig/all_nodes/network-cisco.yaml

This could probably be improved, but there's a mess attempting to deal with the fact that it's impossible to pass a json parameter in to the node in a format that can be easily deserialized (e.g json, so it can be handled via jq in a shell script or directly via a tiny python script).

Steven Hardy (shardy)
Changed in heat:
assignee: nobody → Steven Hardy (shardy)
Changed in heat:
status: New → Triaged
importance: Undecided → Medium
milestone: none → liberty-3
Changed in heat:
milestone: liberty-3 → liberty-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/221765

Changed in heat:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/221766

Changed in heat:
assignee: Steven Hardy (shardy) → Steve Baker (steve-stevebaker)
Changed in heat:
assignee: Steve Baker (steve-stevebaker) → Steven Hardy (shardy)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/221765
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=ec2fd65174ad4899cf4bc7021d8c1ae1661e6bb0
Submitter: Jenkins
Branch: master

commit ec2fd65174ad4899cf4bc7021d8c1ae1661e6bb0
Author: Steven Hardy <email address hidden>
Date: Wed Sep 9 13:46:04 2015 +0100

    Allow map/list items for list_join

    Currently any attempt to join json/map or list parameters/attributes
    results in an error, which is inconvenient if you wish to join
    several paramters/attributes into a single string that may be easily
    consumed in an instance (e.g by splitting each element on a delimiter
    then using jq or python to process the json).

    So, tolerate non-string list items, and attempt to serialize them
    as json, then join the resulting json strings.

    Partial-Bug: #1489028
    Change-Id: I699fa2ec43d173fff1f06887aaa80e3fbac8d668

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/221766
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=66f4178d280f8418ae2358fcf1539811b2c2014c
Submitter: Jenkins
Branch: master

commit 66f4178d280f8418ae2358fcf1539811b2c2014c
Author: Steven Hardy <email address hidden>
Date: Wed Sep 9 14:34:28 2015 +0100

    Allow map/list items for str_replace

    Currently any attempt to substitute placeholders using data from a
    json/map or list parameter/attribute results in an error, which is
    inconvenient if you wish to provide such data to an instance, e.g
    so it can be used in a script via jq/python or whatever.

    So, tolerate non-string parameter values, and attempt to serialize
    them as json, then substitute the resulting json strings.

    Change-Id: I362cc76ac3f68d4649a62459455c0dcae2dcd25d
    Closes-Bug: #1489028

Changed in heat:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in heat:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in heat:
milestone: liberty-rc1 → 5.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.