Nested stacks cannot process non-trivial data structures in parameters

Bug #1420196 reported by Johannes Grassler
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Steven Hardy
Juno
Fix Released
Undecided
Unassigned

Bug Description

Nested stacks currently (Heat Template Version 2014-10-16) support the following data types for nested stacks' parameters (or generally stack parameters for that matter):

  string, number, boolean, comma_delimited_list, json

The latter two look like they could theoretically be used for more complex data structures, such as:

  * The `networks` attribute of OS::Nova::Server (a list of maps)
  * The `allocation_pools` attribute of OS::Neutron::Subnet (a list of maps)

Hence I tried creating a stack that invoked the attached nested stack as follows:

  network:
    type: networks-wrapper.yaml
    properties:
      cidr: 10.10.10.0/24
      allocation_pools:
        - {start: 10.10.10.10, end: 10.10.10.150}
      name:
        list_join:
          - '_'
          - [ heat, { get_param: 'OS::stack_name' } ]

This resulted in a CREATE_FAILED status with the following stack_status_reason:

  Resource CREATE failed: StackValidationFailed: Property
  error : subnet: allocation_pools Property error :
  allocation_pools: 0 ".member.0.start=10.10.10.10" is
  not a map

Note, that it seems to have passed template validation (if the parameter type for the allocation_pools parameter in networks-wrapper.yaml is `json`, for instance, validation simply fails). Apparently the data structure was not recognized as such. Instead, some kind of serialized representation was passed to OS::Neutron::Subnet, which tripped it up.

Another observation: The problem seems to lie with parameters actually passed to the template. If the parameter is not being passed, causing the default value for allocation_pools in networks-wrapper.yaml to be used, the stack will be created without any problems.

Revision history for this message
Johannes Grassler (jgr-launchpad) wrote :
description: updated
description: updated
description: updated
Revision history for this message
Steven Hardy (shardy) wrote :

This is similar to an issue observed by dprince working on tripleo-heat-templates, where we want to pass a server list from a ResourceGroup into a nested stack containing a SoftwareDeployments resource.

I'm looking into a way to fix that, so I'll take this & see if a common solution is possible.

Changed in heat:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Steven Hardy (shardy)
tags: added: tripleo
Steven Hardy (shardy)
Changed in heat:
status: Triaged → Confirmed
Revision history for this message
Steven Hardy (shardy) wrote :

So the problem here seems to be lists specifically - the tripleo use-case mentioned actually works OK, you can pass a map from the parent to the child template via the parameter type "json" without issues.

However, in properties.Schema.from_parameter() we make a 1-1 mapping between "json" paramter type and the "MAP" properties type in the generated schema.

This is actually a bit tricky, given that the json parameter can actually accept any one of three types:

1. String (serialized map or list)
2. Map
3. List

So any of those formats works when interacting via top-level parameters, but only (2) works when the parameter is accessed via a TemplateResource.

I'm looking into the following possible solution:

  * Make string type properties capable of automatically serializing any map or list
  * Move from_parameter to map "json" parameters to a string instead of map

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/157062

Changed in heat:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/157062
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=413fde3247b804e408a9ea9fd863d85391e326aa
Submitter: Jenkins
Branch: master

commit 413fde3247b804e408a9ea9fd863d85391e326aa
Author: Steven Hardy <email address hidden>
Date: Wed Feb 18 16:53:43 2015 +0000

    Allow lists and strings for Json parameters via provider resources

    Currently we have a somewhat bogus internal equivalence between the
    "json" parameter type and the MAP property type. This is a problem
    when exposing json parameters via provider resources, because the
    schema of the exposed property does not exactly match the capabilities
    of the underlying parameter.

    Specifically, json parameters can take strings (json serialized list or
    map), and also lists and maps directly. Currently, we can only pass
    maps, which means some data, for example a list of maps (which is possible
    when interacting directly with a json parameter) is not possible when
    using the same parameter abstracted via a TemplateResource.

    To work around this add a flag which relaxes the type rules and enables
    some coercion of map values, only when they're going via a schema derived
    from a json parameter.

    Change-Id: I8eff36c4343a07b644b84aa9f2f74eceeb62b4a9
    Closes-Bug: #1420196

Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/162587

Thierry Carrez (ttx)
Changed in heat:
milestone: none → kilo-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in heat:
milestone: kilo-3 → 2015.1.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/juno)

Reviewed: https://review.openstack.org/162587
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=40bfa60d84aa38fef5a7c8d278641bfeefad4bb7
Submitter: Jenkins
Branch: stable/juno

commit 40bfa60d84aa38fef5a7c8d278641bfeefad4bb7
Author: Steven Hardy <email address hidden>
Date: Mon May 18 12:02:35 2015 +0200

    Allow lists and strings for Json parameters via provider resources

    Currently we have a somewhat bogus internal equivalence between the
    "json" parameter type and the MAP property type. This is a problem
    when exposing json parameters via provider resources, because the
    schema of the exposed property does not exactly match the capabilities
    of the underlying parameter.

    Specifically, json parameters can take strings (json serialized list or
    map), and also lists and maps directly. Currently, we can only pass
    maps, which means some data, for example a list of maps (which is possible
    when interacting directly with a json parameter) is not possible when
    using the same parameter abstracted via a TemplateResource.

    To work around this add a flag which relaxes the type rules and enables
    some coercion of map values, only when they're going via a schema derived
    from a json parameter.

    Changes required for backport:

    * Changed import of oslo_serialization.jsonutils to
      heat.openstack.common.jsonutils.

    Change-Id: I8eff36c4343a07b644b84aa9f2f74eceeb62b4a9
    Closes-Bug: #1420196
    (cherry-picked from 413fde3247b804e408a9ea9fd863d85391e326aa)

tags: added: in-stable-juno
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.