non-configurable 0 based index variable in OS::Heat::ResourceGroup

[robfrey]

Currently the index variable in OS::Heat::ResourceGroup is zero-based (starts at 0) and this is not configurable. I have a use case wherein I need instance names created using the index variable to end in an incremental integer that begins with 1 (instancename%index%). Adding an attribute (something like index_start_val) to make the starting value of the index variable configurable would resolve this issue and I suspect (or at least hope) that it would be a minor change to implement and document.

[Steven Hardy (shardy)]

One possibility would be to adjust the existing removal_policies interface so it's also respected on create:

http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::ResourceGroup-prop-removal_policies

  AGroup:
    type: OS::Heat::ResourceGroup
    properties:
      count: 10
      removal_policies: [{'resource_list': ['0']}]
      resource_def:
        type: OS::Heat::RandomString

Here, on update it'll remove "0" from the group and give you a group starting from "1", only at present this interface doesn't influence the create group assignments (which is arguably wrong and not very declarative).

Another possibility if all you want is configurable instance names is just to use a mapping, e.g:

[stack@instack ~]$ cat hostname_map.yaml
heat_template_version: 2015-04-30

parameters:
  HostnameMap:
    type: json
    default:
      '0': ahost-1
      '1': ahost-2

resources:
 AGroup:
    type: OS::Heat::ResourceGroup
    properties:
      count: 2
      resource_def:
        type: server_mapped_name.yaml
        properties:
          name_map: {get_param: HostnameMap}
          index: '%index%'
[stack@instack ~]$ cat server_mapped_name.yaml
heat_template_version: 2015-04-30

parameters:
  name_map:
    type: json
  index:
    type: string

resources:
 server:
    type: OS::Nova::Server
    properties:
      image: overcloud-full
      flavor: baremetal
      name: {get_param: [name_map, {get_param: index}]}

[stack@instack ~]$ cat hostname_map.yaml
heat_template_version: 2015-04-30

parameters:
  HostnameMap:
    type: json
    default:
      '0': ahost-1
      '1': ahost-2

resources:
 AGroup:
    type: OS::Heat::ResourceGroup
    properties:
      count: 2
      resource_def:
        type: server_mapped_name.yaml
        properties:
          name_map: {get_param: HostnameMap}
          index: '%index%'
[stack@instack ~]$ cat server_mapped_name.yaml
heat_template_version: 2015-04-30

parameters:
  name_map:
    type: json
  index:
    type: string

resources:
 server:
    type: OS::Nova::Server
    properties:
      image: overcloud-full
      flavor: baremetal
      name: {get_param: [name_map, {get_param: index}]}

If neither of these approaches is palatable we could look into a start_index property, but personally I see that as a pattern we'd be best to avoid, because non-zero based indexes for things like attribute lookups are likely to prove error prone (yeah I know if we use the removal_policies approach we already have this problem, I just don't want to encourage it..)

[Steven Hardy (shardy)]

Oops, sorry for duplicate paste in previous reply, hopefully you get the idea anyway. :)

[robfrey (rob-frey)]

In this particular use case we are using parameters for the count attribute in the ResoureGroup resources as a scaling mechanism. We scale out the number of instances in the ResoureGroups with a simple stack-update command which specifies a new count via a parameter to our template whenever an update to scale out is triggered from an external monitoring platform which makes our scaling decisions for us. This is in part because we wanted to utilize KPIs that don't exist as alarms in Ceilometer, but also because we need to execute approval workflows and create tickets to track the event before actually executing the scaling event itself on our platform. In addition the index variable is not currently supported by AutoScalingGroup resources at this time so leveraging webhooks for scaling resources wasn't possible for our use case. In light of this, because we don't know what the eventual count could end up being in the future, using a map isn't as palatable as it would be otherwise.

I will be testing with the other suggestion you made to see if we can put it to use by immediately performing an update after the create is issued, although I'm not sure I have fully wrapped my head around it quite yet I'll be following up with the results soon.

I'm not sure I fully understand the concerns you addressed with attribute lookups either, if you have time and care to elaborate a little to help me understand them better that would very much be appreciated. Sometimes it takes me a little while to fully grasp Heat concepts and constructs that are new to me, its is a fairly complex solution with a large number of interdependencies.

Thanks for the suggestions and the quick reply.

[Steven Hardy (shardy)]

> I'm not sure I fully understand the concerns you addressed with attribute lookups either

The ResourceGroup can output a list, which exposes a given attribute for all resources inside the group - if we switch to a non-zero based index (even optionally), I'm worried we'll spend more time helping folks debug obscure issues where they forgot that the attributes in the list are all out-by-one relative to the resource name, e.g:

  {get_attr: [agroup, value, 2]}

Would actually give you the value for the "1" resource in the group, which is pretty weird if you're not paying attention and/or didn't originally write the templates.

[Steven Hardy (shardy)]

Wait, that's wrong, the list lookup 2 will return the value for the "3" resource if you start the index at 1, vs the currently intuitive "2" which matches the index.

[Steven Hardy (shardy)]

I chatted with ramishra (who my have a patch for this) on IRC, and I clarified some of my concerns above (which, to be clear, I'm not blocking on, only saying we should consider the potential confusion and document carefully).

Here's an example of why I'm concerned.

heat_template_version: 2013-05-23

resources:
  random_group:
    type: OS::Heat::ResourceGroup
    properties:
      count: 2
      resource_def:
        type: OS::Heat::RandomString

outputs:
  resource0_id:
    value: { get_attr: [random_group, resource.0] }
  resource1_id:
    value: { get_attr: [random_group, resource.1] }
  resource0_value:
    value: { get_attr: [random_group, resource.0.value] }
  resource1_value:
    value: { get_attr: [random_group, resource.1.value] }
  all_values_0:
    value: { get_attr: [random_group, value, 0] }
  all_values_1:
    value: { get_attr: [random_group, value, 1] }
  # This name based lookup doesn't work, it must be a list index!
  all_values_0s:
    value: { get_attr: [random_group, value, '0'] }
  all_values_1s:
    value: { get_attr: [random_group, value, '1'] }

Here we can see the following is equivalent:

  get_attr: [random_group, resource.0.value]

  get_attr: [random_group, value, 0]

But there is a subtle difference - the first takes the resource name "0", and the second takes an integer index to the list of all values.

Now consider if the start index of the RG is 1, these become equivalent:

  get_attr: [random_group, resource.1.value]

  get_attr: [random_group, value, 0]

Note the list index lookup still starts at zero!

I'm willing to bet this will confuse some folks, so we'd have to document this gotcha carefully.

[robfrey (rob-frey)]

I understand the concerns much better now. In light of this, is there some way we could instead introduce the concept of a displayOffset attribute or something to that affect that could be added or subtracted to keep things consistent instead of using a startingValue attribute? Not sure what affect that would have or what that would actually look like in the code or if its even a viable option, I'm just brainstorming here.

[Zane Bitter (zaneb)]

removal_policies is respected on create starting with Newton (bug 1585206), so I think we can call this done.