Server metadata not updated in autoscalinggroup

[Rushi Agrawal]

I created an OS::Heat::AutoScalingGroup, with manual scaling. I am also passing (nova) metadata to the servers in that autoscaling group. But when I change the metadata, the changes are not reflected inside those servers when the stack-update completes. BUT, when I increase the number of servers and update it, the new metadata appears in the newly created servers ONLY.

I am assuming Heat templates are a declarative way to define the stack, so as per the definition, same metadata should be available to all the servers in an autoscaling group

Snip of the template which I'm using to update metadata:

  my_asg:
    type: OS::Heat::AutoScalingGroup
    properties:
      resource:
        type: OS::Nova::Server
        properties:
          image: { get_param: image }
          flavor: { get_param: flavor }
          key_name: { get_param: key_name }
          admin_user: { get_param: admin_user }
          metadata: {"key1": "value1"}
          user_data_format: RAW
          user_data:
            str_replace:
              template: |
                #!/bin/bash -v
                # Assumption is cassandra is already installed
                sleep 180
                SELF_IP=$(ifconfig eth0 | grep "inet addr:" | cut -d ':' -f2 | cut -d ' ' -f1)
                service cassandra stop
                rm -rf /var/lib/cassandra/data/system/*
                sed -i s/127.0.0.1/seed_ip/g /etc/cassandra/cassandra.yaml
                sed -i s/localhost/$SELF_IP/g /etc/cassandra/cassandra.yaml
                service cassandra start
              params:
                seed_ip: { get_attr: [server, networks, private-gre, 0]}
          networks: [{network: {get_param: net_name}}]
      min_size: 1
      desired_capacity: 3
      max_size: 10

  scale_up_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: my_asg}
      cooldown: 60
      scaling_adjustment: 1

  scale_down_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: my_asg}
      cooldown: 60
      scaling_adjustment: '-1'

[Thomas Herve (therve)]

I'm assuming you're talking about nova user_data: it's impossbie to update them. If it was, it's unclear what we should do. Do we run cloud init again? Are scripts supposed to be idempotent?

Please reopen if I'm missing something.

[Rushi Agrawal (rushiagr)]

I'm talking about metadata, and not userdata. As can be seen, my yaml file has metadata: {"key1": "value1"}. I'm talking about that. I have come to know that there are two types of metadata -- Heat one and nova one, so I thought I should mention it..

[Thomas Herve (therve)]

OK, thanks for clarifying. Can you tell us how you access nova metadata on the server then?

[Rushi Agrawal (rushiagr)]

curl 169.254.169.254/openstack/2013-10-17/meta_data.json

[Thomas Herve (therve)]

OK, that's what I thought. The metadata service is read-only after the first boot. Even if you can edit the Nova metadata, it won't be reflected on the metadata server.

If anything, this is a bug in Nova.

[Rushi Agrawal (rushiagr)]

@Thomas, I was able to update metadata successfully, post boot, on a OS::Nova::Server resource. The problem was only when there were servers inside an OS::Heat::AutoScalingGroup, so looked to me like a Heat problem..

[vigneshvar (vigneshvar-a-s)]

@Rushi Agrawal
OS::Nova::Server block is executed only when there is a change in the scale up policy.
So lets assume, you update the stack with a change in meta-data only and not in capacity. Now the above mentioned block is not executed at all, as there is no change in capacity. Hence that concludes, the changes in capacity(in your case) triggers the block. i.e only for new instances that are created. Therefore the update may not be applied to previously created instances.

[vigneshvar (vigneshvar-a-s)]

@Rushi Agrawal
But am guessing, if the "desired_capacity" is changed, it should also apply the changes in "meta_data" ,"flavor",etc to all the existing VMs as well. This currently does not happen now anyway.

[vigneshvar (vigneshvar-a-s)]

@Rushi Agrawal
You might make use of OS::Heat::ResourceGroup, if you wish to update the whole group

[Song Li (lisong-cruise)]

Have a question:
a, If we update the stack with new flavor by -P flavor=new_flavor. Does the flavor can be udpate for the exist instance?
b, If the #a works well, how about setting the metadata values as the parameters?

[Ethan Lynn (ethanlynn)]

I test this bug in latest codes, and it's fixed, metadata will update for old instances.