heat engine doesn't create new instance when ceilometer alarm is in "alarm" status using autoscaling feature "OS::Heat::AutoScalingGroup"

[Pedro Sousa]

Hi,

I've created this heat template:

heat_template_version: 2013-05-23
description: Scaling Policy test
parameters:
  public_net_id:
    type: string
    description: >
      ID of public network for which floating IP addresses will be allocated
  private_net_id:
    type: string
    description: ID of private network into which servers get deployed
  private_subnet_id:
    type: string
    description: ID of private sub network into which servers get deployed

resources:
  instance_group:
    type: OS::Heat::AutoScalingGroup
    properties:
      desired_capacity: 1
      min_size: 1
      max_size: 5
      resource:
        type: my_scaling_unit.yaml
        properties:
          key_name: psousaxenlab
          image: Cirros
          flavor: m1.tiny
          public_net_id: {get_param: public_net_id}
          private_net_id: {get_param: private_net_id}
          private_subnet_id: {get_param: private_subnet_id}
          metadata: {"metering.stack": {get_param: "OS::stack_id"}}

  web_server_scaleup_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: instance_group}
      cooldown: 60
      scaling_adjustment: 1
  web_server_scaledown_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: instance_group}
      cooldown: 60
      scaling_adjustment: -1
  cpu_alarm_high:
    type: OS::Ceilometer::Alarm
    properties:
      description: Scale-up if the average CPU > 50% for 1 minute
      meter_name: cpu_util
      statistic: avg
      period: 60
      evaluation_periods: 1
      threshold: 50
      alarm_actions:
        - {get_attr: [web_server_scaleup_policy, alarm_url]}
      matching_metadata: {'metadata.user_metadata.stack': {get_param: "OS::stack_id"}}
      comparison_operator: gt
      repeat_actions: True
  cpu_alarm_low:
    type: OS::Ceilometer::Alarm
    properties:
      description: Scale-down if the average CPU < 15% for 10 minutes
      meter_name: cpu_util
      statistic: avg
     period: 600
      evaluation_periods: 1
      threshold: 15
      alarm_actions:
        - {get_attr: [web_server_scaledown_policy, alarm_url]}
      matching_metadata: {'metadata.user_metadata.stack': {get_param: "OS::stack_id"}}
      comparison_operator: lt
      repeat_actions: True

Everything works fine, the alarm get created, however when I stress the cpu inside my instance nothing happens and doesn't autoscale, although the ceilometer alarm changes his status from "ok->alarm". Also I don't see nothing in heat-engine logs.

My versions:
openstack-heat-common-2014.1-1.0.el6.noarch
openstack-heat-api-cfn-2014.1-1.0.el6.noarch
openstack-heat-api-cloudwatch-2014.1-1.0.el6.noarch
python-heatclient-0.2.9-1.el6.noarch
openstack-heat-api-2014.1-1.0.el6.noarch
openstack-heat-engine-2014.1-1.0.el6.noarch

openstack-ceilometer-collector-2014.1-2.el6.noarch
openstack-ceilometer-api-2014.1-2.el6.noarch
python-ceilometer-2014.1-2.el6.noarch
openstack-ceilometer-central-2014.1-2.el6.noarch
python-ceilometerclient-1.0.8-1.el6.noarch
openstack-ceilometer-compute-2014.1-2.el6.noarch
openstack-ceilometer-notification-2014.1-2.el6.noarch
openstack-ceilometer-alarm-2014.1-2.el6.noarch
openstack-ceilometer-common-2014.1-2.el6.noarch

Regards
Pgsousa

[Pedro Sousa (pgsousa)]

my_scaling_unit.yaml

heat_template_version: 2013-05-23

description: >
  HOT template to deploy two servers into an existing neutron tenant network and
  assign floating IP addresses to each server so they are routable from the
  public network.

parameters:
  key_name:
    type: string
    description: Name of keypair to assign to servers
  image:
    type: string
    description: Name of image to use for servers
  flavor:
    type: string
    description: Flavor to use for servers
  metadata:
    type: json
  public_net_id:
    type: string
    description: >
      ID of public network for which floating IP addresses will be allocated
  private_net_id:
    type: string
    description: ID of private network into which servers get deployed
  private_subnet_id:
    type: string
    description: ID of private sub network into which servers get deployed

resources:
  server1:
    type: OS::Nova::Server
    properties:
      image: { get_param: image }
      flavor: { get_param: flavor }
      metadata: {get_param: metadata}
      key_name: { get_param: key_name }
      networks:
        - port: { get_resource: server1_port }

  server1_port:
    type: OS::Neutron::Port
    properties:
      network_id: { get_param: private_net_id }
      fixed_ips:
        - subnet_id: { get_param: private_subnet_id }

  server1_floating_ip:
    type: OS::Neutron::FloatingIP
    properties:
      floating_network_id: { get_param: public_net_id }
      port_id: { get_resource: server1_port }

outputs:
  server1_private_ip:
    description: IP address of server1 in private network
    value: { get_attr: [ server1, first_address ] }
  server1_public_ip:
    description: Floating IP address of server1 in public network
    value: { get_attr: [ server1_floating_ip, floating_ip_address ] }

[Thomas Herve (therve)]

Can we get the output of the ceilometer alarm evaluator process, and the heat cfn api too?

Thanks.

[Pedro Sousa (pgsousa)]

# tail -n 200 /var/log/heat/heat-api-cfn.log

2014-05-12 17:34:34.683 6701 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2014-05-12 17:34:34.683 6701 DEBUG root [-] XML response : <ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse> to_xml /usr/lib/python2.6/site-packages/heat/common/wsgi.py:619
2014-05-12 17:35:34.760 6701 DEBUG heat.api.middleware.version_negotiation [-] Processing request: POST /v1/signal/arn:openstack:heat::06f971509c25405e804d33d5c084ae93:stacks/Autoscale16/767c1b6d-4a4a-4a04-baf8-c5e2f02cee64/resources/web_server_scaleup_policy Accept: */* process_request /usr/lib/python2.6/site-packages/heat/api/middleware/version_negotiation.py:53
2014-05-12 17:35:34.762 6701 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 process_request /usr/lib/python2.6/site-packages/heat/api/middleware/version_negotiation.py:68
2014-05-12 17:35:34.762 6701 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2014-05-12 17:35:34.763 6701 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2014-05-12 17:35:34.764 6701 INFO heat.api.aws.ec2token [-] Authenticating with http://127.0.0.1:5000/v2.0/ec2tokens/ec2tokens
2014-05-12 17:35:34.765 6701 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): 127.0.0.1
2014-05-12 17:35:34.775 6701 DEBUG requests.packages.urllib3.connectionpool [-] "POST /v2.0/ec2tokens/ec2tokens HTTP/1.1" 404 93 _make_request /usr/lib/python2.6/site-packages/requests/packages/urllib3/connectionpool.py:344
2014-05-12 17:35:34.776 6701 INFO heat.api.aws.ec2token [-] AWS authentication failure.
2014-05-12 17:35:34.776 6701 DEBUG root [-] XML response : <ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse> to_xml /usr/lib/python2.6/site-packages/heat/common/wsgi.py:619
2014-05-12 17:36:34.671 6701 DEBUG heat.api.middleware.version_negotiation [-] Processing request: POST /v1/signal/arn:openstack:heat::06f971509c25405e804d33d5c084ae93:stacks/Autoscale16/767c1b6d-4a4a-4a04-baf8-c5e2f02cee64/resources/web_server_scaleup_policy Accept: */* process_request /usr/lib/python2.6/site-packages/heat/api/middleware/version_negotiation.py:53
2014-05-12 17:36:34.672 6701 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 process_request /usr/lib/python2.6/site-packages/heat/api/middleware/version_negotiation.py:68
2014-05-12 17:36:34.672 6701 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
2014-05-12 17:36:34.673 6701 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
2014-05-12 17:36:34.674 6701 INFO heat.api.aws.ec2token [-] Authenticating with http://127.0.0.1:5000/v2.0/ec2tokens/ec2tokens
2014-05-12 17:36:34.675 6701 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): 127.0.0.1
2014-05-12 17:36:34.682 6701 DEBUG requests.packages.urllib3.connectionpool [-] "POST /v2.0/ec2tokens/ec2tokens HTTP/1.1" 404 93 _make_request /usr/lib/python2...

[Pedro Sousa (pgsousa)]

update:

I've changed the parameter in heat.conf to:

auth_uri=http://127.0.0.1:5000/v2.0

instead of:

auth_uri=http://127.0.0.1:5000/v2.0/ec2tokens

I use rdo distro (icehouse)

and now it works.

[Mike Spreitzer (mike-spreitzer)]

Please forgive my ignorance of RDO. Are you saying there is a bug in some code that constructs heat.conf? If so, which code? I just looked at my heat.conf, and found it also used "/v2.0" instead of "/v2.0/ec2tokens". I used DevStack to install.

[Mike Spreitzer (mike-spreitzer)]

... and I find that scaling up an OS::Heat::AutoScalingGroup of nested stacks fails me too; the complaint I get is that Nova cannot find the given network (but of course it *did* find the network when the ASG was created).

[Mike Spreitzer (mike-spreitzer)]

I am testing with the templates in https://review.openstack.org/#/c/97366/

[Pedro Sousa (pgsousa)]

It doesn't seem a bug, but a configuration problem, RDO uses puppet recipes
(packstack) to build openstack. I changed the configuration to auth_uri=
http://127.0.0.1:5000/v2.0 (it was auth_uri=
http://127.0.0.1:5000/v2.0/ec2tokens) on heat.conf and now it works.

On Fri, Jun 6, 2014 at 5:56 PM, Mike Spreitzer <email address hidden>
wrote:

> Please forgive my ignorance of RDO. Are you saying there is a bug in
> some code that constructs heat.conf? If so, which code? I just looked
> at my heat.conf, and found it also used "/v2.0" instead of
> "/v2.0/ec2tokens". I used DevStack to install.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1318599
>
> Title:
> heat engine doesn't create new instance when ceilometer alarm is in
> "alarm" status using autoscaling feature "OS::Heat::AutoScalingGroup"
>
> Status in Orchestration API (Heat):
> Incomplete
>
> Bug description:
> Hi,
>
> I've created this heat template:
>
> heat_template_version: 2013-05-23
> description: Scaling Policy test
> parameters:
> public_net_id:
> type: string
> description: >
> ID of public network for which floating IP addresses will be
> allocated
> private_net_id:
> type: string
> description: ID of private network into which servers get deployed
> private_subnet_id:
> type: string
> description: ID of private sub network into which servers get
> deployed
>
>
>
>
> resources:
> instance_group:
> type: OS::Heat::AutoScalingGroup
> properties:
> desired_capacity: 1
> min_size: 1
> max_size: 5
> resource:
> type: my_scaling_unit.yaml
> properties:
> key_name: psousaxenlab
> image: Cirros
> flavor: m1.tiny
> public_net_id: {get_param: public_net_id}
> private_net_id: {get_param: private_net_id}
> private_subnet_id: {get_param: private_subnet_id}
> metadata: {"metering.stack": {get_param: "OS::stack_id"}}
>
> web_server_scaleup_policy:
> type: OS::Heat::ScalingPolicy
> properties:
> adjustment_type: change_in_capacity
> auto_scaling_group_id: {get_resource: instance_group}
> cooldown: 60
> scaling_adjustment: 1
> web_server_scaledown_policy:
> type: OS::Heat::ScalingPolicy
> properties:
> adjustment_type: change_in_capacity
> auto_scaling_group_id: {get_resource: instance_group}
> cooldown: 60
> scaling_adjustment: -1
> cpu_alarm_high:
> type: OS::Ceilometer::Alarm
> properties:
> description: Scale-up if the average CPU > 50% for 1 minute
> meter_name: cpu_util
> statistic: avg
> period: 60
> evaluation_periods: 1
> threshold: 50
> alarm_actions:
> - {get_attr: [web_server_scaleup_policy, alarm_url]}
> matching_metadata: {'metadata.user_metadata.stack': {get_param:
> "OS::stack_id"}}
> comparison_operator: gt
> repeat_actions: True
> cpu_alarm_low:
> type: OS::Ceil...

[Mike Spreitzer (mike-spreitzer)]

In short, this is a bug in RDO.

[Steven Hardy (shardy)]

As per comment #8, this looks like a bug in the puppet manifests used by packstack, not a bug in heat.

The ec2authtoken auth_uri is actually optional, if none is specified, we use the auth_uri specified in the keystone_authtoken instead.

If auth_uri is set in the ec2authtoken section, it's important *not* to append the "ec2tokens" path to the endpoint, because the ec2token middleware already adds that for you:

https://github.com/openstack/heat/blob/master/heat/api/aws/ec2token.py#L75

Heat would work if one of the following is true:

1. Nothing was specified at all in ec2authtoken (uses keystone_authtoken auth_uri)
2. auth_uri in ec2authtoken is http://127.0.0.1:5000/v2.0 or http://127.0.0.1:5000/v2.0/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/98827

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/98827
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=c908b287007092843d9fda50039e0941601974e1
Submitter: Jenkins
Branch: master

commit c908b287007092843d9fda50039e0941601974e1
Author: Steven Hardy <email address hidden>
Date: Mon Jun 9 17:18:55 2014 +0100

    ec2token tolerate fully-qualified ec2authtoken auth_uri

    If (as is currently done by the puppet-heat manifiests) the ec2authtoken
    auth_uri is specified, and it looks like http://127.0.0.1:5000/v2.0/ec2tokens
    ec2token authentication will fail, because we always append "ec2tokens" to
    the auth_uri. Instead, only add it when needed.

    This probably needs to be combined with a change to the puppet manifiests so
    we don't set the ec2authtoken auth_uri, as it's an optional section - we can
    derive the correct path with only the auth_uri from the keystone_authtoken
    section.

    Change-Id: I8c89772b40523b30f1c46b2ca8b68f9e20d5c213
    Closes-Bug: #1318599