Merge of parallel SoftwareDeployment resources

[Mike Spreitzer]

When, as in heat-templates/hot/software-config/example-templates/example-script-template.yaml, there are two SoftwareDeployment resources that reference the same SoftwareConfig and the same server, I see two problems:

(1) each deployment is invoked for every action that is listed in EITHER deployment, and

(2) the pseudo input named "action" is set correctly for only one of the two deployments.

Following are details of an example.

Here are the two SoftwareDeployment resources from that template:

  deployment:
    type: OS::Heat::SoftwareDeployment
    properties:
      config:
        get_resource: config
      server:
        get_resource: server
      input_values:
        foo: fooooo
        bar: baaaaa

  other_deployment:
    type: OS::Heat::SoftwareDeployment
    properties:
      config:
        get_resource: config
      server:
        get_resource: server
      input_values:
        foo: fu
        bar: barmy
      actions:
      - CREATE
      - UPDATE
      - SUSPEND
      - RESUME

The config in this case is based on a bash script. I modified the script to use `logger` to note the invocation remotely. Here is the line from the modified script:

logger -n 172.18.76.4 Hello from example-script.sh on $(hostname) = $deploy_server_id during $deploy_action with foo=$foo and bar=$bar

I created a stack, then suspended it, then resumed it, then deleted it. Here is what I found on the syslog server:

Sep 4 05:58:13 172.24.4.2 <someone>: Hello from example-script.sh on e3-server-txnz7ajpgjn2 = d55d03ab-3497-49e7-bd5d-04cae5896c42 during CREATE with foo=fooooo and bar=baaaaa

Sep 4 05:58:16 172.24.4.2 <someone>: Hello from example-script.sh on e3-server-txnz7ajpgjn2 = d55d03ab-3497-49e7-bd5d-04cae5896c42 during CREATE with foo=fu and bar=barmy

Sep 4 06:01:35 172.24.4.2 <someone>: Hello from example-script.sh on e3-server-txnz7ajpgjn2 = d55d03ab-3497-49e7-bd5d-04cae5896c42 during CREATE with foo=fooooo and bar=baaaaa

Sep 4 06:01:39 172.24.4.2 <someone>: Hello from example-script.sh on e3-server-txnz7ajpgjn2 = d55d03ab-3497-49e7-bd5d-04cae5896c42 during SUSPEND with foo=fu and bar=barmy

Sep 4 06:02:11 172.24.4.2 <someone>: Hello from example-script.sh on e3-server-txnz7ajpgjn2 = d55d03ab-3497-49e7-bd5d-04cae5896c42 during CREATE with foo=fooooo and bar=baaaaa

Sep 4 06:02:15 172.24.4.2 <someone>: Hello from example-script.sh on e3-server-txnz7ajpgjn2 = d55d03ab-3497-49e7-bd5d-04cae5896c42 during RESUME with foo=fu and bar=barmy

[Mike Spreitzer (mike-spreitzer)]

Here is another example. I modified one of the two deployments, giving it an action list that PARTLY overlaps with that of the other deployment. This is what those two now look like:

  deployment:
    type: OS::Heat::SoftwareDeployment
    properties:
      config:
        get_resource: config
      server:
        get_resource: server
      input_values:
        foo: fooooo
        bar: baaaaa
        logserver: {get_param: syslog_server}
      actions:
      - CREATE
      - DELETE

  other_deployment:
    type: OS::Heat::SoftwareDeployment
    properties:
      config:
        get_resource: config
      server:
        get_resource: server
      input_values:
        foo: fu
        bar: barmy
        logserver: {get_param: syslog_server}
      actions:
      - CREATE
      - UPDATE
      - SUSPEND
      - RESUME

With this revised template I did a stack-create, action-suspend, action-resume, and stack-delete. I got 8 messages at the syslog server:

Sep 4 06:38:35 172.24.4.2 <someone>: Hello from example-script.sh on e5-server-k4jdyc27n3h4 = 7f575102-cbba-4d89-9223-6204e0d336bb during CREATE with foo=fooooo and bar=baaaaa

Sep 4 06:38:39 172.24.4.2 <someone>: Hello from example-script.sh on e5-server-k4jdyc27n3h4 = 7f575102-cbba-4d89-9223-6204e0d336bb during CREATE with foo=fu and bar=barmy

Sep 4 06:40:50 172.24.4.2 <someone>: Hello from example-script.sh on e5-server-k4jdyc27n3h4 = 7f575102-cbba-4d89-9223-6204e0d336bb during CREATE with foo=fooooo and bar=baaaaa

Sep 4 06:40:53 172.24.4.2 <someone>: Hello from example-script.sh on e5-server-k4jdyc27n3h4 = 7f575102-cbba-4d89-9223-6204e0d336bb during SUSPEND with foo=fu and bar=barmy

Sep 4 06:41:23 172.24.4.2 <someone>: Hello from example-script.sh on e5-server-k4jdyc27n3h4 = 7f575102-cbba-4d89-9223-6204e0d336bb during CREATE with foo=fooooo and bar=baaaaa

Sep 4 06:41:27 172.24.4.2 <someone>: Hello from example-script.sh on e5-server-k4jdyc27n3h4 = 7f575102-cbba-4d89-9223-6204e0d336bb during RESUME with foo=fu and bar=barmy

Sep 4 06:42:28 172.24.4.2 <someone>: Hello from example-script.sh on e5-server-k4jdyc27n3h4 = 7f575102-cbba-4d89-9223-6204e0d336bb during DELETE with foo=fooooo and bar=baaaaa

Sep 4 06:42:32 172.24.4.2 <someone>: Hello from example-script.sh on e5-server-k4jdyc27n3h4 = 7f575102-cbba-4d89-9223-6204e0d336bb during RESUME with foo=fu and bar=barmy

[Mike Spreitzer (mike-spreitzer)]

I modified again, this time cloning the SoftwareConfig so that each is used by exactly one SoftwareDeployment. In this case the heat engine gets errors, some from trying to find a software config identified as None.

[Steve Baker (steve-stevebaker)]

Could you please attach the modified bash config?

[Mike Spreitzer (mike-spreitzer)]

which one? The one used at first, or the one used in comment #2?

[Mike Spreitzer (mike-spreitzer)]

I have been through several iterations, not saving each. Following is the current version of the template with two distinct SoftwareConfigs and a SoftwareDeployment for each. I used a syslog server on my host VM. This is in a DevStack install inside a single host VM. That may have something to do with why it worked despite the security group.

heat_template_version: 2013-05-23
parameters:
  key_name:
    type: string
  flavor:
    type: string
    default: m1.small
  image:
    type: string
    default: fedora-software-config
  syslog_server:
    type: string
    description: dotted decimal IPv4 address of external syslog server

resources:
  the_sg:
    type: OS::Neutron::SecurityGroup
    properties:
      name: the_sg
      description: Ping and SSH
      rules:
      - protocol: icmp
      - protocol: tcp
        port_range_min: 22
        port_range_max: 22

  config:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      inputs:
      - name: foo
      - name: bar
      - name: logserver
      outputs:
      - name: result
      config: |
        #!/bin/sh -x
        echo "Writing to /tmp/$bar"
        echo $foo > /tmp/$bar
        echo -n "The file /tmp/$bar contains `cat /tmp/$bar` for server $deploy_server_id during $deploy_action" > $heat_outputs_path.result
        logger -n $logserver Hello from config on $(hostname) = $deploy_server_id during $deploy_action with foo=$foo and bar=$bar
        echo "Written to /tmp/$bar"
        echo "Output to stderr" 1>&2

  other_config:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      inputs:
      - name: foo
      - name: bar
      - name: logserver
      outputs:
      - name: result
      config: |
        #!/bin/sh -x
        echo "Writing to /tmp/$bar"
        echo $foo > /tmp/$bar
        echo -n "The file /tmp/$bar contains `cat /tmp/$bar` for server $deploy_server_id during $deploy_action" > $heat_outputs_path.result
        logger -n $logserver Hello from other_config on $(hostname) = $deploy_server_id during $deploy_action with foo=$foo and bar=$bar
        echo "Written to /tmp/$bar"
        echo "Output to stderr" 1>&2

  deployment:
    type: OS::Heat::SoftwareDeployment
    properties:
      config:
        get_resource: config
      server:
        get_resource: server
      input_values:
        foo: fooooo
        bar: baaaaa
        logserver: {get_param: syslog_server}
      actions:
      - CREATE

  other_deployment:
    type: OS::Heat::SoftwareDeployment
    properties:
      config:
        get_resource: other_config
      server:
        get_resource: server
      input_values:
        foo: fu
        bar: barmy
        logserver: {get_param: syslog_server}
      actions:
      - DELETE

  server:
    type: OS::Nova::Server
    properties:
      image: {get_param: image}
      flavor: {get_param: flavor}
      key_name: {get_param: key_name}
      security_groups:
      - {get_resource: the_sg}
      user_data_format: SOFTWARE_CONFIG

outputs:
  result:
    value:
      get_attr: [deployment, result]
  stdout:
    value:
      get_attr: [deployment, deploy_stdout]
  stderr:
    value:
      get_attr: [deployment, dep...

[Steve Baker (steve-stevebaker)]

Moving to heat-templates, since this is likely a 55-heat-config issue.

I'm in the process of writing a unit test framework for the hooks, so we should be able to capture this bug as a unit test.

[Steve Baker (steve-stevebaker)]

I think this is essentially a duplicate of 1376008

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/140886

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140886
Committed: https://git.openstack.org/cgit/openstack/heat-templates/commit/?id=2b33ca539f2bade702c07e18d86220a8148fa29f
Submitter: Jenkins
Branch: master

commit 2b33ca539f2bade702c07e18d86220a8148fa29f
Author: Steve Baker <email address hidden>
Date: Thu Dec 11 11:02:15 2014 +1300

    Write each deployed config json to file

    Previously 55-heat-config assumed that executing a hook with some
    config is an idempotent operation, so it made no effort to prevent
    a hook being invoked with the same config multiple times. This means
    that whenever *any* deployment metadata changes *all* configs are
    run against their hooks again. This would be undesirable for
    non-idempotent configs, or configs which are expensive to execute.

    This change writes out the config json to files in
    /var/run/heat-config/deployed and uses the presence of this file to
    determine whether that config has been deployed already. This also
    improves the debugging experience as a single hook execution can be
    triggered manually by running:
      /var/lib/heat-config/hooks/<hook> < /var/run/heat-config/deployed/<cid>.json

    Closes-Bug: #1376008
    Closes-Bug: #1365302

    Change-Id: Id2d2f623508be3049a7db8a39f5444ccac9257d6