Deployment fails when Keystone and AodhApi services are deployed on different nodes

[Marius Cornea]

source ~/stackrc
export THT=~/templates/tripleo-heat-templates/
openstack overcloud deploy --templates $THT \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/templates/network-environment.yaml \
-e $THT/environments/puppet-pacemaker.yaml \
-e ~/templates/services.yaml \
--control-scale 3 \
--control-flavor controller \
--compute-scale 1 \
--compute-flavor compute \
--block-storage-scale 3 \
--block-storage-flavor block \
--ntp-server clock.redhat.com

[stack@undercloud ~]$ cat templates/services.yaml
parameter_defaults:
  ControllerServices:
    - OS::TripleO::Services::CephMon
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::CinderBackup
    - OS::TripleO::Services::CinderVolume
    - OS::TripleO::Services::Core
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::MySQL
    - OS::TripleO::Services::NeutronDhcpAgent
    - OS::TripleO::Services::NeutronL3Agent
    - OS::TripleO::Services::NeutronMetadataAgent
    - OS::TripleO::Services::NeutronOvsAgent
    - OS::TripleO::Services::RabbitMQ
    - OS::TripleO::Services::HAproxy
    - OS::TripleO::Services::Keepalived
    - OS::TripleO::Services::Memcached
    - OS::TripleO::Services::Pacemaker
    - OS::TripleO::Services::Redis
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::SwiftStorage
    - OS::TripleO::Services::SwiftRingBuilder
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::Tripleo::Services::ManilaShare
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::HeatApi
    - OS::TripleO::Services::Keystone

  BlockStorageServices:
    - OS::TripleO::Services::MongoDb
    - OS::TripleO::Services::AodhApi
    - OS::TripleO::Services::AodhEvaluator
    - OS::TripleO::Services::AodhNotifier
    - OS::TripleO::Services::AodhListener
    - OS::TripleO::Services::SaharaApi
    - OS::TripleO::Services::SaharaEngine
    - OS::TripleO::Services::IronicApi
    - OS::TripleO::Services::IronicConductor
    - OS::TripleO::Services::NovaIronic
    - OS::TripleO::Services::CeilometerApi
    - OS::TripleO::Services::CeilometerCollector
    - OS::TripleO::Services::CeilometerExpirer
    - OS::TripleO::Services::CeilometerAgentCentral
    - OS::TripleO::Services::CeilometerAgentNotification
    - OS::TripleO::Services::Horizon
    - OS::TripleO::Services::GnocchiApi
    - OS::TripleO::Services::GnocchiMetricd
    - OS::TripleO::Services::GnocchiStatsd
    - OS::Tripleo::Services::ManilaApi
    - OS::Tripleo::Services::ManilaScheduler
    - OS::TripleO::Services::SwiftProxy
    - OS::TripleO::Services::NovaApi
    - OS::TripleO::Services::NovaScheduler
    - OS::TripleO::Services::NovaConsoleauth
    - OS::TripleO::Services::NovaVncproxy
    - OS::TripleO::Services::NovaConductor
    - OS::TripleO::Services::NeutronApi
    - OS::TripleO::Services::NeutronCorePlugin
    - OS::TripleO::Services::GlanceApi
    - OS::TripleO::Services::GlanceRegistry
    - OS::TripleO::Services::HeatApiCfn
    - OS::TripleO::Services::HeatApiCloudwatch
    - OS::TripleO::Services::HeatEngine
    - OS::TripleO::Services::CinderScheduler
    - OS::TripleO::Services::CinderApi
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall

Deployment fails with:

[stack@undercloud ~]$ openstack stack failures list overcloud
overcloud.ControllerNodesPostDeployment.ControllerOvercloudServicesDeployment_Step5.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: c051158e-4f90-4445-869d-fd2ff9c30c85
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |

  deploy_stderr: |
    ...
    Error: Must pass password to Class[Aodh::Keystone::Auth] at /etc/puppet/modules/tripleo/manifests/profile/base/keystone.pp:85 on node overcloud-controller-0.localdomain
    Error: Must pass password to Class[Aodh::Keystone::Auth] at /etc/puppet/modules/tripleo/manifests/profile/base/keystone.pp:85 on node overcloud-controller-0.localdomain
    (truncated, view all with --long)
overcloud.ComputeNodesPostDeployment.ComputeOvercloudServicesDeployment_Step4:
  resource_type: OS::Heat::StructuredDeployments
  physical_resource_id: 4c020d56-0693-4eb1-889e-7f53479271a4
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/370573

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/374237

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/374237
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a99c06aa9600c2da548e46ece251fd0181f0283b
Submitter: Jenkins
Branch: master

commit a99c06aa9600c2da548e46ece251fd0181f0283b
Author: Steven Hardy <email address hidden>
Date: Wed Sep 21 14:42:52 2016 +0100

    Tolerate missing keys from role_data in service templates

    Currently we have a few keys which may be considered optional,
    such as monitoring_subscription, logging* and global_config_settings.

    Currently we dereference these directly via get_attr, but this will
    break when heat output validation is fixed, ref bug #1599114 is fixed
    (patches are up for this, so it may be soon).

    Change-Id: If4eed1ca39c10ace9b1cb5ce2dc4b9c70a3dd2f4
    Partial-Bug: #1620829

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/370573
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9d67d7b3b11b688be0b40fc2fcfb1daf5b5157df
Submitter: Jenkins
Branch: master

commit 9d67d7b3b11b688be0b40fc2fcfb1daf5b5157df
Author: Dan Prince <email address hidden>
Date: Thu Sep 15 09:19:15 2016 +0200

    Move keystone::auth into service_config_settings

    This patch moves the keystone::auth settings for all
    services into the new service_config_settings section. This
    is important because we execute the keystone commands via
    puppet only on the role containing the keystone service
    and without these settings it will fail.

    Note that yaql merging/filtering is used here to ensure that
    service_config_settings is optional in service templates,
    and also that we'll only deploy hieradata for a given
    service on a node running the service (the key in
    the service_config_settings map must match the service_name
    in the service template for this to work).

    e.g the following will result in only deploying keystone: 123
    in hiera on the role running the "keystone" service,
    regardless of which service template defines it.

      service_config_settings:
        keystone:
          keystone: 123

    Co-Authored-By: Steven Hardy <email address hidden>
    Change-Id: I0c2fce037a1a38772f998d582a816b4b703f8265
    Closes-bug: 1620829

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 5.0.0.0rc2 release candidate.