Deployment fails when HeatApi and Keystone services run on different nodes

[Marius Cornea]

[stack@undercloud ~]$ cat templates/deploy.command
source ~/stackrc
export THT=~/templates/tripleo-heat-templates/
openstack overcloud deploy --templates $THT \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/templates/network-environment.yaml \
-e $THT/environments/puppet-pacemaker.yaml \
-e ~/templates/services.yaml \
--control-scale 3 \
--control-flavor controller \
--compute-scale 1 \
--compute-flavor compute \
--block-storage-scale 3 \
--block-storage-flavor block \
--ntp-server clock.redhat.com

[stack@undercloud ~]$ cat templates/services.yaml
parameter_defaults:
  ControllerServices:
    - OS::TripleO::Services::CephMon
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::CinderBackup
    - OS::TripleO::Services::CinderVolume
    - OS::TripleO::Services::Core
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::MySQL
    - OS::TripleO::Services::NeutronDhcpAgent
    - OS::TripleO::Services::NeutronL3Agent
    - OS::TripleO::Services::NeutronMetadataAgent
    - OS::TripleO::Services::NeutronOvsAgent
    - OS::TripleO::Services::RabbitMQ
    - OS::TripleO::Services::HAproxy
    - OS::TripleO::Services::Keepalived
    - OS::TripleO::Services::Memcached
    - OS::TripleO::Services::Pacemaker
    - OS::TripleO::Services::Redis
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::SwiftStorage
    - OS::TripleO::Services::SwiftRingBuilder
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::Tripleo::Services::ManilaShare
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::HeatApi

  BlockStorageServices:
    - OS::TripleO::Services::MongoDb
    - OS::TripleO::Services::AodhApi
    - OS::TripleO::Services::AodhEvaluator
    - OS::TripleO::Services::AodhNotifier
    - OS::TripleO::Services::AodhListener
    - OS::TripleO::Services::SaharaApi
    - OS::TripleO::Services::SaharaEngine
    - OS::TripleO::Services::IronicApi
    - OS::TripleO::Services::IronicConductor
    - OS::TripleO::Services::NovaIronic
    - OS::TripleO::Services::CeilometerApi
    - OS::TripleO::Services::CeilometerCollector
    - OS::TripleO::Services::CeilometerExpirer
    - OS::TripleO::Services::CeilometerAgentCentral
    - OS::TripleO::Services::CeilometerAgentNotification
    - OS::TripleO::Services::Horizon
    - OS::TripleO::Services::GnocchiApi
    - OS::TripleO::Services::GnocchiMetricd
    - OS::TripleO::Services::GnocchiStatsd
    - OS::Tripleo::Services::ManilaApi
    - OS::Tripleo::Services::ManilaScheduler
    - OS::TripleO::Services::SwiftProxy
    - OS::TripleO::Services::NovaApi
    - OS::TripleO::Services::NovaScheduler
    - OS::TripleO::Services::NovaConsoleauth
    - OS::TripleO::Services::NovaVncproxy
    - OS::TripleO::Services::NovaConductor
    - OS::TripleO::Services::NeutronApi
    - OS::TripleO::Services::NeutronCorePlugin
    - OS::TripleO::Services::Keystone
    - OS::TripleO::Services::GlanceApi
    - OS::TripleO::Services::GlanceRegistry
    - OS::TripleO::Services::HeatApiCfn
    - OS::TripleO::Services::HeatApiCloudwatch
    - OS::TripleO::Services::HeatEngine
    - OS::TripleO::Services::CinderScheduler
    - OS::TripleO::Services::CinderApi
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall

Deployment fails with the following error:

overcloud.ControllerNodesPostDeployment.ControllerOvercloudServicesDeployment_Step5.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 5dd7576c-a5f1-4440-9a6e-2d04ea7fda2a
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |

  deploy_stderr: |
    ...
    Error: Could not find resource 'Class[Keystone::Roles::Admin]' for relationship on 'Class[Heat::Keystone::Domain]' on node overcloud-controller-0.localdomain
    Error: Could not find resource 'Class[Keystone::Roles::Admin]' for relationship on 'Class[Heat::Keystone::Domain]' on node overcloud-controller-0.localdomain
    (truncated, view all with --long)
overcloud.ComputeNodesPostDeployment.ComputeOvercloudServicesDeployment_Step4.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 3fbc7dba-3b90-42bf-89c1-ba757cc197e1
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
None
  deploy_stderr: |
None

[Alex Schultz (alex-schultz)]

You appear to be missing the keystone role on your controller, is this intentional?

[Marius Cornea (mcornea)]

Yes, that was on purpose. I believe the keystone role shouldn't be locked on the controller node or am I making a wrong assumption?

[Steven Hardy (shardy)]

This appears to be a similar error to https://bugs.launchpad.net/tripleo/+bug/1620829 - I think there's still some assumptions in the puppet layer about keystone being on the same box as the service.

The assumption re Controller role is OK - it should be valid to run keystone elsewhere, but I think for now you'll have an easier time always running OS::TripleO::Services::Keystone on the Controller role (not BlockStorage as above), then we'll get non-keystone stuff to run happily on non-controller roles as a first step.