When Heat API services run on a different node than controller the heat-dbsync fails due to incorrect mysql permissions

[Marius Cornea]

stack@undercloud ~]$ cat templates/deploy.command
source ~/stackrc
export THT=~/templates/tripleo-heat-templates/
openstack overcloud deploy --templates $THT \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/templates/network-environment.yaml \
-e $THT/environments/puppet-pacemaker.yaml \
-e ~/templates/services.yaml \
--control-scale 3 \
--control-flavor controller \
--compute-scale 1 \
--compute-flavor compute \
--block-storage-scale 3 \
--block-storage-flavor block \
--ntp-server clock.redhat.com

[stack@undercloud ~]$ cat templates/services.yaml
parameter_defaults:
  ControllerServices:
    - OS::TripleO::Services::CephMon
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::CinderBackup
    - OS::TripleO::Services::CinderVolume
    - OS::TripleO::Services::Core
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::MySQL
    - OS::TripleO::Services::NeutronDhcpAgent
    - OS::TripleO::Services::NeutronL3Agent
    - OS::TripleO::Services::NeutronMetadataAgent
    - OS::TripleO::Services::NeutronOvsAgent
    - OS::TripleO::Services::RabbitMQ
    - OS::TripleO::Services::HAproxy
    - OS::TripleO::Services::Keepalived
    - OS::TripleO::Services::Memcached
    - OS::TripleO::Services::Pacemaker
    - OS::TripleO::Services::Redis
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::SwiftStorage
    - OS::TripleO::Services::SwiftRingBuilder
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::Tripleo::Services::ManilaShare
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall

  BlockStorageServices:
    - OS::TripleO::Services::MongoDb
    - OS::TripleO::Services::AodhApi
    - OS::TripleO::Services::AodhEvaluator
    - OS::TripleO::Services::AodhNotifier
    - OS::TripleO::Services::AodhListener
    - OS::TripleO::Services::SaharaApi
    - OS::TripleO::Services::SaharaEngine
    - OS::TripleO::Services::IronicApi
    - OS::TripleO::Services::IronicConductor
    - OS::TripleO::Services::NovaIronic
    - OS::TripleO::Services::CeilometerApi
    - OS::TripleO::Services::CeilometerCollector
    - OS::TripleO::Services::CeilometerExpirer
    - OS::TripleO::Services::CeilometerAgentCentral
    - OS::TripleO::Services::CeilometerAgentNotification
    - OS::TripleO::Services::Horizon
    - OS::TripleO::Services::GnocchiApi
    - OS::TripleO::Services::GnocchiMetricd
    - OS::TripleO::Services::GnocchiStatsd
    - OS::Tripleo::Services::ManilaApi
    - OS::Tripleo::Services::ManilaScheduler
    - OS::TripleO::Services::SwiftProxy
    - OS::TripleO::Services::NovaApi
    - OS::TripleO::Services::NovaScheduler
    - OS::TripleO::Services::NovaConsoleauth
    - OS::TripleO::Services::NovaVncproxy
    - OS::TripleO::Services::NovaConductor
    - OS::TripleO::Services::NeutronApi
    - OS::TripleO::Services::NeutronCorePlugin
    - OS::TripleO::Services::GlanceApi
    - OS::TripleO::Services::GlanceRegistry
    - OS::TripleO::Services::HeatApi
    - OS::TripleO::Services::HeatApiCfn
    - OS::TripleO::Services::HeatApiCloudwatch
    - OS::TripleO::Services::HeatEngine
    - OS::TripleO::Services::Keystone
    - OS::TripleO::Services::CinderScheduler
    - OS::TripleO::Services::CinderApi
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall

The THT directory includes https://review.openstack.org/#/c/366049

Deployment fails with:

[stack@undercloud ~]$ openstack stack failures list overcloud
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
overcloud.BlockStorageNodesPostDeployment.VolumeDeployment_Step3.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 272eefa0-4663-4d72-bae4-aec83cb6cb67
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
  deploy_stdout: |
    ...
    Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: 2016-09-06 11:43:03.227 22559 ERROR oslo_db.sqlalchemy.exc_filters _check_mysql_exception(errinfo)
    Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: 2016-09-06 11:43:03.227 22559 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
    Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: 2016-09-06 11:43:03.227 22559 ERROR oslo_db.sqlalchemy.exc_filters raise InternalError(errno, errorvalue)
    Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: 2016-09-06 11:43:03.227 22559 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1130, u"Host '10.0.0.17' is not allowed to connect to this MariaDB server")
    Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: 2016-09-06 11:43:03.227 22559 ERROR oslo_db.sqlalchemy.exc_filters
    Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.InternalError) (1130, u"Host '10.0.0.17' is not allowed to connect to this MariaDB server")
    Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::begin]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Heat::Engine/Service[heat-engine]/ensure: ensure changed 'stopped' to 'running'
    Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 1 events
    Notice: Finished catalog run in 948.26 seconds
    (truncated, view all with --long)
  deploy_stderr: |
    Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]
    Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]
overcloud.ComputeNodesPostDeployment.ComputeOvercloudServicesDeployment_Step4.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: b873197e-907b-4923-8cd3-e8cb42771cdb
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
None
  deploy_stderr: |
None

Checking the MySQL permissions on the controller nodes:
http://paste.openstack.org/show/567132/

Nevertheless it looks that overcloud-blockstorage-0 also runs a mysql server which seems to contain openstack users permissions:
http://paste.openstack.org/show/567134/

[Emilien Macchi (emilienm)]

In progress by dprince: https://review.openstack.org/#/c/370573/

[Emilien Macchi (emilienm)]

dprince fixed the keystone part, and AFIK is working on the dbsync part.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/376984

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/376984
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7ba552520787860c966900c8c1fd46e2aefd046c
Submitter: Jenkins
Branch: master

commit 7ba552520787860c966900c8c1fd46e2aefd046c
Author: Dan Prince <email address hidden>
Date: Mon Sep 26 13:52:46 2016 -0400

    Move db::mysql into service_config_settings

    This patch movs the various db::mysql hiera settings into a
    'mysql' specific service_config_settings section for each
    service so that these will only get applied on the MySQL service
    node. This follows a similar puppet-tripleo change where we
    create the actual databases for all services locally on
    the MySQL service node to avoid permission issues.

    Change-Id: Ic0692b1f7aa8409699630ef3924c4be98ca6ffb2
    Closes-bug: #1620595
    Depends-On: I05cc0afa9373429a3197c194c3e8f784ae96de5f
    Depends-On: I5e1ef2dc6de6f67d7c509e299855baec371f614d

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 5.0.0.0rc2 release candidate.