tripleo

Deployment fails when non-pacemaker Manila services are deployed on a different role than controller

Bug #1633077 reported by Marius Cornea
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Ben Nemec
tripleo ocata-1 "ocata-1"

Bug Description

Deployment fails when running Manila non-pacemaker services on a different role than controller. The controller role only runs the ManilaShare service as it's managed by Pacemaker. All the other Manila related services run on a custom role.

Deployment fails with the following error:

    Warning: Scope(Class[Keystone::Python]): This class is deprecated, has no effect, and will be removed in Newton
    Error: Please set password for manila service user at /etc/puppet/modules/manila/manifests/keystone/authtoken.pp:227 on node overcloud-controller-0.localdomain
    Error: Please set password for manila service user at /etc/puppet/modules/manila/manifests/keystone/authtoken.pp:227 on node overcloud-controller-0.localdomain

Deploy command:
source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates/

openstack overcloud deploy --templates $THT \
-r ~/openstack_deployment/roles/roles_data.yaml \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e $THT/environments/storage-environment.yaml \
-e $THT/environments/puppet-pacemaker.yaml \
-e $THT/environments/manila-cephfsnative-config.yaml \
-e $THT/environments/services/sahara.yaml \
-e $THT/environments/services/ironic.yaml \
-e $THT/environments/tls-endpoints-public-ip.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/disk-layout.yaml \
-e ~/openstack_deployment/environments/public_vip.yaml \
-e ~/openstack_deployment/environments/enable-tls.yaml \
-e ~/openstack_deployment/environments/inject-trust-anchor.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml \
--log-file overcloud_deployment.log &> overcloud_install.log

The roles_data.yaml:
cat openstack_deployment/deploy.command
source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates/

openstack overcloud deploy --templates $THT \
-r ~/openstack_deployment/roles/roles_data.yaml \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e $THT/environments/storage-environment.yaml \
-e $THT/environments/puppet-pacemaker.yaml \
-e $THT/environments/manila-cephfsnative-config.yaml \
-e $THT/environments/services/sahara.yaml \
-e $THT/environments/services/ironic.yaml \
-e $THT/environments/tls-endpoints-public-ip.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/disk-layout.yaml \
-e ~/openstack_deployment/environments/public_vip.yaml \
-e ~/openstack_deployment/environments/enable-tls.yaml \
-e ~/openstack_deployment/environments/inject-trust-anchor.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml \
--log-file overcloud_deployment.log &> overcloud_install.log

[stack@undercloud-0 ~]$ cat ~/openstack_deployment/roles/roles_data.yaml
- name: Controller
  CountDefault: 1
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephClient
    - OS::TripleO::Services::CinderBackup
    - OS::TripleO::Services::CinderVolume
    - OS::TripleO::Services::Core
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::MySQL
    - OS::TripleO::Services::RabbitMQ
    - OS::TripleO::Services::HAproxy
    - OS::TripleO::Services::Keepalived
    - OS::TripleO::Services::Memcached
    - OS::TripleO::Services::Pacemaker
    - OS::TripleO::Services::Redis
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::Tripleo::Services::ManilaShare
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts

- name: Compute
  CountDefault: 1
  HostnameFormatDefault: '%stackname%-novacompute-%index%'
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephClient
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::NovaCompute
    - OS::TripleO::Services::NovaLibvirt
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::ComputeNeutronCorePlugin
    - OS::TripleO::Services::ComputeNeutronOvsAgent
    - OS::TripleO::Services::ComputeCeilometerAgent
    - OS::TripleO::Services::ComputeNeutronL3Agent
    - OS::TripleO::Services::ComputeNeutronMetadataAgent
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::NeutronSriovAgent
    - OS::TripleO::Services::OpenDaylightOvs
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts

- name: BlockStorage
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CinderVolume
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts

- name: ObjectStorage
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::SwiftStorage
    - OS::TripleO::Services::SwiftRingBuilder
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts

- name: CephStorage
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephOSD
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts

- name: ServiceApi
  CountDefault: 1
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephMon
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::CephRgw
    - OS::TripleO::Services::CinderApi
    - OS::TripleO::Services::CinderScheduler
    - OS::TripleO::Services::Core
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Keystone
    - OS::TripleO::Services::GlanceApi
    - OS::TripleO::Services::GlanceRegistry
    - OS::TripleO::Services::HeatApi
    - OS::TripleO::Services::HeatApiCfn
    - OS::TripleO::Services::HeatApiCloudwatch
    - OS::TripleO::Services::HeatEngine
    - OS::TripleO::Services::NeutronDhcpAgent
    - OS::TripleO::Services::NeutronL3Agent
    - OS::TripleO::Services::NeutronMetadataAgent
    - OS::TripleO::Services::NeutronApi
    - OS::TripleO::Services::NeutronCorePlugin
    - OS::TripleO::Services::NeutronOvsAgent
    - OS::TripleO::Services::NovaConductor
    - OS::TripleO::Services::MongoDb
    - OS::TripleO::Services::NovaApi
    - OS::TripleO::Services::NovaMetadata
    - OS::TripleO::Services::NovaScheduler
    - OS::TripleO::Services::NovaConsoleauth
    - OS::TripleO::Services::NovaVncProxy
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::SwiftProxy
    - OS::TripleO::Services::SwiftStorage
    - OS::TripleO::Services::SwiftRingBuilder
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::CeilometerApi
    - OS::TripleO::Services::CeilometerCollector
    - OS::TripleO::Services::CeilometerExpirer
    - OS::TripleO::Services::CeilometerAgentCentral
    - OS::TripleO::Services::CeilometerAgentNotification
    - OS::TripleO::Services::Horizon
    - OS::TripleO::Services::GnocchiApi
    - OS::TripleO::Services::GnocchiMetricd
    - OS::TripleO::Services::GnocchiStatsd
    - OS::Tripleo::Services::ManilaApi
    - OS::Tripleo::Services::ManilaScheduler
    - OS::Tripleo::Services::ManilaBackendGeneric
    - OS::Tripleo::Services::ManilaBackendNetapp
    - OS::Tripleo::Services::ManilaBackendCephFs
    - OS::TripleO::Services::AodhApi
    - OS::TripleO::Services::AodhEvaluator
    - OS::TripleO::Services::AodhNotifier
    - OS::TripleO::Services::AodhListener
    - OS::TripleO::Services::SaharaApi
    - OS::TripleO::Services::SaharaEngine
    - OS::TripleO::Services::IronicApi
    - OS::TripleO::Services::IronicConductor
    - OS::TripleO::Services::NovaIronic
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::OpenDaylight
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts

Marius Cornea (mcornea)
summary: - Deployment fails when Manila services on a difrent role than controller
+ Deployment fails when non-pacemaker Manila services are deployed on a
+ different role than controller
Revision history for this message
Marius Cornea (mcornea) wrote :

Password related hieradata on the custom role:

[root@overcloud-serviceapi-0 hieradata]# grep -Ri manila | grep pass
service_configs.yaml:manila::compute::nova::nova_admin_password: sGtAcGtZwqgnQMQanzECNzHD6
service_configs.yaml:manila::keystone::auth::password: hZhhrRQfHseeAupTWtCGsDEhu
service_configs.yaml:manila::keystone::authtoken::password: hZhhrRQfHseeAupTWtCGsDEhu
service_configs.yaml:manila::network::neutron::neutron_admin_password: ywvRA99KsJPUzsw9djp6vrxaZ
service_configs.yaml:manila::rabbit_password: FszkWAVPhDAXHmaJ2Z2aWvaqh

Password related hieradata on the controller role:
[root@overcloud-controller-0 hieradata]# grep -Ri manila | grep pass
service_configs.yaml:manila::db::mysql::password: hZhhrRQfHseeAupTWtCGsDEhu
service_configs.yaml:manila::rabbit_password: FszkWAVPhDAXHmaJ2Z2aWvaqh

Steven Hardy (shardy)
Changed in tripleo:
status: New → Triaged
importance: Undecided → High
milestone: none → ocata-1
Revision history for this message
Steven Hardy (shardy) wrote :

I think this needs a similar fix to https://bugs.launchpad.net/tripleo/+bug/1631130

Revision history for this message
Steven Hardy (shardy) wrote :

Marius, can you please confirm that you had https://review.openstack.org/#/c/370573/ applied in the build you were testing?

That should wire in the hieradata you're missing here:

https://review.openstack.org/#/c/370573/10/puppet/services/manila-api.yaml

https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/puppet/services/manila-api.yaml#L73

Revision history for this message
Marius Cornea (mcornea) wrote :

I reproduced the issue in the initial report and I can confirm that the THT version included this patch: https://review.openstack.org/#/c/370573/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/393947

Changed in tripleo:
assignee: nobody → Ben Nemec (bnemec)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/393948

James Slagle (james-slagle)
tags: added: newton-backport-potential
Revision history for this message
Marios Andreou (marios-b) wrote :

note this is also discussed at https://bugzilla.redhat.com/show_bug.cgi?id=1384524

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/393947
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f20c044c73d33dd031ce3f760a5c6dd0f7bfb3c9
Submitter: Jenkins
Branch: master

commit f20c044c73d33dd031ce3f760a5c6dd0f7bfb3c9
Author: Ben Nemec <email address hidden>
Date: Fri Nov 4 12:28:18 2016 -0500

    Include keystone authtoken config in manila-share service

    Because manila-share is a pacemaker-managed service, it has to be
    on the controller node. If you deploy the api services to a
    different node, then manila-share loses access to the authtoken
    hieradata generated by manila-api. Adding it explicitly to the
    manila-share config allows this setup to deploy sanely.

    Note that I'm having a different problem with manila db-syncs in
    this setup, so there's likely another patch required to get it
    fully working.

    Change-Id: Iac782fa67ea912d24b9905dd8bbafb8ff28dd669
    Partial-Bug: 1633077

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/393948
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=16004b9e7e03091dd1295fde193d82a77e68dd02
Submitter: Jenkins
Branch: master

commit 16004b9e7e03091dd1295fde193d82a77e68dd02
Author: Ben Nemec <email address hidden>
Date: Fri Nov 4 14:11:36 2016 -0500

    Move db settings from manila-api to manila-base

    manila-share also needs the db configuration so the db-sync works
    correctly when manila-api is running on a non-controller node.

    Change-Id: Ib8a6f10ef6a650275fc011e51acfc4b5c7c99164
    Closes-Bug: 1633077

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/394439

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/394440

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/newton)

Reviewed: https://review.openstack.org/394439
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2b51ded16099287bae7c11bd5b0a8a5176cb1b7c
Submitter: Jenkins
Branch: stable/newton

commit 2b51ded16099287bae7c11bd5b0a8a5176cb1b7c
Author: Ben Nemec <email address hidden>
Date: Fri Nov 4 12:28:18 2016 -0500

    Include keystone authtoken config in manila-share service

    Because manila-share is a pacemaker-managed service, it has to be
    on the controller node. If you deploy the api services to a
    different node, then manila-share loses access to the authtoken
    hieradata generated by manila-api. Adding it explicitly to the
    manila-share config allows this setup to deploy sanely.

    Note that I'm having a different problem with manila db-syncs in
    this setup, so there's likely another patch required to get it
    fully working.

    Change-Id: Iac782fa67ea912d24b9905dd8bbafb8ff28dd669
    Partial-Bug: 1633077
    (cherry picked from commit f20c044c73d33dd031ce3f760a5c6dd0f7bfb3c9)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/394440
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=08dc04059bb70118497e35e339505896bcd0868e
Submitter: Jenkins
Branch: stable/newton

commit 08dc04059bb70118497e35e339505896bcd0868e
Author: Ben Nemec <email address hidden>
Date: Fri Nov 4 14:11:36 2016 -0500

    Move db settings from manila-api to manila-base

    manila-share also needs the db configuration so the db-sync works
    correctly when manila-api is running on a non-controller node.

    Change-Id: Ib8a6f10ef6a650275fc011e51acfc4b5c7c99164
    Closes-Bug: 1633077
    (cherry picked from commit 16004b9e7e03091dd1295fde193d82a77e68dd02)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 5.1.0

This issue was fixed in the openstack/tripleo-heat-templates 5.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 6.0.0.0b1

This issue was fixed in the openstack/tripleo-heat-templates 6.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 5.1.0

This issue was fixed in the openstack/tripleo-heat-templates 5.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.