O->P fails during docker upgrade on docker/services/pacemaker/mysql.

[Sofer Athlan-Guyot]

Hi,

the error can be seen in[1]. The relevant part is:

            "Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend",
            "2017-11-30 18:50:09,849 ERROR: 122732 -- + mkdir -p /etc/puppet",
            "+ '[' -n file,file_line,concat,augeas,cron,file ']'",
            "+ TAGS='--tags file,file_line,concat,augeas,cron,file'",
            "+ /usr/bin/puppet apply --detailed-exitcodes --color=false --logdest syslog --logdest console --tags file,file_line,concat,augeas,cron,file /etc/config.pp",
            "Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at [\"/etc/puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql_bundle.pp\", 119]:[\"/etc/config.pp\", 4]",
            " (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')",
            "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql.pp\", 97]:[\"/etc/config.pp\", 4]",
            "Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
            "Warning: ModuleLoader: module 'cinder' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
            "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/cinder/manifests/db/mysql.pp\", 43]:[\"/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql.pp\", 172]",
            "Warning: ModuleLoader: module 'glance' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
            "Warning: ModuleLoader: module 'heat' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
            "Warning: ModuleLoader: module 'keystone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
            "Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
            "Warning: ModuleLoader: module 'nova' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
            "Warning: Unknown variable: '::nova::db::mysql_api::setup_cell0'. at /etc/puppet/modules/nova/manifests/db/mysql.pp:53:28",
            "Error: Evaluation Error: Resource type not found: Pacemaker::Resource::Bundle at /etc/puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql_bundle.pp:274:23 on node upstream-centos-7-2-node-rdo-cloud-tripleo-54521-22496.localdomain",

[1] https://logs.rdoproject.org/08/523408/6/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z144e9e1f762c4aeba5d375dca6ef5631/undercloud/home/jenkins/failed_upgrade.log.txt.gz

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Puppet-pacemaker is comming from delorean and not from delorean-current[1]. Maybe that's the problem.

[1] https://logs.rdoproject.org/08/523408/6/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z144e9e1f762c4aeba5d375dca6ef5631/subnode-2/var/log/extra/yum-list-installed.txt.gz

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Hum ... unlikely though. The pacemaker version is:

 puppet-pacemaker.noarch 0.6.1-0.20171124095828.5f56a96.el7.centos @delorean

and the manifest/resource/bundle.pp file exists since 2017-05-23...

[Jose Luis Franco (jfrancoa)]

When comparing the puppet-pacemaker package used in master and stable/pike upgrades jobs, we can see that they belong to different branches:

 - master: puppet-pacemaker.noarch 0.6.1-0.20171103172230.9a46ecd.el7.centos @delorean

The git hash 9a46ecd (Pin jwt to 1.5.6 and rubocop to 0.50.0) can be found in puppet-pacemaker master branch https://github.com/openstack/puppet-pacemaker/commits/master

 - pike: puppet-pacemaker.noarch 0.6.1-0.20171124095828.5f56a96.el7.centos @delorean
This other hash 5f56a96, can't be found in master branch, but it matches with the last commit in stable/0.6.x branch (zuul: run ovb-ha-oooq) https://github.com/openstack/puppet-pacemaker/commits/stable/0.6.x

When comparing the content of both branches, master vs stable/0.6.x the failing class puppet::resource::bundle (manifest/resource/bundle.pp) is missing in the stable/0.6.x branch:

 - master: https://github.com/openstack/puppet-pacemaker/tree/master/manifests/resource

 - stable/0.6.x: https://github.com/openstack/puppet-pacemaker/tree/stable/0.6.x/manifests/resource

So, the source of the problem seems to be in the puppet-pacemaker version being installed in stable/pike.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Maybe https://review.openstack.org/#/q/I963aede41403ebbe3b9afb55a725b304a30a0cbb would solve this.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Fix in that release https://review.openstack.org/#/c/540473/ by using the right containers for each release.