Overcloud deploy and standalone deploy fails in CentOS 8 integration jobs.
Task Run init bundle puppet on the host for mysq fails with:
error={"changed": false, "cmd": "puppet apply --detailed-exitcodes --summarize --color=false --modulepath '/etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules' --tags 'pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation' -e '[\"Mysql_datadir\", \"Mysql_user\", \"Mysql_database\", \"Mysql_grant\", \"Mysql_plugin\"].each |String $val| { noop_resource($val) }; include ::tripleo::profile::base::pacemaker; include ::tripleo::profile::pacemaker::database::mysql_bundle'\n", "delta": "0:01:06.700511", "end": "2022-11-27 08:19:17.364950", "failed_when_result": true, "msg": "non-zero return code", "rc": 6, "start": "2022-11-27 08:18:10.664439", "stderr": "Warning: ModuleLoader: module 'tripleo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)\nWarning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5\n (file: /etc/puppet/hiera.yaml)\nWarning: Undefined variable '::deploy_config_name'; \\n (file & line not available)\nWarning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_language.html\\n (file & line not available)\nWarning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at [\"/etc/puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql_bundle.pp\", 249]:[\"unknown\", 1]\n (location: /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:34:in `deprecation')\nWarning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)\nWarning: ModuleLoader: module 'glance' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)\nWarning: ModuleLoader: module 'heat' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)\nWarning: ModuleLoader: module 'keystone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)\nWarning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)\nWarning: ModuleLoader: module 'nova' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)\nWarning: ModuleLoader: module 'placement' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)\nWarning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)\nWarning: This command is deprecated and will be removed. Please use 'pcs property config' instead.\nWarning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.\nWarning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.\nError: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20221127-73016-144x5dc resource create galera ocf:heartbeat:galera log='/var/log/mysql/mysqld.log' additional_parameters='--open-files-limit=16384' enable_creation=true wsrep_cluster_address='gcomm://node-0003304405.internalapi.localdomain' cluster_host_map='node-0003304405:node-0003304405.internalapi.localdomain' meta master-max=1 ordered=true container-attribute-target=host op promote timeout=300s on-fail=block bundle galera-bundle failed: Error: Validation result from agent (use --force to override):. Too many tries\nError: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Ocf[galera]/Pcmk_resource[galera]/ensure: change from 'absent' to 'present' failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20221127-73016-144x5dc resource create galera ocf:heartbeat:galera log='/var/log/mysql/mysqld.log' additional_parameters='--open-files-limit=16384' enable_creation=true wsrep_cluster_address='gcomm://node-0003304405.internalapi.localdomain' cluster_host_map='node-0003304405:node-0003304405.internalapi.localdomain' meta master-max=1 ordered=true container-attribute-target=host op promote timeout=300s on-fail=block bundle galera-bundle failed: Error: Validation result from agent (use --force to override):. Too many tries", "stderr_lines": ["Warning: ModuleLoader: module 'tripleo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)", "Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5", " (file: /etc/puppet/hiera.yaml)", "Warning: Undefined variable '::deploy_config_name'; \\n (file & line not available)", "Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_language.html\\n (file & line not available)", "Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at [\"/etc/puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql_bundle.pp\", 249]:[\"unknown\", 1]", " (location: /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:34:in `deprecation')", "Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)", "Warning: ModuleLoader: module 'glance' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)", "Warning: ModuleLoader: module 'heat' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)", "Warning: ModuleLoader: module 'keystone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)", "Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)", "Warning: ModuleLoader: module 'nova' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)", "Warning: ModuleLoader: module 'placement' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)", "Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)", "Warning: This command is deprecated and will be removed. Please use 'pcs property config' instead.", "Warning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.", "Warning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.", "Error: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20221127-73016-144x5dc resource create galera ocf:heartbeat:galera log='/var/log/mysql/mysqld.log' additional_parameters='--open-files-limit=16384' enable_creation=true wsrep_cluster_address='gcomm://node-0003304405.internalapi.localdomain' cluster_host_map='node-0003304405:node-0003304405.internalapi.localdomain' meta master-max=1 ordered=true container-attribute-target=host op promote timeout=300s on-fail=block bundle galera-bundle failed: Error: Validation result from agent (use --force to override):. Too many tries", "Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Ocf[galera]/Pcmk_resource[galera]/ensure: change from 'absent' to 'present' failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20221127-73016-144x5dc resource create galera ocf:heartbeat:galera log='/var/log/mysql/mysqld.log' additional_parameters='--open-files-limit=16384' enable_creation=true wsrep_cluster_address='gcomm://node-0003304405.internalapi.localdomain' cluster_host_map='node-0003304405:node-0003304405.internalapi.localdomain' meta master-max=1 ordered=true container-attribute-target=host op promote timeout=300s on-fail=block bundle galera-bundle failed: Error: Validation result from agent (use --force to override):. Too many tries"], "stdout": "Notice: Compiled catalog for node-0003304405.localdomain in environment production in 1.68 seconds\nNotice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Property[galera-role-node-0003304405]/Pcmk_property[property-node-0003304405-galera-role]/ensure: created\nNotice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Bundle[galera-bundle]/Pcmk_bundle[galera-bundle]/ensure: created\nNotice: Applied catalog in 61.91 seconds\nChanges:\n Total: 2\nEvents:\n Failure: 1\n Success: 2\n Total: 3\nResources:\n Failed: 1\n Skipped: 171\n Changed: 2\n Out of sync: 3\n Total: 175\nTime:\n Config retrieval: 1.81\n Pcmk bundle: 10.19\n Last run: 1669537157\n Pcmk resource: 45.10\n Pcmk property: 6.33\n Transaction evaluation: 61.89\n Catalog application: 61.91\n Total: 61.91\nVersion:\n Config: 1669537093\n Puppet: 5.5.10", "stdout_lines": ["Notice: Compiled catalog for node-0003304405.localdomain in environment production in 1.68 seconds", "Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Property[galera-role-node-0003304405]/Pcmk_property[property-node-0003304405-galera-role]/ensure: created", "Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Bundle[galera-bundle]/Pcmk_bundle[galera-bundle]/ensure: created", "Notice: Applied catalog in 61.91 seconds", "Changes:", " Total: 2", "Events:", " Failure: 1", " Success: 2", " Total: 3", "Resources:", " Failed: 1", " Skipped: 171", " Changed: 2", " Out of sync: 3", " Total: 175", "Time:", " Config retrieval: 1.81", " Pcmk bundle: 10.19", " Last run: 1669537157", " Pcmk resource: 45.10", " Pcmk property: 6.33", " Transaction evaluation: 61.89", " Catalog application: 61.91", " Total: 61.91", "Version:", " Config: 1669537093", " Puppet: 5.5.10"]}
Logs:
- https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-wallaby/656e9d6/logs/undercloud/home/zuul/standalone_deploy.log
- https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-train/0350cdc/logs/undercloud/home/zuul/standalone_deploy.log
- https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/714aa6a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
- https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-train/df2bd4c/logs/undercloud/home/zuul/overcloud_deploy.log
Affected jobs:
train on CentOS 8:
periodic-tripleo-ci-centos-8-scenario004-standalone-train , periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train, periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-train, periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-train, periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train, periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train, periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train
wallaby on CentOs 8:
periodic-tripleo-ci-centos-8-standalone-wallaby
We see this from: 2022-11-24 13:06:29
