tripleo-ci-centos-8-standalone in stable/wallaby consistently fails because of the failure during creating galera resource

[Takashi Kajinami]

Description
===========
tripleo-ci-centos-8-standalone in stable/wallaby is now consistenly failing.
 https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-standalone&branch=stable%252Fwallaby&skip=0

It seems the standalone deployment fails because of the failure with galera resource creation by puppet.

example:
https://zuul.opendev.org/t/openstack/build/1855d01fe0984c24bc230f36571e07e0

https://zuul.opendev.org/t/openstack/build/1855d01fe0984c24bc230f36571e07e0/log/logs/undercloud/home/zuul/standalone_deploy.log#3227-3228

Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
   (file: /etc/puppet/hiera.yaml)
Warning: Undefined variable '::deploy_config_name';
   (file & line not available)
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html
   (file & line not available)
Warning: This command is deprecated and will be removed. Please use 'pcs property config' instead.
Warning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.
Warning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.
Error: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20221124-90206-ctj9kd resource create galera ocf:heartbeat:galera log='/var/log/mysql/mysqld.log' additional_parameters='--open-files-limit=16384' enable_creation=true wsrep_cluster_address='gcomm://standalone.ctlplane.localdomain' cluster_host_map='standalone:standalone.ctlplane.localdomain' meta master-max=1 ordered=true container-attribute-target=host op promote timeout=300s on-fail=block bundle galera-bundle failed: Error: Validation result from agent (use --force to override):. Too many tries
Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Ocf[galera]/Pcmk_resource[galera]/ensure: change from 'absent' to 'present' failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20221124-90206-ctj9kd resource create galera ocf:heartbeat:galera log='/var/log/mysql/mysqld.log' additional_parameters='--open-files-limit=16384' enable_creation=true wsrep_cluster_address='gcomm://standalone.ctlplane.localdomain' cluster_host_map='standalone:standalone.ctlplane.localdomain' meta master-max=1 ordered=true container-attribute-target=host op promote timeout=300s on-fail=block bundle galera-bundle failed: Error: Validation result from agent (use --force to override):. Too many tries", "stderr_lines": ["Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5", " (file: /etc/puppet/hiera.yaml)", "Warning: Undefined variable '::deploy_config_name'; ", " (file & line not available)", "Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html", " (file & line not available)", "Warning: This command is deprecated and will be removed. Please use 'pcs property config' instead.", "Warning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.", "Warning: This command is deprecated and will be removed. Please use 'pcs constraint config' instead.", "Error: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20221124-90206-ctj9kd resource create galera ocf:heartbeat:galera log='/var/log/mysql/mysqld.log' additional_parameters='--open-files-limit=16384' enable_creation=true wsrep_cluster_address='gcomm://standalone.ctlplane.localdomain' cluster_host_map='standalone:standalone.ctlplane.localdomain' meta master-max=1 ordered=true container-attribute-target=host op promote timeout=300s on-fail=block bundle galera-bundle failed: Error: Validation result from agent (use --force to override):. Too many tries", "Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Ocf[galera]/Pcmk_resource[galera]/ensure: change from 'absent' to 'present' failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20221124-90206-ctj9kd resource create galera ocf:heartbeat:galera log='/var/log/mysql/mysqld.log' additional_parameters='--open-files-limit=16384' enable_creation=true wsrep_cluster_address='gcomm://standalone.ctlplane.localdomain' cluster_host_map='standalone:standalone.ctlplane.localdomain' meta master-max=1 ordered=true container-attribute-target=host op promote timeout=300s on-fail=block bundle galera-bundle failed: Error: Validation result from agent (use --force to override):. Too many tries"], "stdout": "Notice: Compiled catalog for standalone.localdomain in environment production in 2.20 seconds
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Property[galera-role-standalone]/Pcmk_property[property-standalone-galera-role]/ensure: created
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Bundle[galera-bundle]/Pcmk_bundle[galera-bundle]/ensure: created
Notice: Applied catalog in 80.32 seconds
Changes:
            Total: 2
Events:
          Failure: 1
          Success: 2
            Total: 3
Resources:
           Failed: 1
          Skipped: 141
          Changed: 2
      Out of sync: 3
            Total: 145
Time:
      Pcmk bundle: 14.36
         Last run: 1669328593
   Config retrieval: 2.40
    Pcmk resource: 56.58
    Pcmk property: 8.89
   Transaction evaluation: 80.27
   Catalog application: 80.32
            Total: 80.33
Version:
           Config: 1669328511
           Puppet: 7.6.1", "stdout_lines": ["Notice: Compiled catalog for standalone.localdomain in environment production in 2.20 seconds

[Jiri Podivin (jpodivin)]

It appears that there is some sort of parsing error at play.

Log:
----
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_185/865601/1/check/tripleo-ci-centos-8-standalone/1855d01/logs/undercloud/var/log/extra/podman/containers/container-puppet-mysql/stdout.log

Trace:
------
include ::tripleo::packages
['Mysql_datadir', 'Mysql_user', 'Mysql_database', 'Mysql_grant', 'Mysql_plugin'].each |String $val| { noop_resource($val) }
exec {'wait-for-settle': command => '/bin/true' }
include tripleo::profile::pacemaker::database::mysql_bundle
Running puppet
+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,file"' /etc/config.pp
+ logger -s -t puppet-user
<13>Nov 24 22:20:00 puppet-user: Warning: Found multiple default providers for service: swiftinit, pacemaker, base, pacemaker_xml; using swiftinit
<13>Nov 24 22:20:00 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
<13>Nov 24 22:20:00 puppet-user: (file: /etc/puppet/hiera.yaml)
<13>Nov 24 22:20:00 puppet-user: Warning: Undefined variable '::deploy_config_name';
<13>Nov 24 22:20:00 puppet-user: (file & line not available)
<13>Nov 24 22:20:00 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html
<13>Nov 24 22:20:00 puppet-user: (file & line not available)
<13>Nov 24 22:20:00 puppet-user: Could not connect to the CIB: Transport endpoint is not connected
<13>Nov 24 22:20:00 puppet-user: Init failed, could not perform requested operations
<13>Nov 24 22:20:00 puppet-user: -:1: parser error : Document is empty
<13>Nov 24 22:20:00 puppet-user:
<13>Nov 24 22:20:00 puppet-user: ^
<13>Nov 24 22:20:00 puppet-user: Warning: Unknown variable: '::pacemaker::pcs_010'. (file: /etc/puppet/modules/pacemaker/manifests/resource/bundle.pp, line: 159, column: 6)
<13>Nov 24 22:20:00 puppet-user: Notice: Compiled catalog for standalone.localdomain in environment production in 2.37 seconds
<13>Nov 24 22:20:00 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/etc/sysconfig/clustercheck]/ensure: defined content as '{sha256}089a031afe62721961eaec54469382f2409e6208f899d3444da2809306e15024'
<13>Nov 24 22:20:00 puppet-user: Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{sha256}059405bb34a8ec5c13ea553f64cbc450868c0af1b65d608c8cc4a601e4dcc2dd' to '{sha256}f619fa6c8c600bf061999529c2eb34880eb2c96fff28ef227f947a0925b4c41a'
<13>Nov 24 22:20:00 puppet-user: Notice: /Stage[main]/Mysql::Server::Installdb/File[/var/log/mariadb/mariadb.log]/ensure: created
<13>Nov 24 22:20:00 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/root/.my.cnf]/ensure: defined content as '{sha256}6baae0f7392ddab5cdeb48835ece4f217cb7bc1f6c610069fa0981b41544704f'
<13>Nov 24 22:20:00 puppet-user: Notice: Applied catalog in 0.18 seconds
<13>Nov 24 22:20:00 puppet-user: Changes:
<13>Nov 2...

[Jiri Podivin (jpodivin)]

It appears that at least some of the derived jobs are also affected.

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-standalone-tv-validation

[Rabi Mishra (rabi)]

I think it's impacting centos8 wallaby/train check/periodic standalone/multinode jobs.

Some links from the other bug marked as duplicate

  - https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-wallaby/656e9d6/logs/undercloud/home/zuul/standalone_deploy.log
  - https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-train/0350cdc/logs/undercloud/home/zuul/standalone_deploy.log
  - https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/714aa6a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
  - https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-train/df2bd4c/logs/undercloud/home/zuul/overcloud_deploy.log

[Takashi Kajinami (kajinamit)]

As far as I understand the error in comment:1 is not the direct cause.

Looking at the puppet error it seems the pcs command [1] is failing because of some validation error[2].

[1]
pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20221124-90206-ctj9kd resource create galera ocf:heartbeat:galera log='/var/log/mysql/mysqld.log' additional_parameters='--open-files-limit=16384' enable_creation=true wsrep_cluster_address='gcomm://standalone.ctlplane.localdomain' cluster_host_map='standalone:standalone.ctlplane.localdomain' meta master-max=1 ordered=true container-attribute-target=host op promote timeout=300s on-fail=block bundle galera-bundle

[2]
Error: Validation result from agent (use --force to override):. Too many tries"

We initially suspected inconsistent package versions between host and container but we later confirmed the same version is installed in both sides.

[Marios Andreou (marios-b)]

this was filed and marked duplicate https://bugs.launchpad.net/tripleo/+bug/1998080

[Luca Miccini (lmiccini2)]

this looks like a legitimate issue with the latest pcs 0.10.14-6
workaround (before running the standalone deployment):

dnf downgrade -y pcs

or pin it to a previous version, like pcs-0.10.14-5.el8.x86_64.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/865880

[Luca Miccini (lmiccini2)]

I've tried https://review.opendev.org/c/openstack/tripleo-heat-templates/+/865881 and I could deploy a standalone. Maybe we can give this a go.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/865934

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/865935

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/865955

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/865881
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/59300dfb30c5ee9c8186ee4b28f9c57e9b345647
Submitter: "Zuul (22348)"
Branch: master

commit 59300dfb30c5ee9c8186ee4b28f9c57e9b345647
Author: Luca Miccini <email address hidden>
Date: Mon Nov 28 19:38:15 2022 +0100

    Use --force when creating pacemaker resources

    pcs-0.10.14-6 introduced enforce validation at resources creation time.
    Unfortunately this doesn't work in our use case as puppet runs on the
    host but the necessary binaries are installed inside the containers.

    Let's try using --force to workaround.

    Closes-Bug: #1997939

    Change-Id: Id7616ebceb820d9799661c0fbc5f3f234f421ea3

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/865934
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/fde20f941d0452d875ade3c86caf0a7ed34b789f
Submitter: "Zuul (22348)"
Branch: stable/zed

commit fde20f941d0452d875ade3c86caf0a7ed34b789f
Author: Luca Miccini <email address hidden>
Date: Mon Nov 28 19:38:15 2022 +0100

    Use --force when creating pacemaker resources

    pcs-0.10.14-6 introduced enforce validation at resources creation time.
    Unfortunately this doesn't work in our use case as puppet runs on the
    host but the necessary binaries are installed inside the containers.

    Let's try using --force to workaround.

    Closes-Bug: #1997939

    Change-Id: Id7616ebceb820d9799661c0fbc5f3f234f421ea3
    (cherry picked from commit 59300dfb30c5ee9c8186ee4b28f9c57e9b345647)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/865935
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/3fc9478a863980e4448434f413ff3e878ccf04ab
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3fc9478a863980e4448434f413ff3e878ccf04ab
Author: Luca Miccini <email address hidden>
Date: Mon Nov 28 19:38:15 2022 +0100

    Use --force when creating pacemaker resources

    pcs-0.10.14-6 introduced enforce validation at resources creation time.
    Unfortunately this doesn't work in our use case as puppet runs on the
    host but the necessary binaries are installed inside the containers.

    Let's try using --force to workaround.

    Conflicts:
            deployment/rabbitmq/rabbitmq-messaging-rpc-pacemaker-puppet.yaml

    Closes-Bug: #1997939

    Change-Id: Id7616ebceb820d9799661c0fbc5f3f234f421ea3
    (cherry picked from commit 59300dfb30c5ee9c8186ee4b28f9c57e9b345647)
    (cherry picked from commit fde20f941d0452d875ade3c86caf0a7ed34b789f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/865955
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/b17daedddfde90ea19ed71e46ca25b21503bb83d
Submitter: "Zuul (22348)"
Branch: stable/train

commit b17daedddfde90ea19ed71e46ca25b21503bb83d
Author: Luca Miccini <email address hidden>
Date: Mon Nov 28 19:38:15 2022 +0100

    Use --force when creating pacemaker resources

    pcs-0.10.14-6 introduced enforce validation at resources creation time.
    Unfortunately this doesn't work in our use case as puppet runs on the
    host but the necessary binaries are installed inside the containers.

    Let's try using --force to workaround.

    Conflicts:
            deployment/rabbitmq/rabbitmq-messaging-rpc-pacemaker-puppet.yaml

    (wallaby to train)
    Conflicts:
            deployment/database/redis-pacemaker-puppet.yaml
            deployment/rabbitmq/rabbitmq-messaging-rpc-pacemaker-puppet.yaml

    Closes-Bug: #1997939

    Change-Id: Id7616ebceb820d9799661c0fbc5f3f234f421ea3
    (cherry picked from commit 59300dfb30c5ee9c8186ee4b28f9c57e9b345647)
    (cherry picked from commit fde20f941d0452d875ade3c86caf0a7ed34b789f)
    (cherry picked from commit 3fc9478a863980e4448434f413ff3e878ccf04ab)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 18.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates train-eol release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ghanshyam <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/865880
Reason: TrieplO project is retiring now, for details, please see https://review.opendev.org/c/openstack/governance/+/905145 or reach out to OpenStack TC.