tripleo fails to deploy in ci : Failed to call refresh: /usr/bin/clustercheck

[Joe Talerico]

The patch I had didn't touch clustercheck, but I see : http://logs.openstack.org/24/497524/6/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/0b63e4b/logs/undercloud/home/jenkins/failed_deployment_list.log.txt.gz

Capturing the specific error output:

            "Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql/Exec[galera-ready]: Failed to call refresh: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]",
            "Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql/Exec[galera-ready]: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]",
            "Error: Failed to apply catalog: Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 \"No such file or directory\")",

This seems to be happening frequently enough to track :
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22Failed%20to%20call%20refresh%3A%20%2Fusr%2Fbin%2Fclustercheck%5C%22

[Bogdan Dobrelya (bogdando)]

there are multiple hits reported (14 for multiple tripleo CI gates), this should be a high bug to get fixed in the pike scope

[Michele Baldessari (michele)]

Meh I think we lost the CIB collection capability in the CI when we moved to oooq? :/

[Michele Baldessari (michele)]

So the issue seems that the stonith-enabled=false is never set and so pacemaker does not start the db and hence clustercheck fails.

On my working deployment I can see the following:
Aug 29 13:42:07 [19665] overcloud-controller-0 cib: info: cib_perform_op: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']: <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/>

I can't see this here, so maybe we have some dependency issues here. What is a bit odd is that in the pacemaker tripleo profile we have the following (at step1):
    class { '::pacemaker':
      hacluster_pwd => hiera('hacluster_pwd'),
    }
    -> class { '::pacemaker::corosync':
      cluster_members => $pacemaker_cluster_members,
      setup_cluster => $pacemaker_master,
      cluster_setup_extras => $cluster_setup_extras,
      remote_authkey => $remote_authkey,
    }
    if $pacemaker_master {
      class { '::pacemaker::stonith':
        disable => !$enable_fencing,
        tries => $pcs_tries,
      }
    }

The creation of the galera resource happens at step 2, so we should be guaranteed to have stonith property set to false.

[Michele Baldessari (michele)]

Oh I see the problem:
Aug 25 15:44:33 localhost os-collect-config: "Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install fence-agents-all' returned 1: Error downloading packages:",
Aug 25 15:44:33 localhost os-collect-config: "Error: /Stage[main]/Pacemaker::Install/Package[fence-agents-all]/ensure: change from purged to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install fence-agents-all' returned 1: Error downloading packages:",
Aug 25 15:44:33 localhost os-collect-config: "Notice: /Stage[main]/Pacemaker::Service/Service[pcsd]: Dependency Package[fence-agents-all] has failures: true",
Aug 25 15:44:33 localhost os-collect-config: "Notice: /Stage[main]/Pacemaker::Corosync/User[hacluster]: Dependency Package[fence-agents-all] has failures: true",
Aug 25 15:44:33 localhost os-collect-config: "Notice: /Stage[main]/Pacemaker::Corosync/Exec[reauthenticate-across-all-nodes]: Dependency Package[fence-agents-all] has failures: true",
Aug 25 15:44:33 localhost os-collect-config: "Notice: /Stage[main]/Pacemaker::Corosync/Exec[auth-successful-across-all-nodes]: Dependency Package[fence-agents-all] has failures: true",
Aug 25 15:44:33 localhost os-collect-config: "Notice: /Stage[main]/Pacemaker::Corosync/File[etc-pacemaker]: Dependency Package[fence-agents-all] has failures: true",
Aug 25 15:44:33 localhost os-collect-config: "Notice: /Stage[main]/Pacemaker::Corosync/File[etc-pacemaker-authkey]: Dependency Package[fence-agents-all] has failures: true",
Aug 25 15:44:33 localhost os-collect-config: "Notice: /Stage[main]/Pacemaker::Corosync/Exec[Create Cluster tripleo_cluster]: Dependency Package[fence-agents-all] has failures: true",
Aug 25 15:44:33 localhost os-collect-config: "Notice: /Stage[main]/Pacemaker::Corosync/Exec[Start Cluster tripleo_cluster]: Dependency Package[fence-agents-all] has failures: true",
Aug 25 15:44:33 localhost os-collect-config: "Notice: /Stage[main]/Pacemaker::Service/Service[corosync]: Dependency Package[fence-agents-all] has failures: true",
Aug 25 15:44:33 localhost os-collect-config: "Notice: /Stage[main]/Pacemaker::Service/Service[pacemaker]: Dependency Package[fence-agents-all] has failures: true",
Aug 25 15:44:33 localhost os-collect-config: "Notice: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]: Dependency Package[fence-agents-all] has failures: true",
Aug 25 15:44:33 localhost os-collect-config: "Notice: /Stage[main]/Pacemaker::Stonith/Pacemaker::Property[Disable STONITH]/Pcmk_property[property--stonith-enabled]: Dependency Package[fence-agents-all] has failures: true",

Maybe since this is a multinode we should just preinstall fence-agents-all on the nodes in order to avoid these failures?

[Michele Baldessari (michele)]

Added elastic-recheck query here: https://review.openstack.org/499516

[Michele Baldessari (michele)]

This is the full query to match clustercheck failures + fence-agents-all issue:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22Failed%20to%20call%20refresh%3A%20%2Fusr%2Fbin%2Fclustercheck%5C%22%20and%20message%3A%20%5C%22Error%3A%20Execution%20of%20'%2Fusr%2Fbin%2Fyum%20-d%200%20-e%200%20-y%20install%20fence-agents-all'%20returned%201%3A%20Error%20downloading%20packages%5C%22