[CI down] HA CI varying failures -- cannot connect to MariaDB

[Jiří Stránský]

From http://logs.openstack.org/93/208993/1/check-tripleo/gate-tripleo-ironic-overcloud-f21puppet-ha/93f0585/

Aug 06 10:55:17 overcloud-controller-0 os-collect-config[940]: [2015-08-06 10:55:17,140] (heat-config) [INFO] Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")
Aug 06 10:55:17 overcloud-controller-0 os-collect-config[940]: Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")
Aug 06 10:55:17 overcloud-controller-0 os-collect-config[940]: Error: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]
Aug 06 10:55:17 overcloud-controller-0 os-collect-config[940]: Error: /Stage[main]/Main/Exec[galera-ready]/returns: change from notrun to 0 failed: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]

[Jiří Stránský (jistr)]

In another run the forming the cluster seems to have succeeded, but it failed later when trying to create users for OpenStack services.

http://logs.openstack.org/93/208993/1/check-tripleo/gate-tripleo-ironic-overcloud-f21puppet-ha/f839e8c/

Aug 06 08:17:24 overcloud-controller-0 os-collect-config[941]: [2015-08-06 08:17:24,084] (heat-config) [INFO] Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")
Aug 06 08:17:24 overcloud-controller-0 os-collect-config[941]: Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")
Aug 06 08:17:24 overcloud-controller-0 os-collect-config[941]: Error: Execution of '/usr/bin/mysql -e CREATE USER 'neutron'@'192.0.2.8' IDENTIFIED BY PASSWORD '*2D8B749F0D875BBBF90D997FB98ED8F26D1107D7'' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")
Aug 06 08:17:24 overcloud-controller-0 os-collect-config[941]: Error: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_192.0.2.8]/Mysql_user[neutron@192.0.2.8]/ensure: change from absent to present failed: Execution of '/usr/bin/mysql -e CREATE USER 'neutron'@'192.0.2.8' IDENTIFIED BY PASSWORD '*2D8B749F0D875BBBF90D997FB98ED8F26D1107D7'' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")
Aug 06 08:17:24 overcloud-controller-0 os-collect-config[941]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_192.0.2.8]/Mysql_grant[neutron@192.0.2.8/ovs_neutron.*]: Skipping because of failed dependencies
Aug 06 08:17:24 overcloud-controller-0 os-collect-config[941]: Error: Execution of '/usr/bin/mysql -e CREATE USER 'heat'@'192.0.2.8' IDENTIFIED BY PASSWORD '*EA68BB50C10B0CF3CEBC4D1CB5D06032790FE07E'' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")
Aug 06 08:17:24 overcloud-controller-0 os-collect-config[941]: Error: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.0.2.8]/Mysql_user[heat@192.0.2.8]/ensure: change from absent to present failed: Execution of '/usr/bin/mysql -e CREATE USER 'heat'@'192.0.2.8' IDENTIFIED BY PASSWORD '*EA68BB50C10B0CF3CEBC4D1CB5D06032790FE07E'' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")
Aug 06 08:17:24 overcloud-controller-0 os-collect-config[941]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.0.2.8]/Mysql_grant...

[Yanis Guenane (yanis-guenane)]

The deployment itself was working fine on a local setup but kept failing on the CI.

This discarded an issue in the logic of the script/puppet.
After some investigation and some discussion/logs analysis in #clusterlabs, it came out that it was apparently a CPU issue (not enough CPU) causing corosync/totem to behave in a weird way.

Would this be possible to update the CI to use NODE_CPU=2 at least and either confirm or discard this hypothesis ?

[Jiří Stránský (jistr)]

This is no longer a CI problem. It's possible we'll hit it again in the future when corosync version gets updated (if it's still so CPU hungry by then). Setting prio to 'low' for now.

[Jiří Stránský (jistr)]

This hasn't occurred for a while now.