Container(s) with bad ExitCode: [''mysql_init_bundle''], Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'

[wes hayutin]

2020-06-17 15:08:20 |
2020-06-17 15:08:20 | TASK [tripleo_container_manage : Print failing containers] *********************
2020-06-17 15:08:20 | task path: /usr/share/ansible/roles/tripleo_container_manage/tasks/podman/check_exit_code.yml:48
2020-06-17 15:08:20 | Wednesday 17 June 2020 15:08:20 +0000 (0:00:00.090) 0:29:36.067 ********
2020-06-17 15:08:20 | fatal: [centos-8-inap-mtl01-0017196460]: FAILED! => changed=false
2020-06-17 15:08:20 | msg: 'Container(s) with bad ExitCode: [''mysql_init_bundle''], check logs in /var/log/containers/stdouts/'
2020-06-17 15:08:20 |

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_346/736018/2/check/tripleo-ci-centos-8-containers-multinode/346df41/logs/subnode-1/var/log/containers/stdouts/mysql_init_bundle.log

2020-06-17T15:08:15.633736812+00:00 stderr F Error: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20200617-9-uurldb create failed: Error: unable to set attribute galera-role
2020-06-17T15:08:15.639429783+00:00 stderr F Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Property[galera-role-centos-8-inap-mtl01-0017196460]/Pcmk_property[property-centos-8-inap-mtl01-0017196460-galera-role]/ensure: change from 'absent' to 'present' failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20200617-9-uurldb create failed: Error: unable to set attribute galera-role
2020-06-17T15:08:15.641165268+00:00 stdout F Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Bundle[galera-bundle]/Pcmk_bundle[galera-bundle]: Dependency Pcmk_property[property-centos-8-inap-mtl01-0017196460-galera-role] has failures: true
2020-06-17T15:08:15.641450846+00:00 stderr F Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Bundle[galera-bundle]/Pcmk_bundle[galera-bundle]: Skipping because of failed dependencies
2020-06-17T15:08:15.643298313+00:00 stderr F Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Ocf[galera]/Pcmk_resource[galera]: Skipping because of failed dependencies
2020-06-17T15:08:15.644273062+00:00 stderr F Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]: Skipping because of failed dependencies
2020-06-17T15:08:15.659310981+00:00 stderr F Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
2020-06-17T15:08:15.660043189+00:00 stdout F Info: Creating state file /var/lib/puppet/state/state.yaml
2020-06-17T15:08:15.667339198+00:00 stderr F Error: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_346/736018/2/check/tripleo-ci-centos-8-containers-multinode/346df41/logs/subnode-1/var/log/extra/podman/containers/mysql_bootstrap/stdout.log

Thanks for using MariaDB!
200617 15:03:09 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.

# job status... red
https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-containers-multinode&branch=stable%2Fussuri

200617 15:03:09 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
mysqld is alive

[wes hayutin (weshayutin)]

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22with%20bad%20ExitCode%3A%20%5B''mysql_init_bundle%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20voting%3A1

[wes hayutin (weshayutin)]

may be related to: https://bugs.launchpad.net/tripleo/+bug/1879292

[wes hayutin (weshayutin)]

I see:
Jun 17 15:01:37 centos-8-inap-mtl01-0017196460 puppet-user[63800]: Warning: Undefined variable '::deploy_config_name';

Jun 17 15:01:38 centos-8-inap-mtl01-0017196460 puppet-user[64211]: Error: Facter: error while resolving custom fact "rabbitmq_nodename": undefined method `[]' for nil:NilClass

In the journal...
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_346/736018/2/check/tripleo-ci-centos-8-containers-multinode/346df41/logs/subnode-1/var/log/extra/journal.txt