Comment 14 for bug 1712087

Dmitry,

Although this fails...

2020-10-20 08:11:41,385 p=151140 u=lineng n=ansible | TASK [mariadb : Check MariaDB service port liveness] ***************************
2020-10-20 08:11:51,989 p=151140 u=lineng n=ansible | fatal: [osce3]: FAILED! => {"changed": false, "elapsed": 10, "msg": "Timeout when waiting for search string MariaDB in 192.168.1
.6:3306"}
2020-10-20 08:11:51,989 p=151140 u=lineng n=ansible | ...ignoring
2020-10-20 08:11:52,145 p=151140 u=lineng n=ansible | fatal: [osce4]: FAILED! => {"changed": false, "elapsed": 10, "msg": "Timeout when waiting for search string MariaDB in 192.168.1
.7:3306"}
2020-10-20 08:11:52,146 p=151140 u=lineng n=ansible | ...ignoring

... the failure is ignored.

2020-10-20 08:11:52,225 p=151140 u=lineng n=ansible | TASK [mariadb : Divide hosts by their MariaDB service port liveness] ***********
2020-10-20 08:11:52,319 p=151140 u=lineng n=ansible | changed: [osce3]
2020-10-20 08:11:52,344 p=151140 u=lineng n=ansible | changed: [osce4]

Here is where it actually fails:

2020-10-20 08:11:52,424 p=151140 u=lineng n=ansible | TASK [mariadb : Fail on existing but stopped cluster] **************************
2020-10-20 08:11:52,517 p=151140 u=lineng n=ansible | fatal: [osce3]: FAILED! => {"changed": false, "msg": "MariaDB cluster exists but is stopped. Please start it using kolla-ansible
 mariadb_recovery"}
2020-10-20 08:11:52,551 p=151140 u=lineng n=ansible | fatal: [osce4]: FAILED! => {"changed": false, "msg": "MariaDB cluster exists but is stopped. Please start it using kolla-ansible
 mariadb_recovery"}

This check fails when at least one node has a mariadb docker volume, but no node has a mariadb container running.

If you are using the kolla-ansible destroy command to clean up, there is an issue where it only removes volumes that are mounted by existing containers. If you have a stale mariadb volume, this could be causing the issue. Try removing it.