mariadb can't running

[zhangss]

I deleted MariaDB due to misoperation, and cannot recover MariaDB cluster with "kolla ansible mariadb_recovery"

TASK [mariadb : Shutdown MariaDB] **************************************************************************************************************
skipping: [openstack01]
fatal: [openstack02]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}
fatal: [openstack03]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}

TASK [mariadb : Copying MariaDB log file to /tmp] **********************************************************************************************
fatal: [openstack01]: FAILED! => {"changed": true, "cmd": "docker cp mariadb:/var/log/kolla/mariadb/mariadb.log /tmp/mariadb_tmp.log", "delta": "0:00:00.053123", "end": "2020-03-21 00:32:59.714083", "msg": "non-zero return code", "rc": 1, "start": "2020-03-21 00:32:59.660960", "stderr": "Error: No such container:path: mariadb:/var/log/kolla/mariadb/mariadb.log", "stderr_lines": ["Error: No such container:path: mariadb:/var/log/kolla/mariadb/mariadb.log"], "stdout": "", "stdout_lines": []}

[Radosław Piliszek (yoctozepto)]

Needs more info: release, what was deleted (container, volume, which hosts of these?), output of docker regarding containers and volumes on the affected hosts, inventory file, globals.yml

[zhangss (intentc)]

Only delete container,volume(/var/lib/docker) and host are present。
multinode and globals.yml are the default configuration.

[zhangss (intentc)]

when i execute command "docker ps" "ls /var/lib/docker/volumes/_data" on three controller nodes.
controller01:
docker ps -a | grep mariadb
####nothing####

ls /var/lib/docker/volumes/mariadb/_data/
aria_log.00000001 ib_buffer_pool multi-master.info mysql-bin.000016 nova_cell0/
aria_log_control ibdata1 mysql/ mysql-bin.000017 performance_schema/
.bash_history ib_logfile0 mysql-bin.000010 mysql-bin.index placement/
cinder/ ib_logfile1 mysql-bin.000011 mysql-bin.state trove/
galera.cache innobackup.backup.log mysql-bin.000012 mysql.sock xtrabackup_galera_info
glance/ ironic/ mysql-bin.000013 neutron/
grastate.dat ironic_inspector/ mysql-bin.000014 nova/
heat/ keystone/ mysql-bin.000015 nova_api/

controller02
docker ps | grep mariadb
1f02ea8856c9 10.166.16.96:4567/kolla/centos-binary-mariadb:stein "dumb-init -- kolla_…" 8 days ago Up 8 days mariadb
ls /var/lib/docker/volumes/mariadb/_data/
aria_log.00000001 ib_buffer_pool multi-master.info mysql-bin.000016 nova_cell0/
aria_log_control ibdata1 mysql/ mysql-bin.000017 performance_schema/
.bash_history ib_logfile0 mysql-bin.000010 mysql-bin.index placement/
cinder/ ib_logfile1 mysql-bin.000011 mysql-bin.state trove/
galera.cache innobackup.backup.log mysql-bin.000012 mysql.sock xtrabackup_galera_info
glance/ ironic/ mysql-bin.000013 neutron/
grastate.dat ironic_inspector/ mysql-bin.000014 nova/
heat/ keystone/ mysql-bin.000015 nova_api/

Controller03 is the same as controller02

How should I fix the MariaDB cluster？

[zhangss (intentc)]

when i execute "kolla-ansible reconfigure" and "kolla-ansible deploy"

RUNNING HANDLER [mariadb : remove restart policy from master mariadb] **************************************************************************
fatal: [openstack01]: FAILED! => {"changed": true, "cmd": ["docker", "update", "--restart", "no", "mariadb"], "delta": "0:00:00.053263", "end": "2020-03-22 12:31:18.368776", "msg": "non-zero return code", "rc": 1, "start": "2020-03-22 12:31:18.315513", "stderr": "Error response from daemon: No such container: mariadb", "stderr_lines": ["Error response from daemon: No such container: mariadb"], "stdout": "", "stdout_lines": []}

[Radosław Piliszek (yoctozepto)]

As a fix you could try this: https://review.opendev.org/706078

as a workaround you can do the following steps:
1) comment out openstack01 from inventory
2) run mariadb_recovery
3) readd openstack01 to inventory but make sure it is listed LAST
4) run deploy

[zhangss (intentc)]

I have one question, Dose kolla-ansible has no choreography function, Like k8s, if a container is deleted, it can be recovered automatically. Or how can I guarantee the integrity of recovery as much as possible when any high availability component container is deleted.

[Radosław Piliszek (yoctozepto)]

Kolla Ansible depends on classic reliability. MariaDB does not do any better in k8s due to its design.