Comment 5 for bug 1712087

Revision history for this message
Sean Murphy (murp) wrote :

This is our process (which is basically what the kolla bootstrap mariadb function does):

- Check the following file on every controller node

root@controller-1:/var/lib/docker/volumes/mariadb/_data# cat grastate.dat && echo
# GALERA saved state
version: 2.1
uuid: fbf02533-f205-11e7-ad67-ee3d1299a58d
seqno: -1
safe_to_bootstrap: 0

- Choose the controller with the highest seqno value.
- Change save_to_bootstrap: 0 to save_to_bootstrap: 1 on this node
- On this node, run

   docker run --net host --name mariadbbootstrap -v /etc/localtime:/etc/localtime:ro -v kolla_logs:/var/log/kolla/ -v mariadb:/var/lib/mysql -v /etc/kolla/mariadb/:/var/lib/kolla/config_files/:ro --restart on-failure:10 --env KOLLA_CONFIG_STRATEGY=COPY_ALWAYS --env BOOTSTRAP_ARGS='--wsrep-new-cluster' 172.28.0.1:5000/kolla/ubuntu-binary-mariadb:ocata

(You will have to modify the IP add and perhaps the label of the container depending on
your situation). Wait until it looks like the container come up in a healthy state - this
can take about a minute usually.

- Restart the MariaDB Docker containers on the two other controller nodes - do it one by one. They should come up now. Look at the logs to see if they sync with the other mariadb instances in the cluster.
- Stop and remove the bootstrap container -> docker rm -f mariadbbootstrap
- Restart mariadb container on the node on which the rescue container was run. -> docker restart mariadb.