Comment 5 for bug 1816842

Looking at the logs, we see the mariadb-server-1 logs ending here:

{"log":"2019-02-20 01:50:51,545 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\n","stream":"stderr","time":"2019-02-20T01:50:51.54616385Z"}
{"log":"2019-02-20 01:51:01,531 - OpenStack-Helm Mariadb - INFO - Cluster info has been uptodate 0 times out of the required 12\n","stream":"stderr","time":"2019-02-20T01:51:01.531861929Z"}
{"log":"2019-02-20 01:51:01,531 - OpenStack-Helm Mariadb - INFO - Checking to see if cluster data is fresh\n","stream":"stderr","time":"2019-02-20T01:51:01.531891753Z"}
{"log":"2019-02-20 01:51:01,568 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\n","stream":"stderr","time":"2019-02-20T01:51:01.568883913Z"}

The mariadb-server-0 logs show another story:

{"log":"2019-02-20 01:50:58,967 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is ok for node mariadb-server-0\n","stream":"stderr","time":"2019-02-20T01:50:58.967962473Z"}
{"log":"2019-02-20 01:51:08,978 - OpenStack-Helm Mariadb - INFO - Cluster info has been uptodate 1 times out of the required 12\n","stream":"stderr","time":"2019-02-20T01:51:08.978298167Z"}
{"log":"2019-02-20 01:51:08,978 - OpenStack-Helm Mariadb - INFO - Checking to see if cluster data is fresh\n","stream":"stderr","time":"2019-02-20T01:51:08.97833839Z"}
{"log":"2019-02-20 01:51:08,979 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\n","stream":"stderr","time":"2019-02-20T01:51:08.979132811Z"}
{"log":"2019-02-20 01:51:21,870 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /api/v1/namespaces/openst
ack/configmaps/osh-openstack-mariadb-mariadb-state\n","stream":"stderr","time":"2019-02-20T01:51:21.870759801Z"}
{"log":"2019-02-20 01:51:21,870 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /api/v1/namespaces/openst
ack/configmaps/osh-openstack-mariadb-mariadb-state\n","stream":"stderr","time":"2019-02-20T01:51:21.871024039Z"}
{"log":"2019-02-20 01:51:21,882 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-server-1\n","stream":"stderr","time":"2019-02-20T01:51:21.883047989Z"}
{"log":"2019-02-20 01:51:21,883 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-server-0\n","stream":"stderr","time":"2019-02-20T01:51:21.88352455Z"}
{"log":"2019-02-20 01:51:31,893 - OpenStack-Helm Mariadb - INFO - Cluster info has been uptodate 0 times out of the required 12\n","stream":"stderr","time":"2019-02-20T01:51:31.893865638Z"}
{"log":"2019-02-20 01:51:31,893 - OpenStack-Helm Mariadb - INFO - Checking to see if cluster data is fresh\n","stream":"stderr","time":"2019-02-20T01:51:31.893908681Z"}
{"log":"2019-02-20 01:51:31,898 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-server-1\n","stream":"stderr","time":"2019-02-20T01:51:31.898558239Z"}
{"log":"2019-02-20 01:51:31,898 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-server-0\n","stream":"stderr","time":"2019-02-20T01:51:31.898631348Z"}
{"log":"2019-02-20 01:51:31,909 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\n","stream":"stderr","time":"2019-02-20T01:51:31.909790831Z"}
{"log":"2019-02-20 01:51:41,908 - OpenStack-Helm Mariadb - INFO - Cluster info has been uptodate 0 times out of the required 12\n","stream":"stderr","time":"2019-02-20T01:51:41.908430614Z"}
{"log":"2019-02-20 01:51:41,908 - OpenStack-Helm Mariadb - INFO - Checking to see if cluster data is fresh\n","stream":"stderr","time":"2019-02-20T01:51:41.908466489Z"}
{"log":"2019-02-20 01:51:41,912 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-server-1\n","stream":"stderr","time":"2019-02-20T01:51:41.912964648Z"}
{"log":"2019-02-20 01:51:41,912 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is ok for node mariadb-server-0\n","stream":"stderr","time":"2019-02-20T01:51:41.913080719Z"}
{"log":"2019-02-20 01:51:41,931 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\n","stream":"stderr","time":"2019-02-20T01:51:41.931925188Z"}

We see a broken connection at 2019-02-20 01:51:21, after which only mariadb-server-0 is updating its state and mariadb-server-1 appears hung.