Looking at the logs, we see the mariadb-server-1 logs ending here:
{"log":"2019-02-20 01:50:51,545 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\n","stream":"stderr","time":"2019-02-20T01:50:51.54616385Z"}
{"log":"2019-02-20 01:51:01,531 - OpenStack-Helm Mariadb - INFO - Cluster info has been uptodate 0 times out of the required 12\n","stream":"stderr","time":"2019-02-20T01:51:01.531861929Z"}
{"log":"2019-02-20 01:51:01,531 - OpenStack-Helm Mariadb - INFO - Checking to see if cluster data is fresh\n","stream":"stderr","time":"2019-02-20T01:51:01.531891753Z"}
{"log":"2019-02-20 01:51:01,568 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\n","stream":"stderr","time":"2019-02-20T01:51:01.568883913Z"}
The mariadb-server-0 logs show another story:
{"log":"2019-02-20 01:50:58,967 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is ok for node mariadb-server-0\n","stream":"stderr","time":"2019-02-20T01:50:58.967962473Z"}
{"log":"2019-02-20 01:51:08,978 - OpenStack-Helm Mariadb - INFO - Cluster info has been uptodate 1 times out of the required 12\n","stream":"stderr","time":"2019-02-20T01:51:08.978298167Z"}
{"log":"2019-02-20 01:51:08,978 - OpenStack-Helm Mariadb - INFO - Checking to see if cluster data is fresh\n","stream":"stderr","time":"2019-02-20T01:51:08.97833839Z"}
{"log":"2019-02-20 01:51:08,979 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\n","stream":"stderr","time":"2019-02-20T01:51:08.979132811Z"}
{"log":"2019-02-20 01:51:21,870 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /api/v1/namespaces/openst
ack/configmaps/osh-openstack-mariadb-mariadb-state\n","stream":"stderr","time":"2019-02-20T01:51:21.870759801Z"}
{"log":"2019-02-20 01:51:21,870 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /api/v1/namespaces/openst
ack/configmaps/osh-openstack-mariadb-mariadb-state\n","stream":"stderr","time":"2019-02-20T01:51:21.871024039Z"}
{"log":"2019-02-20 01:51:21,882 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-server-1\n","stream":"stderr","time":"2019-02-20T01:51:21.883047989Z"}
{"log":"2019-02-20 01:51:21,883 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-server-0\n","stream":"stderr","time":"2019-02-20T01:51:21.88352455Z"}
{"log":"2019-02-20 01:51:31,893 - OpenStack-Helm Mariadb - INFO - Cluster info has been uptodate 0 times out of the required 12\n","stream":"stderr","time":"2019-02-20T01:51:31.893865638Z"}
{"log":"2019-02-20 01:51:31,893 - OpenStack-Helm Mariadb - INFO - Checking to see if cluster data is fresh\n","stream":"stderr","time":"2019-02-20T01:51:31.893908681Z"}
{"log":"2019-02-20 01:51:31,898 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-server-1\n","stream":"stderr","time":"2019-02-20T01:51:31.898558239Z"}
{"log":"2019-02-20 01:51:31,898 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-server-0\n","stream":"stderr","time":"2019-02-20T01:51:31.898631348Z"}
{"log":"2019-02-20 01:51:31,909 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\n","stream":"stderr","time":"2019-02-20T01:51:31.909790831Z"}
{"log":"2019-02-20 01:51:41,908 - OpenStack-Helm Mariadb - INFO - Cluster info has been uptodate 0 times out of the required 12\n","stream":"stderr","time":"2019-02-20T01:51:41.908430614Z"}
{"log":"2019-02-20 01:51:41,908 - OpenStack-Helm Mariadb - INFO - Checking to see if cluster data is fresh\n","stream":"stderr","time":"2019-02-20T01:51:41.908466489Z"}
{"log":"2019-02-20 01:51:41,912 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-server-1\n","stream":"stderr","time":"2019-02-20T01:51:41.912964648Z"}
{"log":"2019-02-20 01:51:41,912 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is ok for node mariadb-server-0\n","stream":"stderr","time":"2019-02-20T01:51:41.913080719Z"}
{"log":"2019-02-20 01:51:41,931 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\n","stream":"stderr","time":"2019-02-20T01:51:41.931925188Z"}
We see a broken connection at 2019-02-20 01:51:21, after which only mariadb-server-0 is updating its state and mariadb-server-1 appears hung.
Looking at the logs, we see the mariadb-server-1 logs ending here:
{"log":"2019-02-20 01:50:51,545 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\ n","stream" :"stderr" ,"time" :"2019- 02-20T01: 50:51.54616385Z "} stream" :"stderr" ,"time" :"2019- 02-20T01: 51:01.531861929 Z"} ,"stream" :"stderr" ,"time" :"2019- 02-20T01: 51:01.531891753 Z"} n","stream" :"stderr" ,"time" :"2019- 02-20T01: 51:01.568883913 Z"}
{"log":"2019-02-20 01:51:01,531 - OpenStack-Helm Mariadb - INFO - Cluster info has been uptodate 0 times out of the required 12\n","
{"log":"2019-02-20 01:51:01,531 - OpenStack-Helm Mariadb - INFO - Checking to see if cluster data is fresh\n"
{"log":"2019-02-20 01:51:01,568 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\
The mariadb-server-0 logs show another story:
{"log":"2019-02-20 01:50:58,967 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is ok for node mariadb- server- 0\n","stream" :"stderr" ,"time" :"2019- 02-20T01: 50:58.967962473 Z"} stream" :"stderr" ,"time" :"2019- 02-20T01: 51:08.978298167 Z"} ,"stream" :"stderr" ,"time" :"2019- 02-20T01: 51:08.97833839Z "} n","stream" :"stderr" ,"time" :"2019- 02-20T01: 51:08.979132811 Z"} 'Connection aborted.', error(104, 'Connection reset by peer'))': /api/v1/ namespaces/ openst osh-openstack- mariadb- mariadb- state\n" ,"stream" :"stderr" ,"time" :"2019- 02-20T01: 51:21.870759801 Z"} 'Connection aborted.', error(104, 'Connection reset by peer'))': /api/v1/ namespaces/ openst osh-openstack- mariadb- mariadb- state\n" ,"stream" :"stderr" ,"time" :"2019- 02-20T01: 51:21.871024039 Z"} server- 1\n","stream" :"stderr" ,"time" :"2019- 02-20T01: 51:21.883047989 Z"} server- 0\n","stream" :"stderr" ,"time" :"2019- 02-20T01: 51:21.88352455Z "} stream" :"stderr" ,"time" :"2019- 02-20T01: 51:31.893865638 Z"} ,"stream" :"stderr" ,"time" :"2019- 02-20T01: 51:31.893908681 Z"} server- 1\n","stream" :"stderr" ,"time" :"2019- 02-20T01: 51:31.898558239 Z"} server- 0\n","stream" :"stderr" ,"time" :"2019- 02-20T01: 51:31.898631348 Z"} n","stream" :"stderr" ,"time" :"2019- 02-20T01: 51:31.909790831 Z"} stream" :"stderr" ,"time" :"2019- 02-20T01: 51:41.908430614 Z"} ,"stream" :"stderr" ,"time" :"2019- 02-20T01: 51:41.908466489 Z"} server- 1\n","stream" :"stderr" ,"time" :"2019- 02-20T01: 51:41.912964648 Z"} server- 0\n","stream" :"stderr" ,"time" :"2019- 02-20T01: 51:41.913080719 Z"} n","stream" :"stderr" ,"time" :"2019- 02-20T01: 51:41.931925188 Z"}
{"log":"2019-02-20 01:51:08,978 - OpenStack-Helm Mariadb - INFO - Cluster info has been uptodate 1 times out of the required 12\n","
{"log":"2019-02-20 01:51:08,978 - OpenStack-Helm Mariadb - INFO - Checking to see if cluster data is fresh\n"
{"log":"2019-02-20 01:51:08,979 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\
{"log":"2019-02-20 01:51:21,870 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError(
ack/configmaps/
{"log":"2019-02-20 01:51:21,870 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError(
ack/configmaps/
{"log":"2019-02-20 01:51:21,882 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-
{"log":"2019-02-20 01:51:21,883 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-
{"log":"2019-02-20 01:51:31,893 - OpenStack-Helm Mariadb - INFO - Cluster info has been uptodate 0 times out of the required 12\n","
{"log":"2019-02-20 01:51:31,893 - OpenStack-Helm Mariadb - INFO - Checking to see if cluster data is fresh\n"
{"log":"2019-02-20 01:51:31,898 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-
{"log":"2019-02-20 01:51:31,898 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-
{"log":"2019-02-20 01:51:31,909 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\
{"log":"2019-02-20 01:51:41,908 - OpenStack-Helm Mariadb - INFO - Cluster info has been uptodate 0 times out of the required 12\n","
{"log":"2019-02-20 01:51:41,908 - OpenStack-Helm Mariadb - INFO - Checking to see if cluster data is fresh\n"
{"log":"2019-02-20 01:51:41,912 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is too old to make a decision for node mariadb-
{"log":"2019-02-20 01:51:41,912 - OpenStack-Helm Mariadb - INFO - The data we have from the cluster is ok for node mariadb-
{"log":"2019-02-20 01:51:41,931 - OpenStack-Helm Mariadb - INFO - Updating grastate configmap\
We see a broken connection at 2019-02-20 01:51:21, after which only mariadb-server-0 is updating its state and mariadb-server-1 appears hung.