MySQL active status but in read-only mode

[Jadon Naas]

Cross-post of Issue in https://github.com/canonical/mysql-k8s-operator/issues/115. The issue in GitHub has more details and logs.

For a single-unit deployment of MySQL, the MySQL unit reports green active status but the MySQL server is in read-only mode when attempting writes to MySQL.

I deployed the MySQL using the mysql-k8s-operator as part of the deployment of a single-node OpenStack deployment on top of Microk8s. Juju reported the deployment as completing successfully, and I was able to complete some starting configuration tasks that would require writes to the MySQL database for the OpenStack cluster.

However, when I tried to interact with OpenStack maybe 30 minutes later I found that OpenStack was reporting the database was in read-only mode and throwing an exception. OpenStack was not functional because of this issue.

After some investigation, I found that the MySQL pod had been terminated with error 137 and recreated. I did not see any OOM kills on my host server, however.

My OpenStack Canonical team mates asked me to share my experience here alongside their own similar experiences with MySQL entering read-only mode after a pod restart.

I have included my logs and information from Juju and Kubernetes. I don't have the error logs from OpenStack anymore unfortunately.

[James Page (james-page)]

I think we've probably tracked down why all of the pods get restarted; microk8s monitors for changes in interface IP addressing - as part of the configure step we add an IP address to the external networking OVS bridge - this triggers the monitor code in microk8s which causes a re-generation of certificates and a full restart of K8S services.

[James Page (james-page)]

There is a secondary issue here that MySQL goes into read-only mode after the restart - tracked here:

https://github.com/canonical/mysql-k8s-operator/issues/115