unexpected pod restart during high memory pressure

[James Page]

Configured machine only has 10GB of RAM; after cluster bootstrap memory utilisation was at 85%; configure command executed during which (after image was uploaded) all of the pods in the microk8s deployment restarted.

A few issues

a) why did the pods get restarted
b) mysql did restart and recovered but no mysql-router instances where able to reconnect to the backend database:

2023-06-06 10:23:45 metadata_cache INFO [7feee411b640] Starting metadata cache refresh thread
2023-06-06 10:23:45 metadata_cache WARNING [7feee411b640] Failed connecting with Metadata Server mysql-0.mysql-endpoints.openstack.svc.cluster.local:3306: Access denied for user 'mysql_router1_159cr99xjlpp'@'keystone-mysql-router-0.keystone-mysql-router-endpoints.openstac' (using password: YES) (1045)
2023-06-06 10:23:45 metadata_cache ERROR [7feee411b640] Failed fetching metadata from any of the 1 metadata servers.
2023-06-06 10:23:45 routing INFO [7feeb1ffb640] [routing:bootstrap_ro] started: routing strategy = round-robin-with-fallback

[James Page (james-page)]

Just before the mass restart I also see this in the syslog:

Jun 6 09:01:52 microstack-minimum microk8s.daemon-containerd[24327]: time="2023-06-06T10:01:51.359460913+01:00" level=info msg="shim disconnected" id=75949a72c7ef2b38d129ab13bbf644c4a0d57221d9ec0e69a4bd9260e3e1404d
Jun 6 09:01:52 microstack-minimum microk8s.daemon-containerd[24327]: time="2023-06-06T10:01:52.204910617+01:00" level=warning msg="cleaning up after shim disconnected" id=75949a72c7ef2b38d129ab13bbf644c4a0d57221d9ec0e69a4bd9260e3e1404d namespace=k8s.io
Jun 6 09:01:52 microstack-minimum microk8s.daemon-containerd[24327]: time="2023-06-06T10:01:52.204959778+01:00" level=info msg="cleaning up dead shim"
Jun 6 09:01:52 microstack-minimum microk8s.daemon-containerd[24327]: time="2023-06-06T10:01:52.921974888+01:00" level=warning msg="cleanup warnings time=\"2023-06-06T10:01:52+01:00\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=471036 runtime=io.containerd.runc.v2\n"