Stale PID file after reboot preventing the service from starting

[Przemyslaw Lal]

After an abrupt reboot of an Openstack control node due to a hardware problem, all mysql-router units running on that node enter blocked state. mysql-router service couldn't start on them due to stale PID files, for example:

Mar 04 03:09:31 juju-733f2f-7-lxd-13 start.sh[723]: Error: PID file /var/lib/mysql/octavia-mysql-router/mysqlrouter.pid found. Already running?
Mar 04 03:09:31 juju-733f2f-7-lxd-13 sudo[692]: pam_unix(sudo:session): session closed for user root

Openstack API services were affected and couldn't re-connect to MySQL because of that, rendering 1/3 of the control-plane unhealthy.

The workaround was to manually remove stale PID files, restart *-mysql-router services and (optionally) restart affected Openstack services.

Versions:
focal-ussuri
MySQL router charm revision 6 from the stable channel

ii libmysqlclient-dev 8.0.28-0ubuntu0.20.04.3 amd64 MySQL database development files
ii libmysqlclient21:amd64 8.0.28-0ubuntu0.20.04.3 amd64 MySQL database client library
ii mysql-common 5.8+1.0.5ubuntu2 all MySQL database common files, e.g. /etc/mysql/my.cnf
ii mysql-router 8.0.28-0ubuntu0.20.04.3 amd64 route connections from MySQL clients to MySQL servers
ii python3-pymysql 0.9.3-2ubuntu3 all Pure-Python MySQL Driver - Python 3.x

Juju 2.9.46

[Alex Kavanagh (ajkavanagh)]

Triaged as medium as there is a workaround (although it's manual).