aftre mysql inodb recover all mysql--routers are in error state

[vinaya]

example keystone mysql router

juju status keystone
Model Controller Cloud/Region Version SLA Timestamp
openstack maas-controller mymaas/default 2.8.7 unsupported 08:51:56Z

App Version Status Scale Charm Store Channel Rev OS Message
keystone 17.0.0 active 1 keystone charmstore 319 ubuntu Application Ready
keystone-ldap 17.0.0 active 1 keystone-ldap charmstore 31 ubuntu Unit is ready
keystone-mysql-router 8.0.26 error 1 mysql-router charmstore 3 ubuntu hook failed: "start"

Unit Workload Agent Machine Public address Ports Message
keystone/0* active idle 0/lxd/2 10.222.88.115 5000/tcp PO: Unit is ready
  keystone-ldap/0* active idle 10.222.88.115 Unit is ready
  keystone-mysql-router/0* error idle 10.222.88.115 hook failed: "start"

1. this exactly happend after iinodb-cluster recovery with below command.
juju run-action mysql-innodb-cluster/leader reboot-cluster-from-complete-outage --wait

1.tries juju actions for stop/restart on mysql-router still same and restart of related services like jujud-unit-keystone-mysql and keystone-mysql-router service..

debug log is saying
 WARNING unit.keystone-mysql-router/0.start m_helper.connect(self.db_router_user,
unit-keystone-mysql-router-0: 09:07:23 WARNING unit.keystone-mysql-router/0.start File "/var/lib/juju/agents/unit-keystone-mysql-router-0/.venv/lib/python3.8/site-packages/charmhelpers/contrib/database/mysql.py", line 100, in connect
unit-keystone-mysql-router-0: 09:07:23 WARNING unit.keystone-mysql-router/0.start self.connection = MySQLdb.connect(**_connection_info)
unit-keystone-mysql-router-0: 09:07:23 WARNING unit.keystone-mysql-router/0.start File "/var/lib/juju/agents/unit-keystone-mysql-router-0/.venv/lib/python3.8/site-packages/MySQLdb/__init__.py", line 130, in Connect
unit-keystone-mysql-router-0: 09:07:23 WARNING unit.keystone-mysql-router/0.start return Connection(*args, **kwargs)
unit-keystone-mysql-router-0: 09:07:23 WARNING unit.keystone-mysql-router/0.start File "/var/lib/juju/agents/unit-keystone-mysql-router-0/.venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 185, in __init__
unit-keystone-mysql-router-0: 09:07:23 WARNING unit.keystone-mysql-router/0.start super().__init__(*args, **kwargs2)
unit-keystone-mysql-router-0: 09:07:23 WARNING unit.keystone-mysql-router/0.start TypeError: connect() argument 3 must be str, not None
unit-keystone-mysql-router-0: 09:07:23 ERROR juju.worker.uniter.operation hook "start" (via explicit, bespoke hook script) failed: exit status 1
unit-keystone-mysql-router-0: 09:07:23 INFO juju.worker.uniter awaiting error resolution for "start" hook

based on the log it may not receiving the IP of new iinodb clster,,its getting None all the time .

[vinaya (agrahar)]

HI Team,
services are running and i tried restart of service and reboot ofcontianer running keystone but its still same with above error.

root@juju-e1577b-0-lxd-2:~# systemctl status keystone-mysql-router.service
● keystone-mysql-router.service - MySQL Router
     Loaded: loaded (/etc/systemd/system/keystone-mysql-router.service; enabled; vendor preset: enabled)
     Active: active (exited) since Wed 2021-09-29 08:45:53 UTC; 28min ago
    Process: 19218 ExecStart=/var/lib/mysql/keystone-mysql-router/start.sh (code=exited, status=0/SUCCESS)
   Main PID: 19218 (code=exited, status=0/SUCCESS)
      Tasks: 88 (limit: 314572)
     Memory: 37.0M
     CGroup: /system.slice/keystone-mysql-router.service
             ├─19220 sudo ROUTER_PID=/var/lib/mysql/keystone-mysql-router/mysqlrouter.pid /usr/bin/mysqlrouter -c /var/lib/mysql/keystone-mysql-router/mysqlrouter.conf --user=mysql
             └─19221 /usr/bin/mysqlrouter -c /var/lib/mysql/keystone-mysql-router/mysqlrouter.conf --user=mysql

Sep 29 08:45:53 juju-e1577b-0-lxd-2 systemd[1]: Starting MySQL Router...
Sep 29 08:45:53 juju-e1577b-0-lxd-2 systemd[1]: Started MySQL Router.
Sep 29 08:45:53 juju-e1577b-0-lxd-2 sudo[19220]: root : TTY=unknown ; PWD=/ ; USER=root ; ENV=ROUTER_PID=/var/lib/mysql/keystone-mysql-router/mysqlrouter.pid ; COMMAND=/usr/bin/mys>
Sep 29 08:45:53 juju-e1577b-0-lxd-2 sudo[19220]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 29 08:45:53 juju-e1577b-0-lxd-2 start.sh[19221]: PID 19221 written to '/var/lib/mysql/keystone-mysql-router/mysqlrouter.pid'
Sep 29 08:45:53 juju-e1577b-0-lxd-2 start.sh[19221]: logging facility initialized, switching logging to loggers specified in configuration
root@juju-e1577b-0-lxd-2:~# systemctl status jujud-unit-keystone-mysql-router-0.service
● jujud-unit-keystone-mysql-router-0.service - juju unit agent for keystone-mysql-router/0
     Loaded: loaded (/etc/systemd/system/jujud-unit-keystone-mysql-router-0.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-09-29 08:46:48 UTC; 28min ago
   Main PID: 19332 (bash)
      Tasks: 37 (limit: 314572)
     Memory: 59.1M
     CGroup: /system.slice/jujud-unit-keystone-mysql-router-0.service
             ├─19332 bash /etc/systemd/system/jujud-unit-keystone-mysql-router-0-exec-start.sh
             └─19336 /var/lib/juju/tools/unit-keystone-mysql-router-0/jujud unit --data-dir /var/lib/juju --unit-name keystone-mysql-router/0 --debug

Sep 29 08:46:48 juju-e1577b-0-lxd-2 systemd[1]: Started juju unit agent for keystone-mysql-router/0.
root@juju-e1577b-0-lxd-2:~#

and when checked mysql router its saying

root@juju-e1577b-0-lxd-2:/var/lib/mysql/keystone-mysql-router/log# tail -f mysqlrouter.log
2021-09-29 08:45:53 metadata_cache INFO [7fa06c46c700] Potential changes detected in cluster 'jujuCluster' after metadata refresh
2021-09-29 08:45:53 metadata_cache INFO [7fa06c46c700] Metadata for cluster 'jujuCluster' has 1 replicasets:
2021-09-29 08:45:53 metadata_cache INFO [7fa06c46c700] 'default' (3 members, single-primary)
2021-09-29 08:45:53 metadata_cache INFO [7fa06c46c700] 10.222.88.90:3306 / 33060 - mode=RO
2021-09-29 08:45:53 metadata...