[bionic] mysql is not up sometimes in AIO setup

[Andrey Pavlov]

ubuntu 18.04, All-In-One setup

2021-02-18 01:53:49 DEBUG juju-log shared-db:37: Opening db connection for root@None
2021-02-18 01:53:49 WARNING shared-db-relation-changed Traceback (most recent call last):
2021-02-18 01:53:49 WARNING shared-db-relation-changed File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/shared-db-relation-changed", line 1155, in <module>
2021-02-18 01:53:49 WARNING shared-db-relation-changed main()
2021-02-18 01:53:49 WARNING shared-db-relation-changed File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/shared-db-relation-changed", line 1145, in main
2021-02-18 01:53:49 WARNING shared-db-relation-changed hooks.execute(sys.argv)
2021-02-18 01:53:49 WARNING shared-db-relation-changed File "/var/lib/juju/agents/unit-mysql-0/charm/charmhelpers/core/hookenv.py", line 945, in execute
2021-02-18 01:53:49 WARNING shared-db-relation-changed self._hooks[hook_name]()
2021-02-18 01:53:49 WARNING shared-db-relation-changed File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/shared-db-relation-changed", line 853, in shared_db_changed
2021-02-18 01:53:49 WARNING shared-db-relation-changed db_helper)
2021-02-18 01:53:49 WARNING shared-db-relation-changed File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/shared-db-relation-changed", line 805, in configure_db_for_hosts
2021-02-18 01:53:49 WARNING shared-db-relation-changed password = db_helper.configure_db(host, database, username)
2021-02-18 01:53:49 WARNING shared-db-relation-changed File "/var/lib/juju/agents/unit-mysql-0/charm/charmhelpers/contrib/database/mysql.py", line 464, in configure_db
2021-02-18 01:53:49 WARNING shared-db-relation-changed self.connect(password=self.get_mysql_root_password())
2021-02-18 01:53:49 WARNING shared-db-relation-changed File "/var/lib/juju/agents/unit-mysql-0/charm/charmhelpers/contrib/database/mysql.py", line 100, in connect
2021-02-18 01:53:49 WARNING shared-db-relation-changed self.connection = MySQLdb.connect(**_connection_info)
2021-02-18 01:53:49 WARNING shared-db-relation-changed File "/usr/lib/python3/dist-packages/MySQLdb/__init__.py", line 86, in Connect
2021-02-18 01:53:49 WARNING shared-db-relation-changed return Connection(*args, **kwargs)
2021-02-18 01:53:49 WARNING shared-db-relation-changed File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 204, in __init__
2021-02-18 01:53:49 WARNING shared-db-relation-changed super(Connection, self).__init__(*args, **kwargs2)
2021-02-18 01:53:49 WARNING shared-db-relation-changed _mysql_exceptions.OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")
2021-02-18 01:53:50 ERROR juju.worker.uniter.operation runhook.go:136 hook "shared-db-relation-changed" (via explicit, bespoke hook script) failed: exit status 1

/var/log/syslog

Feb 18 01:19:42 juju-7f4505-0-lxd-3 mysql[11797]: * Stopping MySQL (Percona XtraDB Cluster) mysqld
Feb 18 01:19:57 juju-7f4505-0-lxd-3 /etc/init.d/mysql[11897]: MySQL PID not found, pid_file detected/guessed: /var/run/mysqld/mysqld.pid
Feb 18 01:19:57 juju-7f4505-0-lxd-3 /etc/init.d/mysql[11901]: MySQL PID not found, pid_file detected/guessed: /var/run/mysqld/mysqld.pid
Feb 18 01:19:57 juju-7f4505-0-lxd-3 mysql[11797]: ...done.
Feb 18 01:19:57 juju-7f4505-0-lxd-3 systemd[1]: Stopped Percona XtraDB Cluster daemon.
Feb 18 01:19:57 juju-7f4505-0-lxd-3 systemd[1]: system-mysql.slice: Failed to reset devices.list: Operation not permitted
Feb 18 01:19:57 juju-7f4505-0-lxd-3 systemd[1]: Created slice system-mysql.slice.
Feb 18 01:19:57 juju-7f4505-0-lxd-3 systemd[1]: Starting Percona XtraDB Cluster daemon with config /etc/default/mysql.bootstrap...
Feb 18 01:20:00 juju-7f4505-0-lxd-3 mysql[11974]: WSREP: Recovered position 51893779-7187-11eb-95b0-1ab480f92572:2
Feb 18 01:20:01 juju-7f4505-0-lxd-3 systemd[1]: Failed to attach 11978 to compat systemd cgroup /<email address hidden>: Permission denied
Feb 18 01:20:01 juju-7f4505-0-lxd-3 mysql[11978]: * Starting MySQL (Percona XtraDB Cluster) database server mysqld
Feb 18 01:20:01 juju-7f4505-0-lxd-3 /etc/init.d/mysql[12015]: MySQL PID not found, pid_file detected/guessed: /var/run/mysqld/mysqld.pid
Feb 18 01:20:05 juju-7f4505-0-lxd-3 mysql[11978]: ...done.
Feb 18 01:20:05 juju-7f4505-0-lxd-3 systemd[1]: Started Percona XtraDB Cluster daemon with config /etc/default/mysql.bootstrap.
Feb 18 01:24:17 juju-7f4505-0-lxd-3 mysql[16242]: * Stopping MySQL (Percona XtraDB Cluster) mysqld
Feb 18 01:24:17 juju-7f4505-0-lxd-3 /etc/init.d/mysql[16269]: MySQL PID not found, pid_file detected/guessed: /var/run/mysqld/mysqld.pid
Feb 18 01:24:17 juju-7f4505-0-lxd-3 /etc/init.d/mysql[16273]: MySQL PID not found, pid_file detected/guessed: /var/run/mysqld/mysqld.pid
Feb 18 01:24:17 juju-7f4505-0-lxd-3 mysql[16242]: ...done.

[Alex Kavanagh (ajkavanagh)]

Setting to medium as AIO is a test target, and wouldn't be used in production. Still be good to resolve why it is happening.