Comment 1 for bug 1882205

Revision history for this message
Sachin Kulkarni (sachin-trilio) wrote : Re: hook failed: "db-router-relation-changed" due to "The MySQL server is running with the --super-read-only option so it cannot execute this statement"

Hit this issue while deploying "Ussuri" on focal.

mysql-innodb-cluster - Gets in error with "hook failed: "cluster-relation-changed""
Ussuri Bundle - cs:bundle/openstack-base-69
mysql-innodb-cluster charms used - cs:mysql-innodb-cluster-0 and cs:~openstack-charmers-next/mysql-innodb-cluster-42

Debug logs -
unit-mysql-innodb-cluster-6: 11:17:12 INFO unit.mysql-innodb-cluster/6.juju-log cluster:90: Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:69:joined:cluster
unit-mysql-innodb-cluster-6: 11:17:12 DEBUG unit.mysql-innodb-cluster/6.juju-log cluster:90: tracer>
tracer: set flag cluster.available
tracer: ++ queue handler reactive/mysql_innodb_cluster_handlers.py:88:create_remote_cluster_user
tracer: -- dequeue handler reactive/mysql_innodb_cluster_handlers.py:66:send_cluster_connection_info
unit-mysql-innodb-cluster-6: 11:17:12 INFO unit.mysql-innodb-cluster/6.juju-log cluster:90: Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:75:changed:cluster
unit-mysql-innodb-cluster-6: 11:17:12 INFO unit.mysql-innodb-cluster/6.juju-log cluster:90: Invoking reactive handler: hooks/relations/tls-certificates/requires.py:109:broken:certificates
unit-mysql-innodb-cluster-6: 11:17:13 INFO unit.mysql-innodb-cluster/6.juju-log cluster:90: Invoking reactive handler: hooks/relations/mysql-router/provides.py:47:joined:db-router
unit-mysql-innodb-cluster-6: 11:17:13 INFO unit.mysql-innodb-cluster/6.juju-log cluster:90: Invoking reactive handler: hooks/relations/mysql-router/provides.py:53:changed:db-router
unit-mysql-innodb-cluster-6: 11:17:13 INFO unit.mysql-innodb-cluster/6.juju-log cluster:90: Invoking reactive handler: reactive/mysql_innodb_cluster_handlers.py:88:create_remote_cluster_user
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.juju-log cluster:90: Creating remote users.
unit-mysql-innodb-cluster-6: 11:17:14 WARNING unit.mysql-innodb-cluster/6.juju-log cluster:90: This instance is not yet clustered: cannot determine the cluster status.
unit-mysql-innodb-cluster-6: 11:17:14 WARNING unit.mysql-innodb-cluster/6.juju-log cluster:90: Cannot determine the cluster primary RW node for writes.
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.juju-log cluster:90: Opening db connection for root@None
unit-mysql-innodb-cluster-6: 11:17:14 WARNING unit.mysql-innodb-cluster/6.juju-log cluster:90: User clusteruser already exists.
unit-mysql-innodb-cluster-6: 11:17:14 ERROR unit.mysql-innodb-cluster/6.juju-log cluster:90: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/charm/reactive/mysql_innodb_cluster_handlers.py", line 103, in create_remote_cluster_user
    instance.create_cluster_user(
  File "lib/charm/openstack/mysql_innodb_cluster.py", line 519, in create_cluster_user
    m_helper.execute(SQL_CLUSTER_USER_GRANT.format(
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/charmhelpers/contrib/database/mysql.py", line 178, in execute
    cursor.execute(sql)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1290, 'The MySQL server is running with the --super-read-only option so it cannot execute this statement')

unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed Traceback (most recent call last):
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/charm/hooks/cluster-relation-changed", line 22, in <module>
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed main()
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed bus.dispatch(restricted=restricted_mode)
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed _invoke(other_handlers)
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed handler.invoke()
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed self._action(*args)
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/charm/reactive/mysql_innodb_cluster_handlers.py", line 103, in create_remote_cluster_user
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed instance.create_cluster_user(
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed File "lib/charm/openstack/mysql_innodb_cluster.py", line 519, in create_cluster_user
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed m_helper.execute(SQL_CLUSTER_USER_GRANT.format(
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/charmhelpers/contrib/database/mysql.py", line 178, in execute
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed cursor.execute(sql)
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed res = self._query(query)
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed db.query(q)
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed File "/var/lib/juju/agents/unit-mysql-innodb-cluster-6/.venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed _mysql.connection.query(self, query)
unit-mysql-innodb-cluster-6: 11:17:14 DEBUG unit.mysql-innodb-cluster/6.cluster-relation-changed MySQLdb._exceptions.OperationalError: (1290, 'The MySQL server is running with the --super-read-only option so it cannot execute this statement')
unit-mysql-innodb-cluster-6: 11:17:14 ERROR juju.worker.uniter.operation hook "cluster-relation-changed" failed: exit status 1