MySQL units in error state due to 'leader-elected' hook failing
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Invalid
|
Undecided
|
Unassigned | ||
OpenStack Percona Cluster Charm |
Fix Released
|
Undecided
|
Felipe Reyes | ||
percona-cluster (Juju Charms Collection) |
Invalid
|
Undecided
|
Unassigned |
Bug Description
Openstack cloud already deployed. It was already properly working.
Queens + 18.05 charms - Xenial
Suddenly 2 on of 3 MySQL units (the leader is the other one) started to show Juju error messages:
"
2018-07-31 08:15:18 DEBUG worker.uniter.jujuc server.go:178 running hook tool "leader-set"
2018-07-31 08:15:18 DEBUG juju.worker.
2018-07-31 08:15:18 DEBUG leader-elected ERROR cannot write leadership settings: cannot write settings: not the leader
2018-07-31 08:15:18 DEBUG leader-elected Traceback (most recent call last):
2018-07-31 08:15:18 DEBUG leader-elected File "/var/lib/
2018-07-31 08:15:18 DEBUG leader-elected main()
2018-07-31 08:15:18 DEBUG leader-elected File "/var/lib/
2018-07-31 08:15:18 DEBUG leader-elected hooks.execute(
2018-07-31 08:15:18 DEBUG leader-elected File "/var/lib/
2018-07-31 08:15:18 DEBUG leader-elected self._hooks[
2018-07-31 08:15:18 DEBUG leader-elected File "/var/lib/
2018-07-31 08:15:18 DEBUG leader-elected leader_
2018-07-31 08:15:18 DEBUG leader-elected File "/var/lib/
2018-07-31 08:15:18 DEBUG leader-elected return f(*args, **kwargs)
2018-07-31 08:15:18 DEBUG leader-elected File "/var/lib/
2018-07-31 08:15:18 DEBUG leader-elected subprocess.
2018-07-31 08:15:18 DEBUG leader-elected File "/usr/lib/
2018-07-31 08:15:18 DEBUG leader-elected raise CalledProcessEr
2018-07-31 08:15:18 DEBUG leader-elected subprocess.
2018-07-31 08:15:18 ERROR juju.worker.
2018-07-31 08:15:18 DEBUG juju.worker.
2018-07-31 08:15:18 INFO juju.worker.uniter resolver.go:100 awaiting error resolution for "leader-elected" hook
2018-07-31 08:15:18 DEBUG juju.worker.uniter agent.go:17 [AGENT-STATUS] error: hook failed: "leader-elected"
"
Note that the hook being called is 'leader-elected' which is actually a single line of python - the leader-set call.
Yes we can wrap that in an is_leader check, but it looks like something went funky with the leadership managed via juju across these units.