Comment 4 for bug 1768241

Unfortunately, the crashdump file doesn't show when the leader changed; there's no leader-elected for it in the logs. mysql, on the other hand is trying to do something weird with leadership ever 5 minutes. However, there are no mysql charm unit logs in the crashdump:

tar xJvf juju-crashdump-c16a6243-ad52-4eb6-838c-5fbf0fd71c6b.tar.xz | grep mysql
0/lxd/6/var/log/mysql.log
0/lxd/6/var/log/juju/unit-hacluster-mysql-1.log
0/lxd/6/var/log/mysql.err
0/lxd/6/var/log/mysql/
0/lxd/6/var/log/mysql/mysql-bin.index
0/lxd/6/var/log/mysql/error.log
1/lxd/7/var/log/mysql.log
1/lxd/7/var/log/juju/unit-hacluster-mysql-0.log
1/lxd/7/var/log/mysql.err
1/lxd/7/var/log/mysql/
1/lxd/7/var/log/mysql/mysql-bin.000001
1/lxd/7/var/log/mysql/mysql-bin.index
1/lxd/7/var/log/mysql/error.log
12/lxd/7/var/log/mysql.log
12/lxd/7/var/log/juju/unit-hacluster-mysql-2.log
12/lxd/7/var/log/mysql.err
12/lxd/7/var/log/mysql/
12/lxd/7/var/log/mysql/mysql-bin.000001
12/lxd/7/var/log/mysql/mysql-bin.index
12/lxd/7/var/log/mysql/error.log
hacluster-mysql_0
hacluster-mysql_1
hacluster-mysql_2
mysql_0
mysql_1
mysql_2

And ceph-mon/0 is missing:
tar xJvf juju-crashdump-c16a6243-ad52-4eb6-838c-5fbf0fd71c6b.tar.xz | grep ceph-mon
0/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-0-lxd-0.log.3.gz
0/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-0-lxd-0.log.6.gz
0/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-0-lxd-0.log.4.gz
0/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-0-lxd-0.log.7.gz
0/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-0-lxd-0.log.5.gz
0/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-0-lxd-0.log.2.gz
0/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-0-lxd-0.log.1.gz
0/lxd/0/var/lib/charm/ceph-mon/
0/lxd/0/var/lib/charm/ceph-mon/ceph.conf
1/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-1-lxd-0.log.5.gz
1/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-1-lxd-0.log.3.gz
1/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-1-lxd-0.log.2.gz
1/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-1-lxd-0.log.7.gz
1/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-1-lxd-0.log.1.gz
1/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-1-lxd-0.log.6.gz
1/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-1-lxd-0.log
1/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-1-lxd-0.log.4.gz
1/lxd/0/var/log/juju/unit-ceph-mon-1.log
1/lxd/0/var/lib/charm/ceph-mon/
1/lxd/0/var/lib/charm/ceph-mon/ceph.conf
12/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-12-lxd-0.log.6.gz
12/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-12-lxd-0.log
12/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-12-lxd-0.log.3.gz
12/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-12-lxd-0.log.1.gz
12/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-12-lxd-0.log.7.gz
12/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-12-lxd-0.log.2.gz
12/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-12-lxd-0.log.5.gz
12/lxd/0/var/log/ceph/ceph-mon.juju-33f5dc-12-lxd-0.log.4.gz
12/lxd/0/var/log/juju/unit-ceph-mon-2.log
12/lxd/0/var/lib/charm/ceph-mon/
12/lxd/0/var/lib/charm/ceph-mon/ceph.conf
ceph-mon_0
ceph-mon_1
ceph-mon_2

I think, sadly, the juju crash dump util didn't pick them up because it's huge.

(crashdump was shared via google drive as too big for launchpad).

One thing that is dying is mysql: (from debug_log.txt):

unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected ERROR cannot write leadership settings: cannot write settings: not the leader
unit-aodh-2: 04:51:49 INFO unit.aodh/2.juju-log Invoking reactive handler: reactive/aodh_handlers.py:38:setup_amqp_req
unit-filebeat-72: 04:51:49 ERROR unit.filebeat/72.juju-log Unable to find implementation for relation: requires of juju-info
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected Traceback (most recent call last):
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/leader-elected", line 840, in <module>
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected main()
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/leader-elected", line 830, in main
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected hooks.execute(sys.argv)
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected self._hooks[hook_name]()
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/leader-elected", line 802, in leader_elected
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected leader_set(**{'leader-ip': get_relation_ip('cluster')})
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/charmhelpers/core/hookenv.py", line 929, in inner_translate_exc2
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected return f(*args, **kwargs)
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected File "/var/lib/juju/agents/unit-mysql-2/charm/hooks/charmhelpers/core/hookenv.py", line 982, in leader_set
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected subprocess.check_call(cmd)
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected raise CalledProcessError(retcode, cmd)
unit-mysql-2: 04:51:49 DEBUG unit.mysql/2.leader-elected subprocess.CalledProcessError: Command '['leader-set', 'leader-ip=10.14.244.133']' returned non-zero exit status 1