Leadership claims, document larger than capped size

[Adam Collard]

Juju 1.24.0, local environment, upgraded from 1.23.3 (see bug 1466565)

I get repeated chunks of messages like this from debug-log:

http://paste.ubuntu.com/11777688/

None of the charms in question implement the leader hooks, nor does leadership make sense for them. jujud is now back at the top of top(1)

[Curtis Hovey (sinzui)]

The work to fix bug 1468994 may have addressed this. I may be fixed in 1.24.2, otherwise we will fix in 1.24.3

[Darryl Weaver (dweaver)]

I am using juju 1.24.2 and I am still seeing this problem.
Jujud goes a bit mad when started and goes to 100% cpu and spews out a lot of messages about leadership elections.
Then the API can be unresponsive and I get a lot of messages about the api exiting.
If I restart jujud on node 0, it fixes the unresponsiveness for a while, but then fails again after a while.

Here is my all-machines.log from the juju bootstrap node, in case the log files help.

[Darryl Weaver (dweaver)]

I should add that I am running low on memory on the bootstrap node, it has 2GB of memory, but I am seeing some swap use now, this may be an additional factor.

[Darryl Weaver (dweaver)]

Tested increasing memory, in case it was a factor. It was not, I increased memory by 50% and restarted the bootstrap node.

Tried a brand new deployment. A small deployment of percona-cluster and a few machines using add-machine was all working.

I then deploy an openstack bundle without relations and everything is working.
When I then add all the relations for the openstack bundle, that is when the leadership election starts and jujud spins for hours re-running config-changed on a lot of services, juju status tells me there are 27 services still running config-changed.
After leaving it overnight, juju API is no longer responsive and restarting only fixes it for a few minutes until the API fails again and is no longer responsive, although still listening on port 17070.

[Darryl Weaver (dweaver)]

Here is a copy of the juju status output from when it had been running for about an hour.

[Darryl Weaver (dweaver)]

Here is a copy of all-machines.log from the time I started the deployment and running overnight, then trying to get it working by rebooting node 0 and also restarting jujud for machine 0.

[Darryl Weaver (dweaver)]

My openstack bundle included percona-cluster with 3 nodes.
I have re-deployed with 1 node of MySQL instead of 3 percona-cluster nodes and now I am not seeing the same problem.

[William Reade (fwereade)]

We believe this is fixed in 1.24.4; can you still repro?