edge: bundle is blocked by relations not allowed to non-leader

[Greg Lutostanski]

Not sure exactly what is causing this. Our ci for jujuedge has blocked deployments for openstack-base.

Here are crashdumps of two different bad runs (2.3-alpha1+develop-8657339):
https://10.245.162.101/artifacts/65f2ae63-aae6-4cde-822d-d4d12eb84f36/deploy_prepared_bundle_542/juju-crashdump-13bc6e26-87fd-43b6-ab2d-6b17e2950a3e.tar.xz
https://10.245.162.101/artifacts/27db770f-c343-4d16-a1d8-e41792cf53cd/deploy_prepared_bundle_564/juju-crashdump-0b36ebdf-2f1d-4d08-ac29-2b137b708a18.tar.xz

And for comparison a good one for jujubeta (2.2.4):
https://10.245.162.101/artifacts/5a14a3e5-6255-445f-8c0e-0f1faf9efb43/deploy_prepared_bundle_560/juju-crashdump-f2370f48-9eae-4f48-ad07-53495665846b.tar.xz

There is a significant difference in logs collected... I am looking at juju_debuglog.txt from the respective tarballs.

The thing that jumps out at me are the lines similar to:
manifold worker returned unexpected error: failed to initialize uniter for "unit-ceph-mon-1": cannot create relations: "ceph-mon/1" is not leader of "ceph-mon"
(small snippet here: https://pastebin.canonical.com/199251/).

It seems like a definite possibility since some of those that are spewing this message, the units asking for the relation are stuck in a 'blocked' state, causing the deployment to be blocked as a whole.

Last good run is juju_2.3-alpha1+develop-98ba1ff, earliest run that failed in this way is juju_2.3-alpha1+develop-cd0aea6

so with 98ba1ff (good) and cd0aea6 (bad) git commits, could be a short bisect.

[Nicholas Skaggs (nskaggs)]

Greg, can you sanitize and upload those dumps to a readable location? Attaching them to the bug would be ideal.

[Ian Booth (wallyworld)]

This issue is due to introducing the concept of relation status and having the unit agents set the relation status to "joined" after running the joined hook. There's a leadership check in the controller but not one in the uniter.

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/7876

[Greg Lutostanski (lutostag)]

Working in our ci at the moment, I know its just in edge, but since that is what was broken, considering it fixed released for our purposes.

Thanks!