state_leader_test LeadershipSuite.TestCheck has a race condition

[John A Meinel]

The test asserts that after waiting for the leadership to expire, it will then observe that the leasee does not hold the lock anymore.
My guess is that while its true that the app/0 has lost the lease, it is possible for it to get reacquired before the assertion is run.

The test run here shows the issue:
http://qa.jujucharms.com/releases/6015/job/run-unit-tests-race/attempt/3507

state_leader_test.go:116:
    c.Check(err, gc.ErrorMatches, `"application/0" is not leader of "application"`)
... value = nil
... regex string = "\"application/0\" is not leader of \"application\""
... Error value is nil

state_leader_test.go:117:
    c.Check(ops2, gc.IsNil)
... value []txn.Op = []txn.Op{txn.Op{C:"leases", Id:"application-leadership#application#", Assert:bson.M{"holder":"application/0"}, Insert:interface {}(nil), Update:interface {}(nil), Remove:false}}

Note, while this is found by --race, I think that's just an example of it changing the timing of the test. I don't think there is actually a data race, just a false assumption that there isn't anything going on that would renew the leadership.
Though, inserting time.Sleep() doesn't seem to be triggering the issue.

[Andrew Wilkins (axwalk)]

https://github.com/juju/juju/pull/8154

[Andrew Wilkins (axwalk)]

Back-ported to 2.3 branch in https://github.com/juju/juju/pull/8157.