WorkerSuite.TestUpdatesAfterPeriod timed out

[Curtis Hovey]

As seen in
    http://reports.vapour.ws/releases/issue/57990a8c749a564305531ed9

WorkerSuite.TestUpdatesAfterPeriod timed out

worker_test.go:129:
    c.Fatalf("timed out")
... Error: timed out

[Reed O'Brien (reedobrien)]

We found that clock.Advance could sometimes happen out of order and lead to this intermittent failure. We had several discussions about solutions in person at hte 9/2016 quality sprint and on IRC <https://irclogs.ubuntu.com/2016/09/30/%23juju-dev.txt>. We also added a wiki page: <https://github.com/juju/juju/wiki/Intermittent-failures>.

To solve this issue we added a clock.WaitAdvance method that ensures _n_ timers exist on clock.waiting within a given period before actually advancing the time or exiting with an error. Additionally, we have added a check to the clock.Advance function that logs a debug error if the clock is advanced with no timers in clock.waiting. QA should be able to use this as a rule to tell us when/if other intermittent failures are hitting this same problem. We fully expect that there are other cases.

[Reed O'Brien (reedobrien)]

The message sent to the debug log as mentioned above can be found at <https://github.com/juju/testing/blob/master/clock.go#L107>