BundleDeployCharmStoreSuite.TestAddMetricCredentials fails intermittently

[Heather Lanigan]

http://ci.jujucharms.com/job/github-merge-juju/739/console:

[xenial] ok github.com/juju/juju/cmd/juju/action 19.253s
[xenial]
[xenial] ----------------------------------------------------------------------
[xenial] FAIL: bundle_test.go:79: BundleDeployCharmStoreSuite.TestAddMetricCredentials
[xenial]
[xenial] [LOG] 0:00.003 DEBUG juju.environs.tools reading v1.* agent binaries
[xenial] [LOG] 0:00.003 INFO juju.environs.testing uploading FAKE tools 2.4-beta1-precise-amd64
...
[xenial] /home/ubuntu/juju-core_2.4-beta1/src/github.com/juju/testing/stub.go:222:
[xenial] return c.Check(funcNames, jc.DeepEquals, expected)
[xenial] ... obtained []string = []string{"Authorize", "DefaultPlan", "Authorize"}
[xenial] ... expected []string = []string{"DefaultPlan", "Authorize", "Authorize"}
[xenial] ... mismatch at [0]: unequal; obtained "Authorize"; expected "DefaultPlan"
...
[xenial] OOPS: 299 passed, 1 skipped, 1 FAILED
[xenial] --- FAIL: TestPackage (91.20s)
[xenial] FAIL
[xenial] FAIL github.com/juju/juju/cmd/juju/application 94.052s
[xenial] ok github.com/juju/juju/cmd/juju/backups 1.047s

[Anastasia (anastasia-macmood)]

This happens because our testing infrastructure checks if all expected method calls were done in the predetermined order. The order of calls here is not always the same and, in fact, the order is not important to the test.

This is best fixed on both juju-testing side (to provide a check method that just checks the calls and ignores the call order) and juju side (to use new stub testing method in situations like above where the call order does not matter).

Part 1 - PR against juju/testing https://github.com/juju/testing/pull/132

[Anastasia (anastasia-macmood)]

Actually, removing 2.3 milestone as the change was introduced to develop (2.4) only by https://github.com/juju/juju/commit/081b5d3c0fbdfeaadfc2b5a48dfdd3abdd376b8b

[Anastasia (anastasia-macmood)]

PR agaisnt develop (2.4) that fixes the actual intermittent failure: https://github.com/juju/juju/pull/8287