1.24+ cannot upgrade in canonistack

[Curtis Hovey]

The QA team cannot identify a substrate issue that prevents upgrades of 1.24+ to the juju under test.
    http://reports.vapour.ws/releases/issue/558c2d4b749a562bca8600d6

Juju reports there are "no matching tools" when we can see them there and other jobs can see and use them.

1.22 can upgrade to the version under test. We have added --debug, but found no revealing information in the logs. We are certain though that the log lines about streams are lying. Juju is NOT checking the site/streams it claims to be checking. Here are the crucial points we have investigated over 3 months.

1. We can deploy in canonistack. Our localhost does see the streams and agents under test when we wget them.
2. While attempting to upgrade, we ran juju metadata validate-tools on the same host
   A. It reported the streams are fine and found the version under test.
   B. the upgrading juju said no matching tools
3. We can switch the test to use streams from other clouds. eg point joyent and canonistack tests to use streams in AWS
   A, we can deploy in canonistack.
   B. we cannot upgrade in canonistack because of no matching tools
   C. Other substrates can deploy and upgrade
4. After we release the version under test, we can upgrade.
   A. We suspect that juju's streams are registered in canonistack's keystone.
    B That is why the version cloud be upgraded to after release.

As we know juju is using mirrors, but the logs claim to be using streams.canonical.com, we suspect that Juju is not actually looking at the stream specified using agent-stream-url. We can deploy because out localhost is doing the work. Once the state-server takes over, it is check streams in a non-obvious and probably wrong order.

[Curtis Hovey (sinzui)]

Since the release of 1.24.7 to stable CI sees that 1.24.7 can upgrade to 1.25.0, 1.25-beta2, and 1.25.1. It *cannot* upgrade to 1.24.8 or 1.26-alpha1