upgrade is broken on aws, hp and canonistack when using a custom tools-url

[Aaron Bentley]

Since r2132, upgrade has been broken for CI on both hp and canonistack, suggesting it's an openstack provider bug.
It reports that it could not find matching tools, but Curtis confirms that 1.17.0-precise-amd64 is provided on Canonistack, and deploys of 1.17.0 are still working.

http://162.213.35.54:8080/job/canonistack-upgrade/75/consoleFull#1451402014d8608d74-2df3-423a-90f6-1f87b3f4972c

[Curtis Hovey (sinzui)]

We see upgrades are very broken for aws, hp, and canonistack because upgrade dies when a 404 is encountered.
2013-12-16 19:22:22 ERROR juju.cmd supercommand.go:294 invalid URL "https://streams.canonical.com/juju/tools/streams/v1/index.json" not found
Traceback (most recent call last):
  File "/var/lib/jenkins/ci-cd-scripts2/upgrade-juju", line 13, in <module>
    main()
  File "/var/lib/jenkins/ci-cd-scripts2/upgrade-juju", line 10, in main
    jujupy.Environment.from_config(env).upgrade_juju()
  File "/var/lib/jenkins/ci-cd-scripts2/jujupy.py", line 190, in upgrade_juju
    self.client.juju(self, 'upgrade-juju', args)
  File "/var/lib/jenkins/ci-cd-scripts2/jujupy.py", line 108, in juju
    return subprocess.check_call(args)
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('juju', '--show-log', 'upgrade-juju', '-e', 'test-release-aws', '--version', '1.17.0')' returned non-zero exit status 1

http://162.213.35.54:8080/job/aws-upgrade/236/console
http://162.213.35.54:8080/job/hp-upgrade/236/console
http://162.213.35.54:8080/job/canonistack-upgrade/237/console

In all cases the tools-url is set to ensure juju sees the 1.17.0 release candidate. I cannot think of a reason that juju would be looking at any location other than the tools-url.

[John A Meinel (jameinel)]

I think I figured this out.

This is because of Ian's recent patch to move the config warning stuff out of config.Validate and into env.Config.Get() (or wherever it specifically ended up.)

The issue is that in juju-1.16.* the config entry is 'tools-url'. For consistency, he changed it to 'tools-metadata-url' in 1.17 and added code to cause it to move the value from one place to the other.

However, when the code was in Validate, it was being triggered too often, causing spurious warnings (tools-metadata-url exists and thus tools-url should be removed, even though only tools-url existed in the original file).

So he move the verify-and-copy code into the point at which we open the environment, which only happens 1 time for a given command.

However, *upgrade-juju* acts slightly differently. It connects to the state server, then asks for its EnvironConfig, and then locally Validates it and reuploads it.

So it *used* to read the remote value, see that it had tools-url, and then copy that value into tools-metadata-url before searching for tools and then writing the new config back to the server.

However, with Validate not doing the copy, it means we now have a case where when it reads the remote EnvironConfig and then Validates it, it then searches for tools (to make sure it can find the tools you are asking to upgrade to). Since tools-metadata-url is no longer filled out, it fails.
Going further, even if the local 'juju upgrade-juju' process figured it out, on the remote side once machine-0 is upgraded, it *also* needs to have both bits set so that when another agent comes in and says "you want me to upgrade, can you point me to my tools" then machine-0 needs to be able to search for them to hand them to the machine and unit agents.

I'm tempted to just revert the fix for tools-url vs tools-metadata-url warning spam. While bad warnings are bad, it isn't just broken. :)

[John A Meinel (jameinel)]

You probably could test my hypothesis by setting *both* tools-url and tools-metadata-url before you bootstrap with 1.16.4. As long as the juju client ends up setting both of those in Mongo, then the 1.17.0 client should see both URLs and still do the right thing. You'll get more warning fluff, but the upgrade *should* work.

[Curtis Hovey (sinzui)]

Thank you for your help John. I think your hypothesis is correct. I did define both tools-url and tools-metadata-url. I was able to upgrade from 1.16.5 to 1.17.0.