Juju r2282 cannot deploy or upgrade azure

[Curtis Hovey]

Juju 1.17.2 release candidates have not been able to bootstrap since r2282. The most common reasons are for bootstrap look like this.
    http://162.213.35.54:8080/job/azure-deploy/759/console

2014-01-30 19:20:30 INFO juju.provider.azure environ.go:392 picked tools &{"1.17.2-precise-amd64" "https://jujutools.blob.core.windows.net/juju-tools/testing/tools/releases/juju-1.17.2-precise-amd64.tgz" "407455166d378e98426cc1a2a76375ae2d2b269cef2c93756de9d7ebe0b01b42" %!q(int64=5310991)}
2014-01-30 19:21:50 ERROR juju.provider.common bootstrap.go:140 bootstrap failed: cannot start bootstrap instance: POST request failed (307: Temporary Redirect)
2014-01-30 19:22:12 ERROR juju.cmd supercommand.go:294 cannot start bootstrap instance: POST request failed (307: Temporary Redirect)
Command '('juju', '--show-log', 'bootstrap', '-e', 'test-release-azure2', '--constraints', 'mem=2G')' returned non-zero exit status 1
+ EXIT_STATUS=1

or this http://162.213.35.54:8080/job/azure-deploy/739/console

2014-01-30 08:38:57 INFO juju.provider.azure environ.go:392 picked tools &{"1.17.2-precise-amd64" "https://jujutools.blob.core.windows.net/juju-tools/testing/tools/releases/juju-1.17.2-precise-amd64.tgz" "c60abe982aeeb46beb1c03820e8bf53ba220991ebdf59990919fcd79d03a1fb1" %!q(int64=5252511)}
 - juju-test-release-azure2-exz3pc4dj8
Waiting for address
Attempting to connect to 10.0.0.4:22
Attempting to connect to juju-test-release-azure2-exz3pc4dj8.cloudapp.net:22
2014-01-30 08:42:57 ERROR juju.provider.common bootstrap.go:139 bootstrap failed: getting addresses: GET request failed: MissingOrIncorrectVersionHeader - The versioning header is not specified or was specified incorrectly. (http code 400: Bad Request)
Stopping instance...

Azure health checks using 1.16.5 do not see this problem. This whole problem probably relates to bug 1272558 where instances are not destroyed-properly and interfere with the next test.

[Curtis Hovey (sinzui)]

One build progressed farther than others. It got the error after boot strapping and I was able to capture a log. http://162.213.35.54:8080/job/azure-deploy/789/console

juju --show-log expose -e test-release-azure2 wordpress
2014-01-31 13:44:33 INFO juju api.go:231 connecting to API addresses: [juju-test-release-azure2-hwcjmx5e66.cloudapp.net:17070]
2014-01-31 13:44:33 INFO juju apiclient.go:118 state/api: dialing "wss://juju-test-release-azure2-hwcjmx5e66.cloudapp.net:17070/"
2014-01-31 13:44:34 INFO juju apiclient.go:128 state/api: connection established
2014-01-31 13:44:35 INFO juju.cmd supercommand.go:298 command finished
<test-release-azure2> 1 is in state (error: could not create hosted service: POST request failed (307: Temporary Redirect))
+ EXIT_STATUS=1
+ '[' 1 -ne 0 ']'

[Andrew Wilkins (axwalk)]

I've been looking into the redirects. There's a couple of problems:
 - 307 does not incur following of the redirect in the client for requests other than GET and HEAD. This is in accordance to the spec. It's not clear to me whether in Azure we should automatically follow them or not, but it seems that's what is expected.
 - When a GET or HEAD *does* follow redirects, the followup requests drop all of the headers from the original request. This is why we often see MissingOrIncorrectVersionHeader errors from Azure.

[Curtis Hovey (sinzui)]

I ran this from my command line and instantly got this response:

juju --debug bootstrap --constraints mem=4G
2014-02-04 14:00:00 WARNING juju.environs config.go:91 Config attribute "tools-url" (https://jujutools.blob.core.windows.net/juju-tools/tools) is deprecated.
The location to find tools is now specified using the "tools-metadata-url" attribute.
Your configuration should be updated to set "tools-metadata-url" as follows
tools-metadata-url: https://jujutools.blob.core.windows.net/juju-tools/tools.
2014-02-04 14:00:00 DEBUG juju.environs.configstore disk.go:63 Making /home/curtis/.juju/environments
2014-02-04 14:00:00 DEBUG juju.provider.azure environprovider.go:32 opening environment "test-azure".
2014-02-04 14:00:08 INFO juju.environs.bootstrap bootstrap.go:46 bootstrapping environment "test-azure"
2014-02-04 14:00:17 ERROR juju.cmd supercommand.go:294 PUT request failed (307: Temporary Redirect)

My second try 10 seconds latter got much further, but it is the same error from the report.:
....
Waiting for address
Attempting to connect to 10.0.0.4:22
Attempting to connect to juju-test-azure-hfp1upzaup.cloudapp.net:22
2014-02-04 14:02:41 ERROR juju.provider.common bootstrap.go:140 bootstrap failed: getting addresses: GET request failed: MissingOrIncorrectVersionHeader - The versioning header is not specified or was specified incorrectly. (http code 400: Bad Request)
Stopping instance...
2014-02-04 14:03:22 ERROR juju.provider.common bootstrap.go:154 cannot stop failed bootstrap instance "juju-test-azure-hfp1upzaup": DELETE request failed (307: Temporary Redirect)
2014-02-04 14:03:43 ERROR juju.cmd supercommand.go:294 getting addresses: GET request failed: MissingOrIncorrectVersionHeader - The versioning header is not specified or was specified incorrectly. (http code 400: Bad Request)

Are you looking in the gwacl? Do you suspect that Azure had an API change?

[Curtis Hovey (sinzui)]

I think we should be following the redirect. Since MS doesn't warn when they take down regions (or affinity groups), we rely on redirects to find the services that are up.

[Andrew Wilkins (axwalk)]

Yes, the change I've made (just merged) is in gwacl. I don't know what's changed, but the redirects seem to be more frequent than in the past. There is a newish API version, but AFAICT we are still calling things correctly.

Anyway, I've updated gwacl to follow all 307s for all methods when interacting with the management API. I'm updating dependencies.tsv now. I'll mark lp:1269835 as fixed, and we'll see how CI goes.