Azure bootstrap fails: versioning header is not specified

[Curtis Hovey]

[Impact]
Bootstrap of Juju environment on Azure fails

[Test Case]
configure azure environment
juju bootstrap
(fails)

[Regression Potential]
Limited to Azure provider as in gwacl library.

[Original Bug Report]
I cannot bootstrap an azure environment that I could bootstrap 2 weeks ago. The azure-provider fails getting the OS image. Today and in the past, I used juju 1.16.0 to bootstrap. I think something outside of juju has changed

My config relies on the defaults except for the fact that I set the tools-urls to a testing location. Using the real tools location does not help. I have tried setting combinations of image-metadata-url, default-series, and image-stream to get past the error, but no success.

From the pastebin http://pastebin.ubuntu.com/6326844/ :
    2013-10-29 22:55:14 DEBUG juju.environs.simplestreams simplestreams.go:888
    finding products at path "streams/v1/com.ubuntu.cloud:released:azure.sjson"
    2013-10-29 22:55:32 ERROR juju supercommand.go:282
    cannot start bootstrap instance: POST request failed: BadRequest - The versioning
    header is not specified or was specified incorrectly. (http code 400: Bad Request)

@wallyworld investigated the issue for a while

Azure is rejecting certain API calls complaining the
version we are putting in the request header is wrong, when according to the
Azure doco we are sending the right version, and nothing is supposed to have
changed. Hmmmm. But not all API calls are affected.

Also, there's a separate issue with Go going tits up with a closed connection
error. This is sort of a timing issue which is a limitation of how Go's http lib
is implemented. We worked around it in gomaasapi, and the same fix was aplied to
gwacl, but it seems not to have fixed it there.

[Curtis Hovey (sinzui)]

@utlemming did fix a bug in http://cloud-images.ubuntu.com streams recently, some images were not properly described as released.
    beta2 ubuntu-saucy-13.10-amd64-server-beta2
    alpha3 ubuntu-saucy-13.10-amd64-server-alpha3
    release ubuntu-saucy-13.10-amd64-server-20131015

Since I adjusted image-metadata-url, default-series, and image-stream without success, I am don't think stream data is at fault. There are 3 streams available. I cannot bootstrap when I specify the stream (release or beta2), series (saucy or precise). I saw this error for release and precise.

    2013-10-30 00:42:57 DEBUG juju.environs.simplestreams simplestreams.go:443
    index file has no data for product name(s)
    ["com.ubuntu.cloud.release:server:12.04:amd64" "com.ubuntu.cloud.release:server:12.04:i386"]
    2013-10-30 00:43:13 ERROR juju supercommand.go:282
    cannot start bootstrap instance: no OS images found for location]
    "West US", series "precise", architectures ["amd64" "i386"]
    (and endpoint: "https://management.core.windows.net/")

[Curtis Hovey (sinzui)]

I cannot bootstrap with juju the previous stable release 1.14.1. I get the same error:

    2013-10-30 15:33:21 ERROR juju supercommand.go:282
    command failed: cannot start bootstrap instance: POST request failed:
    BadRequest - The versioning header is not specified or was specified incorrectly.
    (http code 400: Bad Request)

I think then that azure or streams have changed.

[Antonio Rosales (arosales)]

This bug also affects me. I am seeing[0] similar behaviour and I am unable to bootstrap in US West or US East.

[0] http://pastebin.ubuntu.com/6330453/

-thanks,
Antonio

[Launchpad Janitor (janitor)]

This bug was fixed in the package juju-core - 1.16.2-0ubuntu1

---------------
juju-core (1.16.2-0ubuntu1) trusty; urgency=low

  * New upstream point release.
    (LP: #1240709, #1240927, #1246320, #1246556, #1245004)
    (LP: #1081247, #1229275, #1239508, #1240423, #1241666, #1243861).
 -- James Page <email address hidden> Thu, 31 Oct 2013 21:22:45 +0000

[Brian Murray (brian-murray)]

Hello Curtis, or anyone else affected,

Accepted juju-core into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/juju-core/1.16.3-0ubuntu0.13.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

2013-11-22 12:17:57 INFO juju.state open.go:106 connection established
environment: azure
machines:
  "0":
    agent-state: started
    agent-version: 1.16.3
    dns-name: juju-azure-3qam297c0t.cloudapp.net
    instance-id: juju-azure-3qam297c0t
    instance-state: Created
    series: precise
services: {}
2013-11-22 12:18:05 INFO juju supercommand.go:286 command finished

[Launchpad Janitor (janitor)]

This bug was fixed in the package juju-core - 1.16.3-0ubuntu0.13.10.1

---------------
juju-core (1.16.3-0ubuntu0.13.10.1) saucy-proposed; urgency=low

  * New upstream stable point release:
    - MAAS: juju destroy-environment also destroys nodes that are not
      controlled by juju/in different juju environments
      (LP: #1229275, #1081247).
    - MAAS: LXC container provisioning broken due to missing secrets
      in API calls (LP: #1246556).
    - MAAS: disambiguate use of environment uuid between state server
      and environment configuration (LP: #1240423).
    - local: provider fails to start due to missing setup of bootstrap
      storage (LP: #1240709).
    - local: local provider deploys fail due to inclusion of lxc package
      within LXC containers (LP: #1247299).
    - Azure: bootstrap fails due to old API version headers (LP: #1246320).
    - client: os.rename does not work on Windows (LP: #1240927).
    - simplestreams: cannot create simplestreams data for Ubuntu Trusty
      (LP: #1241666).
    - cloud-init: Embed full cloud-archive keyring in metadata to avoid
      calls to keyserver.ubuntu.com which fail in egress restricted
      data center environments (LP: #1243861).
    - core: regression - relation name regex is to restrictive (LP: #1245004).
 -- James Page <email address hidden> Thu, 21 Nov 2013 10:30:39 +0000

[Stéphane Graber (stgraber)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[David Britton (dpb)]

Should this also be ported to 1.17, since that is the default with trusty?

[Ian Booth (wallyworld)]

The fix was initially developed for 1.17 and backported to 1.16. So trusty will indeed ship with the fix.