openstack: looping over simplestream urls stops too soon

[Andreas Hasenack]

From what I understood, there is a list for simplestream URL candidates that the code loops over:
- start off with the juju tools url
- append from product-streams if such a service was found in keystone catalog
- append the default cloud-images.ubuntu.com/releases

If you start clean, with nothing in swift, this loop will stop early because swift returns a 401 for the first candidate when there is nothing there, and the code only moves on to the next one in the list if the error is environs.IsNotFoundError:
    for _, baseURL := range baseURLs {
        indexRef, err := getIndexWithFormat(baseURL, indexPath, "index:1.0", requireSigned)
        if err != nil {
            if environs.IsNotFoundError(err) {
                continue
            }
            return nil, err
        }

I think it should also consider the 401 HTTP error case and move on to the next URL in the list in that case. Or maybe publicStorage() from openstack/provider.go should actually create the "public-bucket" container ("juju-dist" by default) with the right ACLs. The code says it would be created, but that's not what happened:
    // If there is a public bucket URL defined, create a public storage using that,
    // otherwise create the public bucket using the user's credentials on the authenticated client.
    publicBucketURL := e.publicBucketURL()
    if publicBucketURL == "" {
        e.publicStorageUnlocked = &storage{
            containerName: ecfg.publicBucket(),
            // this is possibly just a hack - if the ACL is swift.Private,
            // the machine won't be able to get the tools (401 error)
            containerACL: swift.PublicRead,
            swift: swift.New(e.client)}

Here is a sample run:

- swift is made sure to be empty:
$ swift list
juju-dist

$ swift delete juju-dist
streams/v1
streams/v1/index.json
streams/v1/imagemetadata.json

Ok, let's bootstrap:
$ juju bootstrap -v --upload-tools
2013/05/28 17:57:19 INFO environs/openstack: opening environment "canonistack1"
2013/05/28 17:57:22 INFO environs/tools: built 1.11.1.1-raring-amd64 (2320kB)
2013/05/28 17:57:22 INFO environs/tools: uploading 1.11.1.1-precise-amd64
2013/05/28 17:57:47 INFO environs/tools: uploading 1.11.1.1-raring-amd64
2013/05/28 17:58:09 INFO environs/openstack: bootstrapping environment "canonistack1"
2013/05/28 17:58:19 INFO environs: reading tools with major version 1
2013/05/28 17:58:19 INFO environs: filtering tools by series: precise
2013/05/28 17:58:19 INFO environs: filtering tools by version: 1.11.1.1
2013/05/28 17:58:21 ERROR command failed: cannot start bootstrap instance: cannot read index data, cannot access URL "https://swift.canonistack.canonical.com/v1/AUTH_85057xxx/juju-dist/streams/v1/index.json", "401 Unauthorized"
error: cannot start bootstrap instance: cannot read index data, cannot access URL "https://swift.canonistack.canonical.com/v1/AUTH_85057xxx/juju-dist/streams/v1/index.json", "401 Unauthorized"

Unauthorized. See how it only tried the first candidate, the URL derived from the tools.
Bootstrap failed, and I only have the tools now (that's my control bucket):
$ swift list
juju-51e360xxxxx

Let's try again, creating "juju-dist" this time:
$ swift post juju-dist

$ swift list
juju-51e360xxxx
juju-dist

$ juju bootstrap -v --upload-tools
2013/05/28 18:08:46 INFO environs/openstack: opening environment "canonistack1"
2013/05/28 18:08:49 INFO environs/tools: built 1.11.1.1-raring-amd64 (2320kB)
2013/05/28 18:08:49 INFO environs/tools: uploading 1.11.1.1-precise-amd64
2013/05/28 18:09:13 INFO environs/tools: uploading 1.11.1.1-raring-amd64
2013/05/28 18:09:35 INFO environs/openstack: bootstrapping environment "canonistack1"
2013/05/28 18:09:45 INFO environs: reading tools with major version 1
2013/05/28 18:09:46 INFO environs: filtering tools by series: precise
2013/05/28 18:09:46 INFO environs: filtering tools by version: 1.11.1.1
2013/05/28 18:09:48 ERROR command failed: cannot start bootstrap instance: cannot read index data, cannot access URL "https://swift.canonistack.canonical.com/v1/AUTH_850578306acb4f1baeaceae75ff7ada1/juju-dist/streams/v1/index.json", "401 Unauthorized"
error: cannot start bootstrap instance: cannot read index data, cannot access URL "https://swift.canonistack.canonical.com/v1/AUTH_850578306acb4f1baeaceae75ff7ada1/juju-dist/streams/v1/index.json", "401 Unauthorized"

Again unauthorized. Let's fix the container ACLs:
$ swift post juju-dist -r '.r:*'
$ juju bootstrap -v --upload-tools
2013/05/28 18:11:12 INFO environs/openstack: opening environment "canonistack1"
2013/05/28 18:11:15 INFO environs/tools: built 1.11.1.1-raring-amd64 (2320kB)
2013/05/28 18:11:15 INFO environs/tools: uploading 1.11.1.1-precise-amd64
2013/05/28 18:11:39 INFO environs/tools: uploading 1.11.1.1-raring-amd64
2013/05/28 18:12:02 INFO environs/openstack: bootstrapping environment "canonistack1"
2013/05/28 18:12:12 INFO environs: reading tools with major version 1
2013/05/28 18:12:12 INFO environs: filtering tools by series: precise
2013/05/28 18:12:12 INFO environs: filtering tools by version: 1.11.1.1
2013/05/28 18:12:26 INFO environs/openstack: started instance "f6d260c8-2bce-4482-a877-6cfe1bcce030"
2013/05/28 18:12:28 INFO command finished

Now it worked:
$ swift list
juju-51e360f7f9cbfdf5d24fdb84aa0d4112
juju-dist

The juju-dist container is still empty:
$ swift list juju-dist
$

Meaning that the code finally saw the NotFoundError that made it try the next URL in the list, which comes from the "product-stream" keystone service:
$ keystone catalog
(...)
Service: product-streams
+-------------+------------------------------------------------------------------------------------------------------+
| Property | Value |
+-------------+------------------------------------------------------------------------------------------------------+
| adminURL | https://swift.canonistack.canonical.com/v1/AUTH_a48xxxxxxxxxxxxxxxxxxxxxxxxxxxxx/simplestreams/data/ |
| id | db3cxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| internalURL | https://swift.canonistack.canonical.com/v1/AUTH_a48xxxxxxxxxxxxxxxxxxxxxxxxxxxxx/simplestreams/data/ |
| publicURL | https://swift.canonistack.canonical.com/v1/AUTH_a48xxxxxxxxxxxxxxxxxxxxxxxxxxxxx/simplestreams/data/ |
| region | xxxx1 |
+-------------+------------------------------------------------------------------------------------------------------+

[Ian Booth (wallyworld)]

I think there's some confusion around the public bucket behaviour. If a public bucket url is defined, the code assumes the bucket has been created and set up by the cloud administrator and so the code merely creates a storage client to access the already created bucket. That's what this part of the comment is referring to:

    // If there is a public bucket URL defined, create a public storage using that,

If there's no public bucket url defined, the code creates a so called "public" bucket on the user's account and calls the bucket whatever the public bucket name is configured to be (default "juju-dist"). That's what this part of the comment is referring to:

    // otherwise create the public bucket using the user's credentials on the

So AFAICT, this part of the code is behaving as expected.