Bootstrap fails with --upload-tools on private openstack cloud

[Martin Packman]

IS have a test openstack cloud that has no external network conectivity. When using --upload-tools with bootstrap on juju 1.16.0.1, the tools get uploaded but then fails with "index file has no data for cloud". This leaves the environment in a semi-bootstrapped state as provider-state and bootstrap-verify are already created.

The same failure is given by `juju-metadata validate-images`, but trying to upload a json file created by `juju-metadata generate-image` doesn't help.

The log of a bootstrap run looks like:

$ juju bootstrap -v --constraints "cpu-cores=1 mem=2G root-disk=50" --upload-tools
verbose is deprecated with the current meaning, use show-log
2013-10-20 23:02:25 INFO juju.environs open.go:156 environment info already exists; using New not Prepare
2013-10-20 23:02:25 INFO juju.provider.openstack provider.go:116 opening environment "mojo-staging-certification"
2013-10-20 23:02:25 INFO juju.environs.tools build.go:162 found existing jujud
2013-10-20 23:02:25 INFO juju.environs.tools build.go:172 target: /tmp/juju-tools856028815/jujud
2013-10-20 23:02:28 INFO juju.environs.sync sync.go:235 built 1.16.0.1-precise-amd64 (4517kB)
2013-10-20 23:02:28 INFO juju.environs.sync sync.go:66 listing available tools
2013-10-20 23:02:28 INFO juju.environs.sync sync.go:90 found 1 tools
2013-10-20 23:02:28 INFO juju.environs.sync sync.go:100 listing target bucket
2013-10-20 23:02:28 INFO juju.environs.sync sync.go:113 found 0 tools in target; 1 tools to be copied
2013-10-20 23:02:28 INFO juju.environs.sync sync.go:143 copying 1.16.0.1-precise-amd64 from file:///tmp/206940322/tools/releases/juju-1.16.0.1-precise-amd64.tgz
2013-10-20 23:02:28 INFO juju.environs.sync sync.go:157 copying tools/releases/juju-1.16.0.1-precise-amd64.tgz
2013-10-20 23:02:29 INFO juju.environs.sync sync.go:171 downloaded tools/releases/juju-1.16.0.1-precise-amd64.tgz (4517kB), uploading
2013-10-20 23:02:29 INFO juju.environs.sync sync.go:172 download 4517kB, uploading
2013-10-20 23:02:29 INFO juju.environs.sync sync.go:118 copied 1 tools
2013-10-20 23:02:29 INFO juju.environs.sync sync.go:120 generating tools metadata
2013-10-20 23:02:29 INFO juju.environs.tools simplestreams.go:277 Writing tools/streams/v1/index.json
2013-10-20 23:02:29 INFO juju.environs.tools simplestreams.go:277 Writing tools/streams/v1/com.ubuntu.juju:released:tools.json
2013-10-20 23:02:29 INFO juju.environs.sync sync.go:128 tools metadata written
2013-10-20 23:02:30 INFO juju.environs.tools tools.go:85 reading tools with major.minor version 1.16
2013-10-20 23:02:30 INFO juju.environs.tools tools.go:93 filtering tools by version: 1.16.0.1
2013-10-20 23:02:30 INFO juju.environs.tools tools.go:96 filtering tools by series: precise
2013-10-20 23:02:30 INFO juju.environs.boostrap bootstrap.go:50 bootstrapping environment "mojo-staging-certification"
2013-10-20 23:02:30 INFO juju.environs.tools tools.go:85 reading tools with major.minor version 1.16
2013-10-20 23:02:30 INFO juju.environs.tools tools.go:93 filtering tools by version: 1.16.0.1
2013-10-20 23:02:30 INFO juju.environs.tools tools.go:96 filtering tools by series: precise
2013-10-20 23:02:31 ERROR juju supercommand.go:282 cannot start bootstrap instance: index file has no data for cloud {RegionOne http://gidim.internal:5000/v2.0/} not found

[Michaël Van de Borne (mikemowgli)]

Same problem for me. My openstack installation is Grizzly.

$ juju version
1.16.0-saucy-amd64
$ juju -v bootstrap
[...]
2013-10-30 14:23:17 INFO juju.environs.tools tools.go:181 filtering tools by released version
2013-10-30 14:23:17 INFO juju.environs.tools tools.go:85 reading tools with major.minor version 1.16
2013-10-30 14:23:17 INFO juju.environs.tools tools.go:96 filtering tools by series: precise
2013-10-30 14:23:17 INFO juju.environs.boostrap bootstrap.go:71 environs: picked newest version: 1.16.0
2013-10-30 14:23:18 ERROR juju supercommand.go:282 cannot start bootstrap instance: index file has no data for cloud {RegionOne http://192.168.202.103:5000/v2.0/} not found

The index.json file in the swift container looks like:
{
    "index": {
        "com.ubuntu.juju:released:tools": {
            "updated": "Wed, 30 Oct 2013 15:23:17 +0100",
            "format": "products:1.0",
            "datatype": "content-download",
            "path": "streams/v1/com.ubuntu.juju:released:tools.json",
            "products": [
                "com.ubuntu.juju:12.04:amd64",
                "com.ubuntu.juju:12.04:i386",
                "com.ubuntu.juju:12.10:amd64",
                "com.ubuntu.juju:12.10:i386",
                "com.ubuntu.juju:13.04:amd64",
                "com.ubuntu.juju:13.04:i386",
                "com.ubuntu.juju:13.10:amd64",
                "com.ubuntu.juju:13.10:i386"
            ]
        }
    },
    "updated": "Wed, 30 Oct 2013 15:23:17 +0100",
    "format": "index:1.0"
}

My openstack section in environments.yaml looks like:
<<
  openstack:
    type: openstack
    # Specifies whether the use of a floating IP address is required to give the nodes
    # a public IP address. Some installations assign public IP addresses by default without
    # requiring a floating IP address.
    use-floating-ip: true
    admin-secret: a8ab270e79d3dd330ca9d79a6d5505c4
    # Globally unique swift bucket name
    control-bucket: juju-64a08a56c0390b3124ac6edaa6d282e7
    # If set, tools-url specifies from where tools are fetched.
    # tools-url: https://you-tools-url
    # Usually set via the env variable OS_AUTH_URL, but can be specified here
    auth-url: http://192.168.202.103:5000/v2.0/
    # override if your workstation is running a different series to which you are deploying
    default-series: precise
    # The following are used for userpass authentication (the default)
    # auth-mode: userpass
    # Usually set via the env variable OS_USERNAME, but can be specified here
    username: UserA
    # Usually set via the env variable OS_PASSWORD, but can be specified here
    password: something_personal
    # Usually set via the env variable OS_TENANT_NAME, but can be specified here
    tenant-name: TenantA
    # Usually set via the env variable OS_REGION_NAME, but can be specified here
    region: RegionOne
    # USe the following if you require keypair autherntication
    # auth-mode: keypair
    # Usually set via the env variable OS_ACCESS_KEY, but can be specified here
    access-key: something_personal
    # Usually set via the env variable OS_SECRET_KEY, but can be specified here
    secret-key: something_personal
>>

Is that a b...

[Michaël Van de Borne (mikemowgli)]

Hi again,
I noticed several wierd lines in the log when I run the bootstrap command in debug mode:

2013-11-04 14:28:44 DEBUG juju.environs.simplestreams simplestreams.go:458 fetchData failed for "http://192.168.202.4:8080/v1/AUTH_93a48de7ef674f07a96e169383c34399/juju-64a08a56c0390b3124ac6edaa6d282e7/tools/streams/v1/index.sjson": failed to GET object tools/streams/v1/index.sjson from container juju-64a08a56c0390b3124ac6edaa6d282e7

--> in this line, the index.sjson file is called, rather than index.json

2013-11-04 14:28:45 DEBUG juju.environs.simplestreams simplestreams.go:458 fetchData failed for "http://192.168.202.4:8080/v1/AUTH_93a48de7ef674f07a96e169383c34399/juju-64a08a56c0390b3124ac6edaa6d282e7/streams/v1/index.sjson": failed to GET object streams/v1/index.sjson from container juju-64a08a56c0390b3124ac6edaa6d282e7

--> later on, in this line, the 'tools' directory is missing in the path (just before the 'streams' directory.

Note: I just updated to 1.16.2 and this error is still present.

yours,

m.

[Michaël Van de Borne (mikemowgli)]

I think that the "index file has no data for cloud ..." error is because juju cannot found any reference of the openstack private cloud in http://cloud-images.ubuntu.com/releases/streams/v1/index.json (which, I guess, is pretty normal, actually, given that the private openstack cloud shouldn't be referenced in this public cloud list).

And I think that juju looking for a reference in this index.json is a fallback because it couldn't find the local index.json in swift bucket, due to the path errors I mentionned above.

I really aren't a developer, but in my opinion, the line from file environs/simplestreams/simplestreams.go

DefaultIndexPath = "streams/v1/index"

should be replaced by

DefaultIndexPath = "tools/streams/v1/index"

or something like that.

This is kind of blocking for me at the moment. Can anyone confirm my guess?

Thank you.

michaël

[Ian Booth (wallyworld)]

If this is a private cloud with custom images, what needs to be available for bootstrap to succeed is simplestreams metadata describing what image id to use for a particular series/arch. Separate to that, additional simplestreams metadata is used to locate a tools tarball to use.

The upload-tools process generates the tools metadata and tools tarball and uploads to the Openstack cloud storage (aka control bucket). The various tools index metadata files included with this bug report are correct - they are a red herring. Index metadata file do not require any cloud records be defined - if there are none, then it is expected a simple product file be used and any cloud filtering will be done using the product data. If no tools can be found, an error would have occurred and bootstrap would have aborted.

The DefaultIndexPath is correct - "streams/v1/index". The search path includes "<control_bucket>/tools" so the index path is relative to the tools container.

The other "not found" log items refer to the fact that signed metadata files (ending with .sjson) cannot be found. This is to be expected because only official metadata produced by Canonical for certifified public clouds is currently signed.

The other metadata is index metadata and this is what the error is complaining about. Unless I have missed it, I can't see this image metadata index file attached to the bug report. I am assuming the image meatdata has been uploaded to the control bucket - like for tools, this is in the search path for locating image ids. In 1.16, the image metadata is placed in a folder "<control_bucket>/streams/v1". The next Juju release it will be "<control_bucket>/images/streams/v1"

My guess is that the image metadata is wrong. In Juju 1.16, the image metadata generation tool is still more of a developer prototype rather than an end user tool, but the steps are essentially:

1. juju metadata generate-image -i <imageid> -d <local_dir>
2. upload contents of local_dir to control bucket

See juju help metadata generate-image for more info.

Without additional params, generate-image takes region, series, endpoint, arch from the current environment. The region/endpoint form the cloud specification and this is what is placed in the image index file. This not matching is what the error in the bug report is about.

[Haw Loeung (hloeung)]

Hi,

Ian's advice above seems to have worked for 1.16.5 with some slight modifications:

1. juju metdata generate-image in 1.16.5 doesn't understand '-d' and puts them in the $JUJU_HOME environment variable.
2. After uploading these files to a new control bucket, you'd have to add the correct read ACLs. For existing environments, you'd have to remove bootstrap-verify and provider-state

Thanks,

Haw

[Robert Brockway (brockwar)]

On Mon, 10 Feb 2014, Haw Loeung wrote:

> Hi,
>
> Ian's advice above seems to have worked for 1.16.5 with some slight
> modifications:
>
> 1. juju metdata generate-image in 1.16.5 doesn't understand '-d' and puts them in the $JUJU_HOME environment variable.
> 2. After uploading these files to a new control bucket, you'd have to add the correct read ACLs. For existing environments, you'd have to remove bootstrap-verify and provider-state

Thanks Haw. I'll take a look in the morning.

Cheers,

Rob

[Ian Booth (wallyworld)]

Ah, yes sorry. When this came up on IRC I forgot that the generate-image stuff in 1.16 was quite limited.
In 1.17 onwards it has improved a lot. And as of 1.17.2, the image metadata is automatically uploaded so the manual upload steps are no longer required.

[Dave Johnston (dave-johnston)]

I'm also getting this issue with a private Openstack cloud.

I've followed the steps:

1. juju metadata generate-image -i <imageid> -d <local_dir>
2. upload contents of local_dir to control bucket

As mentioned above the '-d' directive doesn't work, but the files do get generated.
After uploading them I delete bootstrap-verify and provider-state from the bucket,

But when I try to re-run 'juju bootstrap' I get the same error.
It looks like the index file is overwritten.

2014-02-16 21:01:26 INFO juju.environs.tools simplestreams.go:277 Writing tools/streams/v1/index.json

How can I replace the index.json file, and get the bootstrap to complete ?

Cheers

[Ian Booth (wallyworld)]

The log message above refers to tools metadata not image metadata. So it is not relevant here.
For Juju 1.16, the procedure should be to upload the image metadata to the control container/bucket. That is, streams/v1/*.json gets uploaded. All json files are required, not just the index file.

[Dave Johnston (dave-johnston)]

Still not getting any further.
I run
juju metadata generate-image -i 65f4ecf9-d63e-4af2-b534-736ae4ee08ee -r "RegionOne" -u http://10.20.70.82:35357/v2.0/

Which generates index.json and imagemetadata.json

Both get uploaded to streams/v1/*.json

AFter rerunning bootstramp I see:
2014-02-18 20:10:08 ERROR juju supercommand.go:282 cannot start bootstrap instance: index file has no data for cloud {RegionOne http://10.20.70.82:35357/v2.0} not found

{
 "index": {
   "com.ubuntu.cloud:custom": {
     "updated": "Sun, 16 Feb 2014 20:30:23 +0000",
     "clouds": [
       {
         "region": "RegionOne",
         "endpoint": "http://10.20.70.82:35357/v2.0/"
       }
     ],
     "cloudname": "custom",
     "datatype": "image-ids",
     "format": "products:1.0",
     "products": [
       "com.ubuntu.cloud:server:12.04:amd64"
     ],
     "path": "streams/v1/imagemetadata.json"
   }
 },
 "updated": "Sun, 16 Feb 2014 20:30:23 +0000",
 "format": "index:1.0"
}

From the error message it does look like it find index.json, but that it's not finding a match for the specified cloud. Any ideas whats wrong with this setup?

[Ian Booth (wallyworld)]

I suspect, but am not 100% sure, that in Juju 1.16 the endpoint matching was sensitive to trailing "/". So the json metadata contains an endpoint with a training "/" but the configured auth url doesn't. Try adding a "/" to the auth url in the environment config, or remove the "/" from the json metadata and see if that helps. If it doesn't work, we'll need to dig a bit deeper.

[Dave Johnston (dave-johnston)]

Hi Ian,

That's worked. I removed the slash, and it finds the metadata and attempts to launch the machine. I still get a failure as it's trying to use the first Openstack flavour, which doesn't have enough memory for the ubuntu image.

I just need ot see how I can specify the flavour in the imagemetadata.json and that should get me bootstrapped.

Thanks

[Dave Johnston (dave-johnston)]

I managed to work around specifying the flavour by setting constraints during bootstrap:

juju bootstrap -v --constraints "mem=2048 root-disk=20480root-disk"

This seems to ensure the correct flavour is picked. Unfortunatley I don't get an further as my Openstack cloud has multiple networks available, so I get the error:

caused by: request (http://10.20.70.82:8774/v2/f7099e3a24db4a01835433dca97834c8/servers) returned unexpected status: 400; error info: {"badRequest": {"message": "Multiple possible networks found, use a Network ID to be more specific.", "code": 400}}

I'm not sure this parameter can be specified with a constraint

[Ian Booth (wallyworld)]

Hey Dave

Glad you made progress and sorry you ran into those issues. The fix to pick a better default flavour (one with 1GB RAM) on Openstack, as well as support for Openstack with multiple networks have already been committed and will be included in the next Juju release, 1.18. We hope to have that out within a week or so (ASAP). 1.18 also fixes usability issues with custom image metadata and makes the process much easier.

[Anastasia (anastasia-macmood)]

Originally this bug report was talking about configuring simplestreams images for private cloud which has been sorted according to comment #12.

Then the bug morphed to figuring out how to get a specific, desired openstack flavor and using constraints resolves this aspect according comment #13.

I think that this bug is fixed.