Unable to bootstrap with --metadata-source

[Nicolas Thomas]

Context:

From a "jumpserver" on a private openstack. I validate nova list etc.. works..

For 2.0 :

Tried:
$juju bootstrap openstack openstack --metadata-source=/home/ubuntu/ --config network=private --upload-tools --debug -v
2016-06-11 10:52:44 INFO juju.cmd supercommand.go:60 running juju [2.0-beta7 gc go1.6.1]
2016-06-11 10:52:44 DEBUG juju.cmd.juju.commands bootstrap.go:354 preparing controller with config: map[controller-uuid:d3e4dea8-83e8-41ff-8c7b-4911fe6bdce6 network:private type:openstack name:admin uuid:d3e4dea8-83e8-41ff-8c7b-4911fe6bdce6]
2016-06-11 10:52:44 INFO juju.provider.openstack provider.go:75 opening model "admin"
2016-06-11 10:52:44 DEBUG juju.provider.openstack provider.go:725 authURL: http://172.27.60.36:35357/v2.0/
2016-06-11 10:52:44 INFO cmd cmd.go:129 Creating Juju controller "local.openstack" on openstack/RegionOne
2016-06-11 10:52:44 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "default cloud images"
2016-06-11 10:52:44 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "default ubuntu cloud images"
2016-06-11 10:52:45 DEBUG juju.environs.simplestreams simplestreams.go:454 skipping index "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson" because of missing information: index file has no data for cloud {RegionOne http://172.27.60.36:35357/v2.0/} not found
2016-06-11 10:52:45 INFO juju.provider.common destroy.go:22 destroying model "admin"
2016-06-11 10:52:45 INFO juju.provider.common destroy.go:33 destroying instances
2016-06-11 10:52:45 DEBUG juju.provider.openstack provider.go:1080 terminating instances []
2016-06-11 10:52:45 INFO juju.provider.common destroy.go:53 destroying storage
2016-06-11 10:52:45 DEBUG juju.provider.openstack provider.go:725 authURL: http://172.27.60.36:35357/v2.0/
2016-06-11 10:52:46 DEBUG juju.provider.openstack provider.go:725 authURL: http://172.27.60.36:35357/v2.0/
2016-06-11 10:52:47 ERROR cmd supercommand.go:448 index file has no data for cloud {RegionOne http://172.27.60.36:35357/v2.0/} not found

I notice the same behavior with 1.25 ..

[Cheryl Jennings (cherylj)]

This looks a lot like bug #1452422 which claims to have been fixed already. Can you try to work around it by specifying image-metadata-url? (See comment #4 in that bug)

[Nicolas Thomas (thomnico)]

From discussion on irc it looks like --metadata-source= is not a valid option anymore and one MUST set-up its own web-server (not a pleasant first time experience) to have it working.

If this is the case please consider this bug as "remove deprecated option --metadata-source= if not used).

Thanks

[Cheryl Jennings (cherylj)]

Nicolas - who did you speak with to determine that --metadata-source is not a valid option anymore?

In talking with other core folks, I'm not sure that it should be deprecated / removed.

[Nicolas Thomas (thomnico)]

Hi Cheryl,

Can't remember and does not really matter..

I did a script to automate the process and use swift here
http://bazaar.launchpad.net/~thomnico/sademos/juju-on-openstack-resumed/view/head:/openstack/configure-juju-on-openstack

Which is much better anyway.

If not deprecated then it is totally broken ... this part is your call :)

On Thu, Jun 30, 2016 at 10:17 AM, Cheryl Jennings
<email address hidden> wrote:
> Nicolas - who did you speak with to determine that --metadata-source is
> not a valid option anymore?
>
> In talking with other core folks, I'm not sure that it should be
> deprecated / removed.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1591488
>
> Title:
> remove deprecated option --metadata-source
>
> Status in juju-core:
> Triaged
>
> Bug description:
> Context:
>
> From a "jumpserver" on a private openstack. I validate nova list etc..
> works..
>
> For 2.0 :
>
> Tried:
> $juju bootstrap openstack openstack --metadata-source=/home/ubuntu/ --config network=private --upload-tools --debug -v
> 2016-06-11 10:52:44 INFO juju.cmd supercommand.go:60 running juju [2.0-beta7 gc go1.6.1]
> 2016-06-11 10:52:44 DEBUG juju.cmd.juju.commands bootstrap.go:354 preparing controller with config: map[controller-uuid:d3e4dea8-83e8-41ff-8c7b-4911fe6bdce6 network:private type:openstack name:admin uuid:d3e4dea8-83e8-41ff-8c7b-4911fe6bdce6]
> 2016-06-11 10:52:44 INFO juju.provider.openstack provider.go:75 opening model "admin"
> 2016-06-11 10:52:44 DEBUG juju.provider.openstack provider.go:725 authURL: http://172.27.60.36:35357/v2.0/
> 2016-06-11 10:52:44 INFO cmd cmd.go:129 Creating Juju controller "local.openstack" on openstack/RegionOne
> 2016-06-11 10:52:44 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "default cloud images"
> 2016-06-11 10:52:44 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "default ubuntu cloud images"
> 2016-06-11 10:52:45 DEBUG juju.environs.simplestreams simplestreams.go:454 skipping index "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson" because of missing information: index file has no data for cloud {RegionOne http://172.27.60.36:35357/v2.0/} not found
> 2016-06-11 10:52:45 INFO juju.provider.common destroy.go:22 destroying model "admin"
> 2016-06-11 10:52:45 INFO juju.provider.common destroy.go:33 destroying instances
> 2016-06-11 10:52:45 DEBUG juju.provider.openstack provider.go:1080 terminating instances []
> 2016-06-11 10:52:45 INFO juju.provider.common destroy.go:53 destroying storage
> 2016-06-11 10:52:45 DEBUG juju.provider.openstack provider.go:725 authURL: http://172.27.60.36:35357/v2.0/
> 2016-06-11 10:52:46 DEBUG juju.provider.openstack provider.go:725 authURL: http://172.27.60.36:35357/v2.0/
> 2016-06-11 10:52:47 ERROR cmd supercommand.go:448 index file has no data for cloud {RegionOne http://172.27.60.36:35357/v2.0/} not found
>
>
> I notice the same behavior with 1.25 ..
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1591488/+subscriptions

--
Best Regards,
   ...

[Cheryl Jennings (cherylj)]

Yeah, the thought is that we need to fix this, or provide some other way to get this behavior.

[Jonathan Gonzalez V. (jonathan-abdiel)]

Hi,

I ran into the same bug today and I solved using this in the bootstrap line

--config image-metadata-url=file:///home/zeus/meta-data/images/

And now it works! =)

[Jonathan Gonzalez V. (jonathan-abdiel)]

Hi again,

I got the same problem and the bootstrap fails after get mongo ready, it's like it just forgot the image-metadata-url value and again it took the old one :/, here it's the log:

2016-07-08 23:32:56 INFO juju.state state.go:268 starting singular lease manager
2016-07-08 23:32:56 INFO juju.state state.go:280 creating cloud image metadata storage
2016-07-08 23:32:56 INFO juju.state state.go:283 starting presence watcher
2016-07-08 23:32:56 DEBUG juju.agent.agentbootstrap bootstrap.go:100 connected to initial state
2016-07-08 23:32:56 DEBUG juju.network network.go:353 no lxc bridge addresses to filter for machine
2016-07-08 23:32:56 WARNING juju.network address.go:279 no spaces to select addresses from
2016-07-08 23:32:56 DEBUG juju.worker.lease manager.go:217 waking to check leases at 2016-07-08 23:33:56.405296125 +0000 UTC
2016-07-08 23:32:56 DEBUG juju.state address.go:137 setting API hostPorts: [[138.219.230.208:17070 10.10.30.26:17070]]
2016-07-08 23:32:56 INFO juju.provider.openstack provider.go:75 opening model "admin"
2016-07-08 23:32:56 DEBUG juju.provider.openstack provider.go:725 authURL: https://cloud.beebop.sh:5000/v2.0
2016-07-08 23:32:56 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "image-metadata-url"
2016-07-08 23:32:56 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "default cloud images"
2016-07-08 23:32:56 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "default ubuntu cloud images"
2016-07-08 23:32:59 DEBUG juju.environs.simplestreams simplestreams.go:454 skipping index "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson" because of missing information: index file has no data for cloud {Bombus https://cloud.beebop.sh:5000/v2.0} not found
2016-07-08 23:32:59 DEBUG juju.state open.go:333 closed state without error
2016-07-08 23:32:59 ERROR cmd supercommand.go:448 cannot set initial model constraints: index file has no data for cloud {Bombus https://cloud.beebop.sh:5000/v2.0} not found
2016-07-08 23:32:59 INFO juju.provider.common destroy.go:22 destroying model "admin"
2016-07-08 23:32:59 INFO juju.provider.common destroy.go:33 destroying instances
2016-07-08 23:33:00 DEBUG juju.provider.openstack provider.go:1080 terminating instances [8f86d45c-3959-4e29-a51e-ca5d86fc2022]
2016-07-08 23:33:01 DEBUG juju.provider.openstack firewaller.go:269 deleting security group "juju-7768e102-2362-490f-8c24-829cb586546a-7768e102-2362-490f-8c24-829cb586546a-0"
2016-07-08 23:33:02 INFO juju.provider.common destroy.go:53 destroying storage
2016-07-08 23:33:02 DEBUG juju.provider.openstack provider.go:725 authURL: https://cloud.beebop.sh:5000/v2.0
2016-07-08 23:33:02 DEBUG juju.provider.openstack provider.go:725 authURL: https://cloud.beebop.sh:5000/v2.0
2016-07-08 23:33:04 DEBUG juju.provider.openstack firewaller.go:269 deleting security group "juju-7768e102-2362-490f-8c24-829cb586546a-7768e102-2362-490f-8c24-829cb586546a"
2016-07-08 23:33:04 ERROR cmd supercommand.go:448 failed to bootstrap model: subprocess encountered error code 1

[Anastasia (anastasia-macmood)]

PR against master: https://github.com/juju/juju/pull/5903

[Anastasia (anastasia-macmood)]

Fix for this bug will also address this one: https://bugs.launchpad.net/juju-core/1.25/+bug/1567747
It will need to be updated.

[Anastasia (anastasia-macmood)]

PR previously mentioned is obsolete. New PR: https://github.com/juju/juju/pull/6023

[Anastasia (anastasia-macmood)]

You can now bootstrap with --metadata-source.

However, the deploy also needs to be fixed to pick up the images. Currently, when trying to deploy to controller or default model, I get image not found error. I suspect the same will happen if I were to add a new model and try to deploy to it.

I am investigating this further since we are failing to either store metadata from --metadata-source or retrieve it when starting a new machine.

[Anastasia (anastasia-macmood)]

This PR enabled bootstrap: https://github.com/juju/juju/pull/6023

[Anastasia (anastasia-macmood)]

I have isolated this \o/
The problem is that on some occasions, we save metadata without going through apiserver layer but by calling persistence (state layer) directly. The apiserver layer will populate some defaults if values are missing, for example, empty "stream" will contain "released".

In other words, now we end up with some data in database that we cannot easily retrieve using our filters. Our filters are pre-populated with sensible defaults when we look for available image metadata.

Related pieces of code in both provisioner and bootstrap need to be modified to either go through apiserver or to ensure data contains our standard defaults before persisting.

[Anastasia (anastasia-macmood)]

PR that ensures that we can deploy with image metadata when bootstrapping with custom images:
https://github.com/juju/juju/pull/6060

This should be the final fix for this bug \o/

[Anastasia (anastasia-macmood)]

Whilst this bug is fixed, related bug https://bugs.launchpad.net/juju/+bug/1616305 needs to be addressed as well to enable deployment to other, non-"controller" models.
Together these fixes will provide ability to bootstrap and deploy to private clouds as well as in the environments with limited/no internet connectivity.