Juju fails to bootstrap on OpenStack

[Noemi Aloisi]

I wan't create o juju controller on Openstack Cluod Canonical Queen version.
Error received:
- ERROR juju.cmd.juju.commands bootstrap.go:697 failed to get list of flavour details
caused by: parse https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2/: invalid character " " in host name

I have create a project and one VM and I have followed this documentation:
- https://jaas.ai/docs/creating-a-controller
- https://cloud.garr.it/apps/juju-openstack/

Following I give you all my step:

cat acilia_cloud.yaml

clouds:
  acilia:
    type: openstack
    auth-types: [access-key, userpass]
    regions:
      RegionTwo:
        endpoint: XXXXXXXXX

/opt/noemi# cat credentials.yaml
credentials:
  acilia:
    juju-canonicalk8s:
      auth-type: userpass
      project-domain-name: default
      user-domain-name: default
      tenant-name: canonicalk8s
      username: XXXXXXXXX
      password: XXXXXXXXX

$ juju add-cloud acilia acilia_cloud.yaml
$ juju add-credential acilia -f credentials.yaml --replace
$ juju show-cloud acilia
$ juju list-credentials --format yaml --show-secrets

$ juju metadata generate-image -d /opt/noemi/simplestreams -i 8ad030a0-2fd4-444a-be48-4b6b46522bee -s bionic -r RegionTwo -u https://authr2.cs1.cloudspc.it:5000/v3

$ juju bootstrap acilia canonicalk8s --config network=03db2881-b30b-4a61-833f-1ba25b26162c --config external-network=b01c4b9e-26b8-41b0-a12f-2f5fdab84e70 --config use-floating-ip=true --metadata-source /opt/noemi/simplestreams --debug

b01c4b9e-26b8-41b0-a12f-2f5fdab84e70 --config use-floating-ip=true --metadata-source /opt/noemi/simplestreams --debug
10:15:07 INFO juju.cmd supercommand.go:57 running juju [2.6.9 gc go1.10.4]
10:15:07 INFO cmd authkeys.go:114 Adding contents of "/root/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
....
10:15:07 INFO juju.provider.openstack provider.go:162 opening model "controller"
.....
10:15:08 ERROR juju.cmd.juju.commands bootstrap.go:697 failed to get list of flavour details
caused by: parse https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2/: invalid character " " in host name
10:15:08 DEBUG juju.cmd.juju.commands bootstrap.go:698 (error details: [{/build/juju-core-sU8dy9/juju-core-2.6.9/obj-x86_64-linux-gnu/src/github.com/juju/juju/cmd/juju/commands/bootstrap.go:731: } {failed to get list of flavour details
caused by: parse https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2/: invalid character " " in host name}])
10:15:08 DEBUG juju.cmd.juju.commands bootstrap.go:1332 cleaning up after failed bootstrap
10:15:08 INFO juju.provider.common destroy.go:21 destroying model "controller"
10:15:08 INFO juju.provider.common destroy.go:32 destroying instances
10:15:08 ERROR juju.cmd.juju.commands bootstrap.go:1334 error cleaning up: destroying controller model: destroying instances: failed to get list of server details
caused by: parse https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2/: invalid character " " in host name
10:15:08 INFO cmd supercommand.go:502 command finished

[Richard Harding (rharding)]

Thanks for the heads up on the issue. It looks like Juju is running across a space in trying to retrieve the flavor details. Somehow it's built the URL to get the file from as

"https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2/" and it's not expecting to have the second bit of data there?

I'pm not sure if this is in the actual image metadata bit or in the config somewhere. Are you seeing those values in your setup?

[Noemi Aloisi (naloisi)]

Hi Richard,
Thanks for your quicly responce.

I have also tried to create juju bootstrap using following command:

$ juju bootstrap acilia canonicalk8s --config network=03db2881-b30b-4a61-833f-1ba25b26162c --config external-network=b01c4b9e-26b8-41b0-a12f-2f5fdab84e70 --config use-floating-ip=true --metadata-source /opt/noemi/simplestreams --bootstrap-constraints instance-type=Medium --debug

but I receive same issue.

There was another way to give flavor detail on image?

[Noemi Aloisi (naloisi)]

Any update?

I have also tried giving allurl path of flavor

instance-type=https://computer2.cs1.cloudspc.it:8774/e9add44d475740ca8661f3c2213d3470/flavors/6a29cd24-ca4c-40cf-8e57-8c83dc1844f0

but problem persist and log don't change

[Heather Lanigan (hmlanigan)]

@naloisi,

This sounds like another issue that came thru, can you please provide a full juju bootstrap --debug output? I'm specifically interested in the line containing "DEBUG goose"

I suspect it's related to the data juju receives from the compute service when requesting a list of versions.

Thank you,
Heather

[James Page (james-page)]

Reference data:

{"versions": [
   {"status": "SUPPORTED", "updated": "2011-01-21T11:33:21Z", "links": [{"href": "http://10.245.161.158:8774/v2/", "rel": "self"}], "min_version": "", "version": "", "id": "v2.0"},

   {"status": "CURRENT", "updated": "2013-07-23T11:33:21Z", "links": [{"href": "http://10.245.161.158:8774/v2.1/", "rel": "self"}], "min_version": "2.1", "version": "2.60", "id": "v2.1"}

]}

[James Page (james-page)]

Is the [api] compute_link_prefix configuration option in use in this OpenStack deployment?

[James Page (james-page)]

(for context that would be in /etc/nova/nova.conf on a nova-cloud-controller unit in a juju deployment, or the controller node in a non-juju deployment).

[Noemi Aloisi (naloisi)]

Hi Heather Lanigan,

Full log was this:

17:13:38 INFO juju.cmd supercommand.go:57 running juju [2.6.6 gc go1.10.4]
17:13:38 DEBUG juju.cmd supercommand.go:58 args: []string{"juju", "bootstrap", "acilia", "canonicalk8s", "--config", "config.yaml", "--bootstrap-series=bionic", "--debug"}
17:13:38 DEBUG juju.cmd.juju.commands bootstrap.go:1038 authenticating with region "" and credential "juju-canonicalk8s" ()
17:13:38 DEBUG juju.cmd.juju.commands bootstrap.go:1166 provider attrs: map[use-default-secgroup:false network:default external-network:Internet use-openstack-gbp:false policy-target-group: use-floating-ip:true]
17:13:39 INFO cmd authkeys.go:114 Adding contents of "/home/gverona/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
17:13:39 DEBUG juju.cmd.juju.commands bootstrap.go:1225 preparing controller with config: map[logging-config: network:default image-stream:released max-action-results-age:336h enable-os-refresh-update:true authorized-keys:ssh-rsa juju-client-key
 net-bond-reconfigure-delay:17 ftp-proxy: juju-ftp-proxy: juju-no-proxy:127.0.0.1,localhost,::1 agent-metadata-url: no-proxy:127.0.0.1,localhost,::1 https-proxy: update-status-hook-interval:5m external-network:Internet ssl-hostname-verification:true use-floating-ip:true resource-tags: apt-ftp-proxy: use-default-secgroup:false use-openstack-gbp:false test-mode:false container-inherit-properties: agent-stream:released image-metadata-url: snap-store-proxy: provisioner-harvest-mode:destroyed max-status-history-age:336h http-proxy: egress-subnets: development:false max-status-history-size:5G apt-mirror: snap-store-assertions: automatically-retry-hooks:true name:controller logforward-enabled:false transmit-vendor-metrics:true disable-network-management:false container-image-metadata-url: juju-https-proxy: uuid:b278ecd6-a957-4618-8c24-88d8a1340499 container-image-stream:released fan-config: container-networking-method: cloudinit-userdata: ignore-machine-addresses:false enable-os-upgrade:true proxy-ssh:false snap-https-proxy: backup-dir: max-action-results-size:5G default-series:bionic policy-target-group: type:openstack apt-https-proxy: firewall-mode:instance apt-no-proxy: snap-http-proxy: apt-http-proxy: juju-http-proxy:]
17:13:39 INFO juju.provider.openstack provider.go:162 opening model "controller"
17:13:40 DEBUG goose client.go:547 auth details: &{Token:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TenantId:e9add44d475740ca8661f3c2213d3470 TenantName:canonicalk8s UserId:f29c29d55dd3450dae6359b182605e0b Domain: RegionServiceURLs:map[RegionTwo:map[datamover:http://10.135.1.15:8784/v2 compute:https://computer2.cs1.cloudspc.it:8774/v2.1 alarming:https://apir2.cs1.cloudspc.it:8042 image:https://apir2.cs1.cloudspc.it:9292 s3:https://apir2.cs1.cloudspc.it:8080 volumev2:https://apir2.cs1.cloudspc.it:8776/v2/e9add44d475740ca8661f3c2213d3470 identity:https://authr2.cs1.cloudspc.it:5000/v3 workloads:http://10.134.0.30:8780/v1/e9add44d475740ca8661f3c2213d3470 metric:https://apir2.cs1.cloudspc.it:8041 object-store:https://apir2.cs1.cloudspc.it:8080/v1/AUTH_e9add44d475740ca8661f3c2213d3470 cloudformation:https://apir2.cs1.cloudspc.it:8000/v1 placement:https://computer2.cs1.cloudspc.it...

[Noemi Aloisi (naloisi)]

Hi James Page,

We have 3 node of nova and into file I don't found compute_link_prefix.

I don't khow if can help us but we have 3 url of nova

openstack endpoint list | grep 8774
| 31b23cb3b1264b7aa346797160392eb7 | RegionTwo | nova | compute | True | admin | https://computer2-adm.cs1.cloudspc.it:8774/v2.1 |
| 3c051391361f4972861c5dc29ba7d729 | RegionTwo | nova | compute | True | public | https://computer2.cs1.cloudspc.it:8774/v2.1 |
| a3a71904a73647a586112ec0acb8f95f | RegionTwo | nova | compute | True | internal | https://computer2-int.cs1.cloudspc.it:8774/v2.1

and we need use only public url.

[James Page (james-page)]

@naloisi

Could you provide the output of:

  curl https://computer2.cs1.cloudspc.it:8774/

Thanks

[James Page (james-page)]

Also do you have some sort of proxy/loadbalancer infront of the deployment?

[James Page (james-page)]

This might be due to mis-processing of the headers added by the proxy but its hard to tell.

The following configuration option in oslo_middleware is set to false by default - enabling this may help

[oslo_middleware]
# Whether the application is behind a proxy or not. This determines if the
# middleware should parse the headers or not (boolean value)
#enable_proxy_headers_parsing = false

[James Page (james-page)]

That said I think we enable that by default for charm based deployments, because services are always fronted by haproxy and apache2

[Noemi Aloisi (naloisi)]

Hi James Page,

We have a proxy on top of Openstack.

I have executed curl from VM that Iwant use to create juju bootstrap and I receive:
           $curl https://computer2.cs1.cloudspc.it:8774/
{"versions": [{"status": "SUPPORTED", "updated": "2011-01-21T11:33:21Z", "links": [{"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2/", "rel": "self"}], "min_version": "", "version": "", "id": "v2.0"}, {"status": "CURRENT", "updated": "2013-07-23T11:33:21Z", "links": [{"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2.1/", "rel": "self"}], "min_version": "2.1", "version": "2.60", "id": "v2.1"}]}

I have also tried to call flavor list and I don't receive any error:

$ n_ep="`openstack endpoint list -f value -c URL -c 'Service Name' --interface public| grep nova| cut -f 2 -d ' '`"

$ echo $n_ep
https://computer2.cs1.cloudspc.it:8774/v2.1

$ token=`openstack token issue -c id -f value`

$ curl -H "X-Auth-Token: $token" $n_ep/flavors
{"flavors": [{"id": "0b1ab4ad-f4c6-4384-9a54-a6c82b0f5538", "links": [{"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2.1/flavors/0b1ab4ad-f4c6-4384-9a54-a6c82b0f5538", "rel": "self"}, {"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/flavors/0b1ab4ad-f4c6-4384-9a54-a6c82b0f5538", "rel": "bookmark"}], "name": "Small"}, {"id": "1ce2c4f0-c026-497d-96ed-fe5f925778ac", "links": [{"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2.1/flavors/1ce2c4f0-c026-497d-96ed-fe5f925778ac", "rel": "self"}, {"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/flavors/1ce2c4f0-c026-497d-96ed-fe5f925778ac", "rel": "bookmark"}], "name": "Large"}, {"id": "541f8b78-c4fe-4e21-ad92-b7a09fe53737", "links": [{"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2.1/flavors/541f8b78-c4fe-4e21-ad92-b7a09fe53737", "rel": "self"}, {"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/flavors/541f8b78-c4fe-4e21-ad92-b7a09fe53737", "rel": "bookmark"}], "name": "X-Medium"}, {"id": "66f0d29b-e507-4dea-8447-eb85131a6c44", "links": [{"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2.1/flavors/66f0d29b-e507-4dea-8447-eb85131a6c44", "rel": "self"}, {"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/flavors/66f0d29b-e507-4dea-8447-eb85131a6c44", "rel": "bookmark"}], "name": "X-Small"}, {"id": "6a29cd24-ca4c-40cf-8e57-8c83dc1844f0", "links": [{"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2.1/flavors/6a29cd24-ca4c-40cf-8e57-8c83dc1844f0", "rel": "self"}, {"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/flavors/6a29cd24-ca4c-40cf-8e57-8c83dc1844f0", "rel": "bookmark"}], "name": "Medium"}, {"id": "ece65898-f63d-46e4-9ba0-62d61fce9d20", "links": [{"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/v2.1/flavors/ece65898-f63d-46e4-9ba0-62d61fce9d20", "rel": "self"}, {"href": "https://computer2.cs1.cloudspc.it:8774, computer2.cs1.cloudspc.it:8774/flavors/ece65898-f63d-46e...

[James Page (james-page)]

Can you also direct curl the backend API service, bypassing the proxy? I'm trying to determine what impact the proxy may/may not be having on the response provided by the API.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.