HP Bootstrap fails: no endpoints known for service type: product-streams

[Curtis Hovey]

http://juju-ci.vapour.ws:8080/job/hp-deploy-trusty-amd64/1713/console is an example where juju 1.21-alpha2 cannot bootstrap in Hp Cloud. No test that tries to do this succeeds. Tests that bootstrap with stable and upgrade are fine. The test was run from the trusty-slave in Hp Cloud and from the Jenkins master in AWS.

jenkins@trusty-slave:~/workspace/hp-deploy-trusty-amd64$ juju --debug bootstrap -e test-release-hp 2>&1 | tee ~/missing-product-bootstrap.log
2014-09-19 13:38:15 INFO juju.cmd supercommand.go:37 running juju [1.21-alpha2-trusty-amd64 gc]
2014-09-19 13:38:15 INFO juju.provider.openstack provider.go:231 opening environment "test-release-hp"
2014-09-19 13:38:16 DEBUG juju.environs.configstore disk.go:330 writing jenv file to /var/lib/jenkins/cloud-city/environments/test-release-hp.jenv
2014-09-19 13:38:16 INFO juju.network network.go:102 setting prefer-ipv6 to false
2014-09-19 13:38:16 DEBUG juju.environs imagemetadata.go:103 trying datasource "keystone catalog"
2014-09-19 13:38:17 DEBUG juju.cmd.juju common.go:89 Destroying environment.
2014-09-19 13:38:17 INFO juju.cmd cmd.go:113 Bootstrap failed, destroying environment
2014-09-19 13:38:17 INFO juju.provider.common destroy.go:15 destroying environment "test-release-hp"
2014-09-19 13:38:17 DEBUG juju.provider.openstack provider.go:1082 terminating instances []
2014-09-19 13:38:20 ERROR juju.cmd supercommand.go:323 failed to bootstrap environment: no endpoints known for service type: product-streams

[Curtis Hovey (sinzui)]

These are the suspected commits that broken bootstrap in HP
commit e718691 Merge pull request #774 from axw/image-metadata-envstorage
commit 5d6082e Merge pull request #795 from mjs/ssh-fails-during-upgrade

[Menno Finlay-Smits (menno.smits)]

Given the problem symptoms, and the contents of 5d6082e, it is HIGHLY unlikely that 5d6082e is involved here.

[Ian Booth (wallyworld)]

This is an issue with HP Cloud itself. The error message indicates that the keystone catalog is missing the product-streams url used for searching simplestreams metadata. Someone must have deleted this configuration.

This is what it should look like - here's the output from Canonistack:

$ keystone catalog

<snip>
Service: product-streams
+-------------+------------------------------------------------------------------------------------------------------+
| Property | Value |
+-------------+------------------------------------------------------------------------------------------------------+
| adminURL | https://swift.canonistack.canonical.com/v1/AUTH_a48765cc0e864be980ee21ae26aaaed4/simplestreams/data/ |
| id | 45649fbaa9de4dd68c0ff696eda1802e |
| internalURL | https://swift.canonistack.canonical.com/v1/AUTH_a48765cc0e864be980ee21ae26aaaed4/simplestreams/data/ |
| publicURL | https://swift.canonistack.canonical.com/v1/AUTH_a48765cc0e864be980ee21ae26aaaed4/simplestreams/data/ |
| region | lcy02 |
+-------------+------------------------------------------------------------------------------------------------------+
<snip>

[Ian Booth (wallyworld)]

I've marked as Invalid. Please re-open if you think there's a Juju issue to address.

[Ian Booth (wallyworld)]

Ooops, it really is a regression, sorry

[Andrew Wilkins (axwalk)]

This is related to a change I made last week to the metadata lookup. Lookup should not fail if there is no URL defined in Keystone, which is what is happening now to cause the regression. Fix is in the works.