Azure provider: Juju unable to locate any image when "image-stream: released" is set (doesn't validate content of image-stream)

[Antonio Rosales]

Using:
juju --version
1.13.3-quantal-amd64

and setting the following for my azure config in environments.yaml:
    image-stream: released
    default-series: precise

From what I can tell Juju is parsing the released simple stream data incorrectly:

2013-08-31 00:07:55 DEBUG juju.environs.simplestreams simplestreams.go:356 read metadata index at "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson"
2013-08-31 00:07:55 DEBUG juju.environs.simplestreams simplestreams.go:360 skipping index because of error getting latest metadata "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson": index file has no data for product name(s) ["com.ubuntu.cloud.released:server:12.04:amd64" "com.ubuntu.cloud.released:server:12.04:i386"] not found
2013-08-31 00:08:12 ERROR juju supercommand.go:282 command failed: cannot start bootstrap instance: no OS images found for location "West US", series "precise", architectures ["amd64" "i386"] (and endpoint: "https://management.core.windows.net/")

Juju is looking in:
    http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson
for
    "com.ubuntu.cloud.released:server:12.04:amd64" or "com.ubuntu.cloud.released:server:12.04:i386"

Juju should be looking for:
    com.ubuntu.cloud:server:12.04:amd64" or "com.ubuntu.cloud:server:12.04:i386"

Note the com.ubuntu.com.cloud syntax is different for daily and released simple streams
  daily = com.ubuntu.cloud.daily:server
  released = com.ubuntu.cloud:server

This is why daily streams bootstrap:
    image-stream: released
    default-series: precise

I think Juju just needs to update is parsing to drop the ".released" in the com.ubuntu.com simple stream parsing.

[Antonio Rosales (arosales)]

It is important to use "released" streams as there is no warranty or QA on daily images.

[John A Meinel (jameinel)]

Reading the code here:
http://bazaar.launchpad.net/~go-bot/juju-core/trunk/view/head:/provider/azure/config.go#L34

It would appear that the default image-stream is "". Which would be the "released" stream, rather than the string "released".

In bug #1218329 you mention that you tried "image-stream: " but I wonder if what you actually need is to either just delete the entry (or comment it out) or set it to:

image-stream: ""

Which is probably actually different from:

image-stream:

[John A Meinel (jameinel)]

I'm trying to sort out what we should be doing. AFAIK Azure is the only place where we expose "image-stream". All the other providers only ever work with released data. Only Azure exposed the dailies because there were no official released ones.

Are we planning on allowing non-released to be used in the future? (again, we don't provide a way to access those streams on any other cloud that I can tell).

So *if*: image-stream: "" works as a workaround for this, I'm tempted to just deprecate the configuration entirely.

If we really do need to keep this then I would certainly consider that we should validate the contents of "image-stream" (right now we just accept any string that you supply and put it into the search as ".$STREAM".

[Dave Cheney (dave-cheney)]

Please do not change the milestone, if required we can backport this to the 1.14 stable series.

[Andrew Wilkins (axwalk)]

I have tested `image-stream: ""`, and it works.

[Antonio Rosales (arosales)]

Testing:
    # image-stream: released
    default-series: precise
also is able to bootstrap.

It seems in this case the simple stream releases index is able to be parsed correctly:

2013-09-02 17:14:55 DEBUG juju.environs.simplestreams simplestreams.go:356 read metadata index at "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson"
2013-09-02 17:14:55 DEBUG juju.environs.simplestreams simplestreams.go:491 candidate matches for products ["com.ubuntu.cloud:server:12.04:amd64" "com.ubuntu.cloud:server:12.04:i386"] are [0xc200202b80]
2013-09-02 17:14:55 DEBUG juju.environs.simplestreams simplestreams.go:836 finding products at path "streams/v1/com.ubuntu.cloud:released:azure.sjson"
2013-09-02 17:15:51 INFO juju supercommand.go:284 command finished

The Azure boilerplate I would suggest is at:
lp:~a.rosales/juju-core/update-azure-boilerplate

I do think further analysis is needed as to why 'image stream: released' doesn't work. However, for now we have a work around that we can default and recommend in the documentation.

-thanks,
Antonio

[John A Meinel (jameinel)]

It is because we don't validate the actual content of the "image-stream" field, we just tack it into the lookup string. And ".released" is not in the lookup string, but ".daily" is. (but we wouldn't complain if you put
 image-stream: "quijybo"
It would just happily look for com.ubuntu.cloud-image.quijybo:12.04

I think we knock this down from Critical.

[John A Meinel (jameinel)]

The basic is that 'released' isn't valid in that field but a user might think it should be. However, I'm thinking that rather than validating the field, we should just remove the field entirely. It is the only provider that actually gives you a way to use a daily image. And it was only because of necessity (we only had daily Saucy images, no released Precise ones).
This certainly isn't a critical fix for 1.16 anymore.

[Anastasia (anastasia-macmood)]

This area has changed dramatically.