tools tarballs need to be stored in a directory named after their stream

[Ian Booth]

The current directory structure for accessing tools metadata and tarballs is:

http://streams.canonical.com/juju/tools/streams
http://streams.canonical.com/juju/tools/releases

The tarballs for all tools for all streams (proposed, testing, devel, released) are put in the releases directory.

We want to move to a model where separate directories are used for tarballs for each stream:

http://streams.canonical.com/juju/tools/released
http://streams.canonical.com/juju/tools/devel
http://streams.canonical.com/juju/tools/proposed
http://streams.canonical.com/juju/tools/testing

The paths to the tarballs are inside the products metadata file. The juju metadata generate-tools command needs to be updated to write the new paths to the product metadata file.

[Curtis Hovey (sinzui)]

This isn't really fixed.
We ran generate-tools with agents in the "releases" dir. the product file says they are in a non-existent dir named "released". I don't see an option to tell juju which dir the agents are in. I don't see an option to specify the purpose of the stream...devel, proposed, released.

[Curtis Hovey (sinzui)]

I agree with this structure I see in the commit...

+<metadata-source-directory>
+ |-tools
+ | |-streams
+ | |-v1
+ | |-released
+ | |-proposed
+ | |-testing
+ | |-devel
+ |
+ |-images
+ |-streams
+ |-v1

since our tree only has streams/ and releases/, releases is the dir name to use in the path for the products. The old rule for 1.20.x and older is "released" purpose is in dir "releases/" when the callee passes an option to specify the agent dir and purpose, those are used in the product file and index respectively. If the generate-streams does not find the dir...it must raise an error so that someone can fix the command options or rename the dir.

[Curtis Hovey (sinzui)]

Another, optionless solution for generate-tools is that is scans the tools dir and created a purposeful section for each dir of agents found. "releases" is used for "released" if there isn't a dir named "released"

For our publication of tools. we would add rules to ensure only the product file we were adding or removing to changed.

[Curtis Hovey (sinzui)]

I see this in the docs of the commit
    + juju generate-tools -d <metadata_dir> --stream <stream>
but this is not in the generate-tools help docs. --streams looks promising. Though it is clear it guessed at which dir was "released" and didn't honour the dirname, If <stream> is meant to be synonymous with the dirname, then an error should be raised when the dir doesn't exist.

[Ian Booth (wallyworld)]

To confirm, the fix for this will also start using the more correct content ids.

eg for stream = proposed

content id will be "com.ubuntu.juju:proposed:tools"

Index file will contain:

{
    "index": {
        "com.ubuntu.juju:proposed:tools": {
            "updated": "Thu, 30 Oct 2014 08:00:54 +1000",
            "format": "products:1.0",
            "datatype": "content-download",
            "path": "streams/v1/com.ubuntu.juju:proposed:tools.json",
            "products": [
                "com.ubuntu.juju:12.04:amd64"
            ]
        }
    },
    "updated": "Thu, 30 Oct 2014 08:00:54 +1000",
    "format": "index:1.0"
}

1.20 will not be able to find tools with these content ids. But 1.20, as is the case now, will source proposed tools from http://streams.canonical.com/juju/proposed/tools/ using tools-metadata-url, and the metadata here will still use the old content id for proposed tools ie "com.ubuntu.juju:released:tools"