RFC: Add url/urls/mirrors attribute for product items

[Daniele Rondina]

Hi,

currently, simplestreams protocol is used inside the LXD project for share LXD images through HTTPS endpoint (HTTP is not permitted).

Through [simplestreams](https://github.com/MottainaiCI/simplestreams-builder) project I use it in Linux Sabayon for share custom LXD images to Sabayon users.

The main problem is that we haven't a direct access Sabayon Mirrors and often these are available only through HTTP. The idea is to use HTTPS for retrieve CRC, MD5 and then redirect users to HTTP endpoint of our mirrors. From the security side, we hare compliant because hashing of the images are fetched through HTTPS.

For fix this the idea is to add a new attribute to product item "url" that is complementary or alternative to the field "path":

{
 "datatype": "image-downloads",
 "updated": "Sat, 04 May 2013 01:58:57 +0000",
 "content_id": "sabayon:images:download",
 "products": {
  "sabayon:0.1:arm": {
   "arch": "arm",
   "stream": "devel",
   "versions": {
    "20190111": {
     "items": {
      "uec.tar.gz": {
       "ftype": "lxc.tar.gz",
       "path": "/lxd-images/sabayon/myimage.tar.gz",
       "url": "http://mymirror.edu/lxd-images/sabayon/myimages.tar.gz",
       "sha256": "xxxxxx",
       "md5": "797e2d488c799eab0a8eb09a9c1ff4a3",
       "size": 7314153
      },
...
}

Here, there are different possibilities:

* url: with a complete path of the resource
* urls: a list of URL to use for fetch the resource
* mirrors: a field "mirrors" that contains the list of nodes to use as a prefix for "path" field.

This resolves my problem with Sabayon mirrors but could be also a solution to separate the node that exposes index.json and images.json from nodes that contain LXC/LXD images. Also for scalability.

WDYT?

Thanks in advance

[Dan Watkins (oddbloke)]

Hi Daniele,

Thanks for this suggestion. I've added the lxd package in Ubuntu; as they would need to support this new syntax, I'd like to hear their thoughts before we go any further.

Thanks!

Dan

[Stéphane Graber (stgraber)]

Considering we can quite likely get the reporter to extend our simplestreams implementation to support this, I'm fine with it :)

It's not something we need for any of the existing servers on our end (be that cloud-images.ubuntu.com or images.linuxcontainers.org) but I see how this may be useful for some other Linux distributions.

[Daniele Rondina (geaaru)]

Hi Dan and Stéphane,

thank you very much for the support.

So, what is the solution choice?

Probably, the better choice is `urls` or `mirrors` so it's possible inside LXD code handle failover with a secondary mirror.

So, for a recap:

Solution a)

{
 "datatype": "image-downloads",
 "updated": "Sat, 04 May 2013 01:58:57 +0000",
 "content_id": "sabayon:images:download",
 "products": {
  "sabayon:0.1:arm": {
   "arch": "arm",
   "stream": "devel",
   "versions": {
    "20190111": {
     "items": {
      "uec.tar.gz": {
       "ftype": "lxc.tar.gz",
       "urls": [
           "http://mymirror.edu/lxd-images/sabayon/myimages.tar.gz",
           "http://mymirror2.edu/lxd-images/sabayon/myimages.tar.gz"
       ],
       "sha256": "xxxxxx",
       "md5": "797e2d488c799eab0a8eb09a9c1ff4a3",
       "size": 7314153
      },
...
}

From LXD client-side the logic could be that if `urls` is not empty I get a random entry from url array and I try to fetch image else use path field as now.

Solution b)

{
 "datatype": "image-downloads",
 "updated": "Sat, 04 May 2013 01:58:57 +0000",
 "content_id": "sabayon:images:download",
 "products": {
  "sabayon:0.1:arm": {
   "arch": "arm",
   "stream": "devel",
   "versions": {
    "20190111": {
     "items": {
      "uec.tar.gz": {
       "ftype": "lxc.tar.gz",
       "path": "/lxd-images/sabayon/myimage.tar.gz",
       "mirrors": [
           "http://mymirror.edu",
           "http://mymirror2.edu/sabayon/"
       ],
       "sha256": "xxxxxx",
       "md5": "797e2d488c799eab0a8eb09a9c1ff4a3",
       "size": 7314153
      },
...
}

With this solution, `mirrors` contains the hostname with an optional prefix and from LXD client-side if mirrors have values then a random field of mirrors field is used and value of path field is used for creating image URL (mirrors[0]+path)

Please, let me know how we want to proceed so I try to push a PR to LXD project after my holidays.

Thanks again.

[Daniele Rondina (geaaru)]

Ref merge request for this issue:
https://code.launchpad.net/~geaaru/simplestreams/+git/simplestreams/+merge/379255

[Daniele Rondina (geaaru)]

Can anyone take a look at this, please?