keyring DuplicateDestinationError when updating from custom s-i server

[Vojtech Bocek]

I have my own system-image server set-up. When I want to update Ubuntu Touch installation installed from my server over multiple versions (e.g from 210 to 214), system-image-cli tries to download keyring multiple times (for each delta) and fails with systemimage.download.DuplicateDestinationError (more info in the system-image-cli.log).
This error does not appear on system-image.ubuntu.com, because it doesn't have keyring file. I've worked around the issue on my server by disabling delta generation for keyring files for the moment and removing keyring files from existing deltas in index.json by hand.

[Barry Warsaw (barry)]

I'm tempted to close this as Won't Fix since this is really outside the specs for system-image. There's nothing special about your keyring files, they aren't keyrings in the sense of living under the gpg/ directory at the top level of your server. So s-i just treats these as data files, and data file destination names *must* be unique. If the intent was to distribute new archive-master keyrings to the clients, I don't think this is the right way to do it.

You'll need to make sure each data file (keyring or otherwise) referenced by your index.json is named uniquely. Since the names have no semantic value on the client, the names don't matter. You can always hash in a timestamp or other unique information to make it work.

[Barry Warsaw (barry)]

From IRC:

<stgraber> barry: so the trick with keyring-<hash>.tar.xz is that it is
           constent BUT included in all delta updates
<stgraber> barry: that's required for the corner case where a delta update of
           the ubuntu rootfs overwrites a file that's usually part of the
           keyring tarball
<stgraber> barry: so yeah, keyring-<hash>.tar.xz is a bit special because it's
           inclued in all full images AND all delta images. Obviously if your
           update path includes multiple delta images, it'll be in your
           download list multiple times. [14:59]
<stgraber> barry: right, so I can guarantee that I'll never give you the same
           filename with two different content [15:00]
<stgraber> that's the trick we do so porters don't have to repack the whole
           Ubuntu rootfs for their port and can just use the one from the
           public server

The only sane semantics then are to treat files with the same name as identical, put them in a set, and only download each instance of the file once. However, if the hash is given, complain if they are not the same. Also complain if the source url is not identical.

[Stéphane Graber (stgraber)]

Sounds good to me. Technically the two extra checks you mention are unlikely to be needed since all the files come from /pool/ on the server, so we'd get a failure on the server side long before any problem would happen on the client side :)

But I'm not against the tests as they may end up being useful if we re-organize the server's structure some day.

[Barry Warsaw (barry)]

I'm bumping this past s-i 2.2 so as not to hold up other critical bug fixes.

[Michael Hall (mhall119)]

I am currently experiencing this on the 'devel' channel with mako, on r17 trying to get to r22, it wants to go 17->19->20->22. However, due to broken builds for 18 and 19, 20 is actually a duplicate of 17.

Because of this, it's trying to download two delta images that happen to be the same, giving the duplicate destination error seen here: http://paste.ubuntu.com/7453344/

[Vojtech Bocek (vbocek)]

This bug happend on official s-i.u.com as well. Image 19 on devel channel was broken, so ~ogra ran copy-image on the server to copy last good image (17) as a new revision (17). After next good image 22 was proposed (and probably before that), it resulted into two deltas which contain the same file. This leads to that exception during update.

See the attached index.json, it is from http://system-image.ubuntu.com/ubuntu-touch/utopic/flo/index.json on 2014-05-11. Deltas 17->19 and 19->20 contain the same ubuntu file. I'm not sure if that is correct, how can two deltas directly after each other contain the same file?

[Vojtech Bocek (vbocek)]

I've experienced this on my custom server as well, even though the deltas are generated localy, so I started digging - the corrupted index.json was caused by this code in s-i-server[1]. It causes previously generated deltas with the correct target and destination hash to be accepted even if target and destination hash are swapped.
As you can see, it is support for some old wrong code which produced badly named deltas, so maybe it can be removed. I'll try to remember to ping stgraber about it once he's back from his vacation.

[1]: http://bazaar.launchpad.net/~ubuntu-system-image/ubuntu-system-image/server/view/head:/lib/systemimage/generators.py#L91