master tarballs publication is non atomic

[Julien Danjou]

Ceilometer relies on nova-master.tgz for this tests. But I think that the way the tarballs are published isn't atomic, meaning we can download a tar.gz file that is being uploaded, before its upload is finished.

This failed one of our test in this case:

http://logs.openstack.org/28/41628/1/gate/gate-ceilometer-docs/591b61a/console.html

[Jeremy Stanley (fungi)]

We currently upload tarballs with SCP, which does not provide any mechnisms to aid in atomicity.

http://bugs.debian.org/313044

This would instead need a specialized upload script which saves the file to a unique tempoary name on the same remote filesystem, then renames the new file to the intended name replacing the old copy in the process (this turns out to be an atomic operation on most common filesystems supported by the Linux kernel).

Alternatively, and something we've already talked about wanting anyway, we start uploading these files into a public Swift object store at a provider who doesn't put global URLs behind a lengthy caching reverse-proxy/CDN. We would need to write tools to update and publish or proxy a browsable index of those files (something missing from the Swift providers on which the project currently has donated resources--basically we'd have to reinvent the equivalent of Apache's mod_autoindex).

[Julien Danjou (jdanjou)]

I think using rsync could also fix the problem, as by default it does not update file in place.

[aeva black (tenbrae)]

It appears this bug is the root cause of a failure in Ironic's tests for the last few hours, though not in the way described in the bug report. The uploaded Nova tarball appears to have been corrupted by more than one upload job running in parallel

(Ironic currently fetches that tarball to run unit tests on the Nova code we're maintaining in our tree)