promote-to-release loses packages on failed copies

[Colin Watson]

https://launchpad.net/ubuntu/+source/qtbase-opensource-src/5.2.1+dfsg-1ubuntu10/+publishinghistory was copied to the release pocket by proposed-migration, but the copy OOPSed with a timeout; promote-to-release didn't notice this, and went ahead and deleted the package from -proposed anyway. This can be recovered by repeating the copy, but only if somebody actually notices. While there are worse failure modes, this is pretty bad.

Now that copies are in celery and therefore usually quick, I think we should make promote-to-release do all its copies, wait for a bit, check that the copies happened, and only delete the ones that happened. However, we do have to be a little bit careful because it isn't completely unheard of for us to copy things back and forward between release and -proposed more than once. I think the correct algorithm is probably to check the most recent publication in the release pocket ordered by date_created and require it to be greater than the date_created of the -proposed publication we're copying.

[Colin Watson (cjwatson)]

Although it would be much slower, there's an argument for doing the checks one copy at a time; it would lessen the chance of a later copy being performed that depends on the first copy. In the absence of being able to do a transactional set of copies, proposed-migration *tends* to perform the copies roughly in dependency order.

Of course, if there's a large set of circularly-dependent copies, we're screwed either way. So maybe it isn't worth the slowness given that this type of failure is rare.