publish-build-ebs can leak instances

[Scott Moser]

A while ago, we changed the ec2 publishing scripts to launch instances in each region concurrently rather than serially during the publish to ebs phase.

Ever since then, in a build failure, there was the possibility of an instance being orphaned and never killed.
That resulted in m1.large instances sitting around doing nothing, and charging us money.
I'd then find them later and kill them, but damage was done.

We need do do everything we can to not leak have instances left running.

I think there are 2 things to do here:
 a.) fix the scripts so that they correctly terminate instances on failure
 b.) in the event of future failure, having a 'reaper' that would kill instances up to long.

[Scott Moser (smoser)]

Attaching example script showing bash signal handling.

This attached scripts basically mimics what publish-build-ec2 does.

[Scott Moser (smoser)]

marking fix-commited. i pushed code to do this, and a 'reap-ec2-instances' script.
reap-ec2-instances is not running from cron yet, but that will be done at a later date.

[Scott Moser (smoser)]

fix-released.
we are not running the reaper, but the ec2-image2ebs code now doesn't leak instances so easily.
this was fixed in revno 411.