publish-build-ebs can leak instances
Bug #927823 reported by
Scott Moser
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu on EC2 |
Fix Released
|
Medium
|
Scott Moser |
Bug Description
A while ago, we changed the ec2 publishing scripts to launch instances in each region concurrently rather than serially during the publish to ebs phase.
Ever since then, in a build failure, there was the possibility of an instance being orphaned and never killed.
That resulted in m1.large instances sitting around doing nothing, and charging us money.
I'd then find them later and kill them, but damage was done.
We need do do everything we can to not leak have instances left running.
I think there are 2 things to do here:
a.) fix the scripts so that they correctly terminate instances on failure
b.) in the event of future failure, having a 'reaper' that would kill instances up to long.
Related branches
affects: | ubuntu → ubuntu-on-ec2 |
To post a comment you must log in.
Attaching example script showing bash signal handling.
This attached scripts basically mimics what publish-build-ec2 does.