ec2test should not fail silently if it exceeds its time-to-live

[Māris Fogels]

When the ec2 testrunner fails due to an issue in the test infrastructure, an automatic timer kicks in to kill the instance. This happens silently, which is annoying to users: the instance just disappears. ec2test should send a mail to the user before it dies stating that the automated shutdown has been triggered.

[Jonathan Lange (jml)]

How would this work? The remote.py script can't do it. You'd have to have a second script on the instance that sends an email and shuts down, I think, and schedule *that* script to be run with 'at'.

[Māris Fogels (mars)]

On Tue, Sep 21, 2010 at 4:35 AM, Jonathan Lange <email address hidden> wrote:
> How would this work? The remote.py script can't do it. You'd have to
> have a second script on the instance that sends an email and shuts down,
> I think, and schedule *that* script to be run with 'at'.
>

Yes, that is what I was thinking.

Getting the shutdown script up there would be a bit ugly - maybe a
here document pushed up to the instance via SSH.

[Jonathan Lange (jml)]

This might have been done, actually.

[Jonathan Lange (jml)]

EC2TestRunner.configure_system sets up the timeout for the shutdown. It works by scheduling a shutdown call in the future using 'at'. If that shutdown is triggered, the test run will fail silently. Have verified this experimentally.

On the plus side, this happens rarely these days. On the negative side, if it starts happening more often, it will be a great pain. It's also hard to fix.