vmbuilder EC2: Retry ec2-upload-bundle on failures

[Eric Hammond]

It is common experience for the ec2-upload-bundle command to fail when uploading one of the bundle files. Amazon's recommendation is to either:

1. Add a --retry option to ec2-upload-bundle. The drawback of this is that it will retry forever if failures are constant, but it is an easy fix.

2. Detect the failure and re-run ec2-upload-bundle using the --part option to tell it to start uploading at the failed part number. Exit with an error after some number of failures.

(Slightly obscure) docs for these options are available here:

http://docs.amazonwebservices.com/AWSEC2/2008-05-05/DeveloperGuide/index.html?CLTRG-ami-upload-bundle.html

Here's a sample of the failure (which I've experienced several times in only a dozen builds and that was using hosts on Amazon EC2 with their network).

  Uploaded PREFIX.part.05 to https://s3.amazonaws.com:443/BUCKET/PREFIX.part.05.
  Bundle upload failed.
  , stderr: Error: failed to upload "/tmp/vmbuildern9QPuK/PREFIX.part.06", Server.InternalError(500): We encountered an internal error. Please try again.

This isn't urgent, but fixing this will reduce headaches for many people who will wait through a complete vmbuild run just to have the upload fail. It will also reduce support questions.

Until retries are automated, the workaround is for the user simply to run vmbuilder again.

[Eric Hammond (esh)]

Taking this ticket since I am tired of having to restart 30+ minute builds when a single piece of the bundle fails to upload.

[Eric Hammond (esh)]

I went the easy route and just added the --retry option to ec2-upload-bundle

The fix is available on this branch:

  https://code.launchpad.net/~esh/vmbuilder/ec2exec-fixes-2

[Eric Hammond (esh)]

The fix is now on the following branch:
https://code.launchpad.net/~nijaba/vmbuilder/ec2exec

[Eric Hammond (esh)]

The fix has been merged into vmbuilder trunk