vmbuilder EC2: Retry ec2-upload-bundle on failures

Bug #287860 reported by Eric Hammond on 2008-10-23
2
Affects Status Importance Assigned to Milestone
VMBuilder
Undecided
Eric Hammond
vm-builder (Ubuntu)
Wishlist
Eric Hammond

Bug Description

It is common experience for the ec2-upload-bundle command to fail when uploading one of the bundle files. Amazon's recommendation is to either:

1. Add a --retry option to ec2-upload-bundle. The drawback of this is that it will retry forever if failures are constant, but it is an easy fix.

2. Detect the failure and re-run ec2-upload-bundle using the --part option to tell it to start uploading at the failed part number. Exit with an error after some number of failures.

(Slightly obscure) docs for these options are available here:

http://docs.amazonwebservices.com/AWSEC2/2008-05-05/DeveloperGuide/index.html?CLTRG-ami-upload-bundle.html

Here's a sample of the failure (which I've experienced several times in only a dozen builds and that was using hosts on Amazon EC2 with their network).

  Uploaded PREFIX.part.05 to https://s3.amazonaws.com:443/BUCKET/PREFIX.part.05.
  Bundle upload failed.
  , stderr: Error: failed to upload "/tmp/vmbuildern9QPuK/PREFIX.part.06", Server.InternalError(500): We encountered an internal error. Please try again.

This isn't urgent, but fixing this will reduce headaches for many people who will wait through a complete vmbuild run just to have the upload fail. It will also reduce support questions.

Until retries are automated, the workaround is for the user simply to run vmbuilder again.

Nick Barcet (nijaba) on 2008-10-29
Changed in vmbuilder:
assignee: nobody → zulcss
status: New → Confirmed
Nick Barcet (nijaba) on 2008-11-01
Changed in vm-builder:
importance: Undecided → Wishlist
status: New → Confirmed
Eric Hammond (esh) wrote :

Taking this ticket since I am tired of having to restart 30+ minute builds when a single piece of the bundle fails to upload.

Changed in vmbuilder:
assignee: zulcss → esh
Changed in vm-builder:
assignee: nobody → esh
Eric Hammond (esh) wrote :

I went the easy route and just added the --retry option to ec2-upload-bundle

The fix is available on this branch:

  https://code.launchpad.net/~esh/vmbuilder/ec2exec-fixes-2

Changed in vm-builder:
status: Confirmed → Fix Committed
Eric Hammond (esh) wrote :

The fix is now on the following branch:
https://code.launchpad.net/~nijaba/vmbuilder/ec2exec

Changed in vmbuilder:
status: Confirmed → Fix Committed
Eric Hammond (esh) wrote :

The fix has been merged into vmbuilder trunk

Eric Hammond (esh) on 2009-09-05
Changed in vmbuilder:
status: Fix Committed → Fix Released
Changed in vm-builder (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers