It should batch-unfreeze glacier S3 objects to speed up restorations

Bug #1740833 reported by Yajo
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Duplicity
New
Undecided
Unassigned

Bug Description

Scenario: Duplicity is configured with S3 backend, S3 is configured to move objects to Glacier storage after 1 week, full backups are stored every Sunday, partial backups happen once each night, today is Friday, the user wants to restore a backup from last week's Wednesday.

User executes `duplicity [options] restore s3://bucket/folder /local/folder --time 9D`.

What happens: The backup takes about 2-4 days to restore.

What should happen: The backup should take about 5-12h to restore.

Why it happens: Duplicity unfreezes (move from Glacier to Standard S3 storage mode) archives automatically (cool) one by one (not cool). Unfreezing one archive can take usually 5-12h. Assuming you need to unfreeze files from 4 days to get the full chain, that gives you a 20-48h minimum wait, unless files are split into i.e. 200MB chunks, which can produce more objects to unfreeze, summing extra 5-12h each.

Why it should not happen: If you go into S3 Management Console, you can manually select as many Glacier files as you want and unfreeze them all at once, having to wait those 5-12h, yes, but just once for all of them. Duplicity should do that automatically through Boto too.

Thanks!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.