It should batch-unfreeze glacier S3 objects to speed up restorations
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Duplicity |
New
|
Undecided
|
Unassigned |
Bug Description
Scenario: Duplicity is configured with S3 backend, S3 is configured to move objects to Glacier storage after 1 week, full backups are stored every Sunday, partial backups happen once each night, today is Friday, the user wants to restore a backup from last week's Wednesday.
User executes `duplicity [options] restore s3://bucket/folder /local/folder --time 9D`.
What happens: The backup takes about 2-4 days to restore.
What should happen: The backup should take about 5-12h to restore.
Why it happens: Duplicity unfreezes (move from Glacier to Standard S3 storage mode) archives automatically (cool) one by one (not cool). Unfreezing one archive can take usually 5-12h. Assuming you need to unfreeze files from 4 days to get the full chain, that gives you a 20-48h minimum wait, unless files are split into i.e. 200MB chunks, which can produce more objects to unfreeze, summing extra 5-12h each.
Why it should not happen: If you go into S3 Management Console, you can manually select as many Glacier files as you want and unfreeze them all at once, having to wait those 5-12h, yes, but just once for all of them. Duplicity should do that automatically through Boto too.
Thanks!