New feature: Support for Amazon Glacier

[Matthew Copperwaite]

Amazon has this morning announced a new service called Glacier that allows significantly cheaper storage for a time delayed return of the data. This is perfect for backup and therefore Deja Dup. It would be extremely useful to have this service supported.

http://aws.amazon.com/glacier/
http://docs.amazonwebservices.com/amazonglacier/latest/dev/introduction.html

[Matthew Copperwaite (mattcopp)]

A ticket is open with the boto guys to add Glacier functionality at their end. From that point on it's about adding the support to Deja Dup.

https://github.com/boto/boto/issues/929

[Willem Ligtenberg (wligtenberg)]

In that thread they say that glacier functionality has been merged into develop.
https://github.com/boto/boto/issues/929

Maybe now is a good time to see if it can be integrated into deja dup?

[Matthew Copperwaite (mattcopp)]

Ah, yes. I totally forgot this was the reason I chased that up. The develop branch still means it's in testing, but from working with the code I don't think it will change dramatically from now on.

[Rotonen (joni-orponen)]

Glacier has hit master on boto ~22 days ago.

https://github.com/boto/boto/tree/master/boto/glacier

Since then the release 2.6.0 has been tagged on github and pushed to pypi.

http://pypi.python.org/pypi/boto

2.6.0 includes the glacier work.

The work here can resume now?

[kmontag (kmontag)]

Big +1 for this feature - any updates?

[Michael Terry (mterry)]

No updates. This isn't really on my radar, since there would need to be some UI work for this. It would be tricky to present to the user how long to wait when restoring and to be able to start downloading when it is ready.

[Eric Friedrich (eric-friedrich84)]

Plus duplicity (underlying deja-dup) doesn't yet support Glacier. That would likely need to happen first before Deja Dup adds support.

[Cameron Eagans (cweagans)]

I'd like to take this on, but I'm not sure what actually needs done. Sounds like duplicity needs to support Glacier, but not sure what else. If you guys can point me in the right direction, I'd love to spend some time on it.

[Eric Friedrich (eric-friedrich84)]

Hey Cameron-
  I've looked at this a bit before. Duplicity does need to support glacier
and I did a bit of basic work in that area previously, but not enough to
get it production ready.

Here's what I've done so far:
https://code.launchpad.net/~eric-friedrich84/duplicity/glacier

However the duplicity list and retrieval is _very_ rough around the edges.
I've also gathered some feedback from one of the duplicity maintainer's:
http://lists.nongnu.org/archive/html/duplicity-talk/2013-01/msg00008.html

If you feel like this might provide a good base to continue work, I'm glad
to keep this effort going.

--Eric

On Tue, Apr 9, 2013 at 9:24 PM, Cameron Eagans <email address hidden> wrote:

> I'd like to take this on, but I'm not sure what actually needs done.
> Sounds like duplicity needs to support Glacier, but not sure what else.
> If you guys can point me in the right direction, I'd love to spend some
> time on it.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1039471
>
> Title:
> New feature: Support for Amazon Glacier
>
> Status in Déjà Dup Backup Tool:
> Confirmed
>
> Bug description:
> Amazon has this morning announced a new service called Glacier that
> allows significantly cheaper storage for a time delayed return of the
> data. This is perfect for backup and therefore Deja Dup. It would be
> extremely useful to have this service supported.
>
> http://aws.amazon.com/glacier/
>
> http://docs.amazonwebservices.com/amazonglacier/latest/dev/introduction.html
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/deja-dup/+bug/1039471/+subscriptions
>

[Gaëtan André (gaetan-andre)]

Hello,

Where are we on this ?

[Stefan Fussenegger (sfussenegger)]

Would Object Lifecycle Management (Object Archival) make this any easier? Or would it even work as long as objects would be manually restored to S3 before being restored by Deja Dup?

http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html

[Matthew Copperwaite (mattcopp)]

So potentially with that it could be coded so that the file is uploaded to S3 and then automatically sent to Glacier. That would definitely be a work around for the upload but I think the problem is the retrieval procedure with the 4hr delay.

[Stefan Fussenegger (sfussenegger)]

Yes, retrieval would require some extra work, something like

if (object.storageClass == glacier) {
  requestRestore(object)
  alert("Restoring data from Glacier. This will take 3 to 5 hours. Told ya!")
  while (object.restore != complete) wait()
}
object = downloadFromS3(object)

Obviously, restoring one object at a time will make it horribly slow, but I think as long as you keep talking to S3 only, it shouldn't be much more work than that.

The more important question though is how moving files to Glacier would have an impact on performing backups. If any. If it didn't, it would be safe to user these rules already - with the need to restore files manually from Glacier to S3 before they can be restored from S3.

[Rune K. Svendsen (runeks)]

Amazon Glacier support for Duplicity was recently added, see bug #1039511.

[Mike (mike-zandvliet)]

If Deja-Dup implemented support for the new duplicity command line option "--file-prefix-manifest", then we could use lifecycle rules in S3 to give us cheap Glacier storage while maintaining the manifest files on S3.

I don't think it would even need any change to the GUI. Just start using --file-prefix-manifest for all new backups, and in the restore functionality have it look for both manifest files with and without the prefix.

Without this, support for S3 is really only there in name. For all practical purposes, S3 is too expensive to use for medium or large backups. Glacier on the other hand is perfectly priced.

In terms of restoring from Glacier, I'd suggest that as an interim solution, we can let the user handle the Glacier>S3 part of the restore via the AWS website or any of the other tools out there (i.e. s3cmd). In the longer term, yes - it would be good for Deja-Dup to initiate the Glacier>S3 restore and to wait till it is available. But let's not hold off for that to be built. Right now just implementing "--file-prefix-manifest" gives us everything we need to get started with Glacier.

[Marcos Alano (mhalano)]

Any news about that?

[Michael Terry (mterry)]

Support for S3 has been removed, per our cloud policy that prefers consumer oriented cloud services. Duplicity can still be used directly to backup or restore your S3 files.

https://wiki.gnome.org/Apps/DejaDup/Clouds