Doesn't check for free space in archive dir when backing up

[hackel]

Running latest version from ppa:
deja-dup 10.1-0jaunty1
duplicity 0.6.02-0jaunty1

When initiating a backup, deja-dup calls duplicity with --archive-dir=/home/user/.cache/deja-dup however it needs to check to make sure this directory actually has sufficient free space available. In my case, it silently ate up all my free space (only about 400M at the time) and then stopped the backup (after only backing up about 30G out of 120). This then caused multiple other applications to crash.

Perhaps more importantly, Deja Dup should give some indication of the necessity for this free space in the first place! I assumed that when backing up *to* another drive (with 500G+ free) it would not be a problem. You should also let the user choose the location for these files, in case they need to store them on another drive or partition with sufficient free space.

How much space does the archive dir usually consume? Is it dependent on the number of files backed up? Is this directory required in order to restore later, or is it only for making future incremental backups? I've never used duplicity before so I'm not familiar with it, but if possible it might be worth having an option to disable it completely. It sounds like it is only storing unencrypted copies of the signature files here, right? The same files which are on the backup medium itself, so they couldn't they just be re-read?

[Michael Terry (mterry)]

Thanks, hackel for the bug report! It is not expected that your archive dir would consume so much space.

Ken, this sounds like odd archive dir behavior. Got any ideas?

[Kenneth Loafman (kenneth-loafman)]

I'll look into it. That does seem a bit high. A full backup will create a full signature set, and that could be fairly large.

[Michael Terry (mterry)]

From dup bug 688902:

"I want to be explicitly notfied that a program will create large amounts of data for such a backup, and as a user I want to be asked if I really want that, and that I can chose a special location for this data alternatively."

[Kenneth Loafman (kenneth-loafman)]

Duplicity does check space requirements on the local machine. There is no
way to check availability on the remote machine.

...Ken

On Thu, Jun 9, 2011 at 7:55 AM, Michael Terry
<email address hidden>wrote:

> >From dup bug 688902:
>
> "I want to be explicitly notfied that a program will create large
> amounts of data for such a backup, and as a user I want to be asked if I
> really want that, and that I can chose a special location for this data
> alternatively."
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/403109
>
> Title:
> Doesn't check for free space in archive dir when backing up
>
> Status in Déjà Dup Backup Tool:
> Confirmed
> Status in Duplicity - Bandwidth Efficient Encrypted Backup:
> New
>
> Bug description:
> Running latest version from ppa:
> deja-dup 10.1-0jaunty1
> duplicity 0.6.02-0jaunty1
>
> When initiating a backup, deja-dup calls duplicity with --archive-
> dir=/home/user/.cache/deja-dup however it needs to check to make sure
> this directory actually has sufficient free space available. In my
> case, it silently ate up all my free space (only about 400M at the
> time) and then stopped the backup (after only backing up about 30G out
> of 120). This then caused multiple other applications to crash.
>
> Perhaps more importantly, Deja Dup should give some indication of the
> necessity for this free space in the first place! I assumed that when
> backing up *to* another drive (with 500G+ free) it would not be a
> problem. You should also let the user choose the location for these
> files, in case they need to store them on another drive or partition
> with sufficient free space.
>
> How much space does the archive dir usually consume? Is it dependent
> on the number of files backed up? Is this directory required in order
> to restore later, or is it only for making future incremental
> backups? I've never used duplicity before so I'm not familiar with
> it, but if possible it might be worth having an option to disable it
> completely. It sounds like it is only storing unencrypted copies of
> the signature files here, right? The same files which are on the
> backup medium itself, so they couldn't they just be re-read?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/deja-dup/+bug/403109/+subscriptions
>

[Michael Terry (mterry)]

Ken, I recall that it checks free space requirements in /tmp, but does it do so for ~/.cache?

[Kenneth Loafman (kenneth-loafman)]

No, it does not. Will add that check as well.

On Thu, Jun 9, 2011 at 9:27 AM, Michael Terry
<email address hidden>wrote:

> Ken, I recall that it checks free space requirements in /tmp, but does
> it do so for ~/.cache?
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/403109
>
> Title:
> Doesn't check for free space in archive dir when backing up
>
> Status in Déjà Dup Backup Tool:
> Confirmed
> Status in Duplicity - Bandwidth Efficient Encrypted Backup:
> New
>
> Bug description:
> Running latest version from ppa:
> deja-dup 10.1-0jaunty1
> duplicity 0.6.02-0jaunty1
>
> When initiating a backup, deja-dup calls duplicity with --archive-
> dir=/home/user/.cache/deja-dup however it needs to check to make sure
> this directory actually has sufficient free space available. In my
> case, it silently ate up all my free space (only about 400M at the
> time) and then stopped the backup (after only backing up about 30G out
> of 120). This then caused multiple other applications to crash.
>
> Perhaps more importantly, Deja Dup should give some indication of the
> necessity for this free space in the first place! I assumed that when
> backing up *to* another drive (with 500G+ free) it would not be a
> problem. You should also let the user choose the location for these
> files, in case they need to store them on another drive or partition
> with sufficient free space.
>
> How much space does the archive dir usually consume? Is it dependent
> on the number of files backed up? Is this directory required in order
> to restore later, or is it only for making future incremental
> backups? I've never used duplicity before so I'm not familiar with
> it, but if possible it might be worth having an option to disable it
> completely. It sounds like it is only storing unencrypted copies of
> the signature files here, right? The same files which are on the
> backup medium itself, so they couldn't they just be re-read?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/deja-dup/+bug/403109/+subscriptions
>

[David Roundy (roundyd)]

I'll just add that at least as of the version of deja-dup 14.2-1squeeze1which is in Debian squeeze, the error message simply states that disk is full, leading one naturally to assume that the *backup* disk is full, not the disk that is being backed up. This is quite confusing, so besides doing the check beforehand, it would be great to figure out how to give useful error messages, since it's not possible to determine in advance whether there will really be room for a few gigabytes of temporary files.

[Michał J. Gajda (mgajda)]

Just removed 40GB of .cache/deja-dup, parts of which were dated few months backwards. That's definitely too much for 250GB disk to backup!

[Dražen Lučanin (kermit666)]

A possible workaround benvdh suggested to me until this gets fixed:

> Basically what I did was to move the ~/.cache/deja-dup folder to my external harddrive and add a simlink with deja-dup as the link_name in ~./cache.

and

> Another thing that helped was to reduce the time backup's needed to be kept. I changed it from forever to a year. Which reduced the cache to 13GB.

This advice might be helpful to others who share our problems.

[Sicco van Sas (sicco)]

I'm having the same problem. Deja-dup kept on failing to backup saying that there was not enough space left, while my backup hard drive contains more than enough free space (808GB versus 461GB needed to backup). It only became apparent to me that deja-dup meant that my main hard drive was out of disk space when Ubuntu gave my a warning saying that my hard disk was nearly full. The deja-dup .cache folder took up 26GB.

It would indeed be nice if a more descriptive error message was displayed and/or if deja-dup indicates how much disk space it needs on the main hard drive before starting the backup. I'll use Dražen Lučanin's workaround for now.

[ragnaroknroll (ragnaroknroll)]

Just ran into this exact same problem. Would really appreciate a fix.

[Felipe Castillo (fcastillo.ec)]

I haven't tried yet, so I thought I'll ask here before doing it. Will move the directory to a different drive and using a symlink work? Will deja-dup show any problems with this approach?