Incremental backups fail due to available space prediction

[sebastian-s]

for quite some time now NSSbackup is running in the background.

I usually check for a full backup once a month. I move the full backup folder to an external HDD and collect them there, some I burn on DVD in addition.

On every Incremental backup I get an corrupt backup.

This does happen if the backup destination has little or much free space. (With much I mean >2 gig and I think that the changes are about 50 MB)
I know that once I have a full backup (about 6 GB) on the HDD there is not enough space left for another one but usually there are about 2 GB which should be sufficient for a incremental backup I thought.

The strange thing is that the backup folder starts as a
    2010-01-05_08.00.10.089553.ubuntu.inc
and becomes a
   2010-01-05_08.00.10.089553.ubuntu.corrupt
after a while.

Full backups do work. Recovering files from a full backup does work too.

I can provide many other log files of similar nature if it helps ;-)

[Oumar Aziz OUATTARA (wattazoum)]

Hi,

This "problem" is known. This fact is that NSsbackup can't determine the real size needed by an incremental backup.
This is because the job of deciding what is to be put in an incremental backup is left to TAR.

So basically, the size that we put as required for a backup is the worst case, the size of a full uncompressed backup (even though less might actually be used).

But I think what can be done for this Bug is to add a parameter that would allow the backup to proceed even though the needed space is not available. But there is a big risk in it: the risk of the user not noticing to the disk is getting full and to have no space left on the disk to have a working system.

Maybe there is a way to ask to TAR to stop before the disk is full but this needs to be checked.

[sebastian-s (sebastian-s)]

Hi,

although it its highly unlikely that an incremental backup is the same size as a full one I would still go with the option you suggested.

I think that in the rare occasion of a incremental backup being same or close to the size of a full backup the backup process can still fail (though tar). This would allow the other incremental backups with little changes to proceed.

Just my thought...

[Oumar Aziz OUATTARA (wattazoum)]

What can be done is to remove the files.tgz file in the following occasion:
- if the backup fails (due to TAR)
- and the disk is limited in space (value to be configurable) after the backup.

I will try to look into this solution when I get some time.

[Jean-Peer Lorenz (peer.loz)]

I've made some good progress on this issue using real incremental space prediction (see linked branch). The code stilll needs some testing and clean up, however I'm confident to merge it within the next 2 weeks.

[Jean-Peer Lorenz (peer.loz)]

Fix committed: the required space is predicted using a similar algorithm as Tar does.

Note: The required space is calculated on the base of the uncompressed backup size. The actual required space will be less if the snapshot is compressed.

[djalfie (djalfie)]

Is there any option to force the backup and ignore this warning when using gzip compression?
Maybe would be better to autoremove old snapshots when there isn't enough space (here using the actual size).

[Jean-Peer Lorenz (peer.loz)]

Thanks for your contribution. A fix solving the bug is already committed and will be released within this month. With the fix applied, there is no longer a need to force the backup (or to incorporate an option to do so). Let my explain how it works:

* in the case of full backups the required uncompressed space is computed and the backup is done only if enough disk space is available.
* if an incremental backup is created the required uncompressed space of the files actually being backed up is computed. The backup is aborted if not enough disk space is available.

Simple and consistent.

Some words about an additional option: It is, IMO, not a good idea to force a backup if not enough disk space is available. Such option would be dangerous for the most of the users. Moreover, if you know you have only little disk space available you can check your space before doing a manual backup. I think this is similar to an additional action triggered by such option. So, frankly speaking I believe If someone would use this option he knows that there is only little disk space and should check it manually instead.

To the autoremove thing: there is already such feature - the purging of old snapshots. Introducing another autoremove is dangerous again since I usually want to keep some old snapshots instead of remove them. Moreover, it is hard to decide for software (meaning to implement) what snapshots should be removed and what to keep. And disk space is getting cheaper and cheaper.

I aggree that a useful feature would be the fallback to an incremental backup if there is not enough space for a full backup. However, this requires an existing and valid snapshot as parent/base.

Hope this helps,
Jean-Peer