In full rsync mode deleted files and folders are not removed from the snapshot

[Mauro]

I don't know if it's intentional, but for me it's a bug.
When full rsync mode is enabled, if I delete a file or a folder from source, it won't be deleted in the next snapshot.
Also, if the deletion is the only change in source, BIT says there's no difference and hence doesn't take any snapshot.

BTW: isn't there a way to get a snapshot in full rsync mode even if there's no change? The checkbox for this gets disabled when you choose "full rsync mode".

[Germar (germar)]

I can't confirm this here. Deleted files or folders are gone in next snapshot. And this would be a serious bug otherwise

Could you please describe your setup? Local or remote drive, source and destination filesystem, rsync version on both if remote...

'Check for changes' is disabled with 'Full rsync mode' because there is no such check. With normal mode this will deactivate the rsync --dry-run. But with 'Full rsync mode' BIT will create a new snapshot anyways and delete it afterwards if there was no changes. This is still faster than before.

[Mauro (mauromol)]

It's a local backup. In my test case, both source and destination folders are on the same ext4 filesytem, but I observe this even on an older Debian Wheezy on a NAS box with BIT 1.0.10 installed on it, backing up from an ext4 volume to another ext4 filesystem on a different device.

Here is my test-case profile:

profile4.name=test
profile4.qt4.last_path=/
profile4.qt4.places.SortColumn=1
profile4.qt4.places.SortOrder=0
profile4.qt4.settingsdialog.exclude.SortColumn=1
profile4.qt4.settingsdialog.exclude.SortOrder=0
profile4.qt4.settingsdialog.include.SortColumn=1
profile4.qt4.settingsdialog.include.SortOrder=0
profile4.snapshots.automatic_backup_anacron_period=1
profile4.snapshots.automatic_backup_anacron_unit=20
profile4.snapshots.automatic_backup_day=1
profile4.snapshots.automatic_backup_mode=0
profile4.snapshots.automatic_backup_time=0
profile4.snapshots.automatic_backup_weekday=7
profile4.snapshots.backup_on_restore.enabled=true
profile4.snapshots.bwlimit.enabled=false
profile4.snapshots.bwlimit.value=3000
profile4.snapshots.check_for_changes=true
profile4.snapshots.continue_on_errors=false
profile4.snapshots.copy_links=false
profile4.snapshots.copy_unsafe_links=false
profile4.snapshots.cron.ionice=true
profile4.snapshots.cron.nice=true
profile4.snapshots.custom_backup_time=8,12,18,23
profile4.snapshots.dont_remove_named_snapshots=true
profile4.snapshots.exclude.1.value=.gvfs
profile4.snapshots.exclude.10.value=/dev/*
profile4.snapshots.exclude.11.value=/run/*
profile4.snapshots.exclude.12.value=/etc/mtab
profile4.snapshots.exclude.13.value=/var/cache/apt/archives/*.deb
profile4.snapshots.exclude.14.value=lost+found/*
profile4.snapshots.exclude.15.value=/tmp/*
profile4.snapshots.exclude.16.value=/var/tmp/*
profile4.snapshots.exclude.17.value=/var/backups/*
profile4.snapshots.exclude.18.value=.Private
profile4.snapshots.exclude.2.value=.cache/*
profile4.snapshots.exclude.3.value=.thumbnails*
profile4.snapshots.exclude.4.value=[Tt]rash*
profile4.snapshots.exclude.5.value=*.backup*
profile4.snapshots.exclude.6.value=*~
profile4.snapshots.exclude.7.value=.dropbox*
profile4.snapshots.exclude.8.value=/proc/*
profile4.snapshots.exclude.9.value=/sys/*
profile4.snapshots.exclude.bysize.enabled=false
profile4.snapshots.exclude.bysize.value=500
profile4.snapshots.exclude.size=18
profile4.snapshots.full_rsync=true
profile4.snapshots.include.1.type=0
profile4.snapshots.include.1.value=/home/mauro/tmp/test bit/src
profile4.snapshots.include.size=1
profile4.snapshots.local.nocache=false
profile4.snapshots.local.password.save=false
profile4.snapshots.local.password.use_cache=true
profile4.snapshots.local_encfs.path=/home/mauro/tmp/test bit/dst
profile4.snapshots.log_level=3
profile4.snapshots.min_free_inodes.enabled=true
profile4.snapshots.min_free_inodes.value=2
profile4.snapshots.min_free_space.enabled=true
profile4.snapshots.min_free_space.unit=20
profile4.snapshots.min_free_space.value=1
profile4.snapshots.mode=local
profile4.snapshots.no_on_battery=false
profile4.snapshots.notify.enabled=true
profile4.snapshots.path=/home/mauro/tmp/test bit/dst
profile4.snapshots.path.host=hppb
profile4.snapshots.path.profile=4
profile4.snapshots.path.user=mauro
pro...

[Mauro (mauromol)]

As an additional comment: this test case doesn't show that if I have a deletion in folder "test" and other changes in another folder (not in "test"), then a new snapshot IS actually taken, but the deletion of files in "test" is not applied to this new snapshot. I observed this in real cases on the NAS box I mentioned.

[Mauro (mauromol)]

Forgot to write about the rsync version: the test case was conducted on a Linux Mint 17.1 (based on Ubuntu 14.04):
rsync version 3.1.0 protocol version 31
The NAS box is using a backported rsync version:
rsync version 3.1.1 protocol version 31

[Germar (germar)]

Okay, I was able to reproduce it with your config. Thanks

First think I recognized is that paths for 'save config file' are not quoted correctly and so break with spaces. But that's not the root of this problem. I'll skip new release again and fix this first ;D

BTW BIT 1.0.10 didn't have 'full rsync mode'. So that must be something different.

[Germar (germar)]

Okay, here is what happens:
- BIT will decide if there where changes and a new snapshot should be taken/kept based on the output of rsync
- with 'Full rsync mode' deactivated we create hard-links in a separate step with 'cp -aRl' and rsync will sync the current local status over the existing hard-links. If a file was locally removed rsync will remove the hard-link on remote, too and print a message about what it has done
- with 'Full rsync mode' activated rsync will create hard-links with '--link-dest' and sync over current local status in one step. So if there was a local file deleted rsync will simply just not hard-link that file on remote and won't say anything about that. So BIT doesn't know what happened and if there where no other changes it will suppose that there was no changes at all and remove the new snapshot again.

I'm not sure if there is a way to tell rsync to print out a message when there was more files in --link-dest than local which could be used to determine there actually was a change.

But I'll add an option (as you suggested) to keep snapshots with 'Full rsync mode' regardless if there were changes or not.

[Mauro (mauromol)]

IMHO if we're unlucky and in full rsync mode there's no way to know if there are changes or not in this particular case, having BIT create a new snapshot every time even if it might be equal to the previous one when full rsync mode is enabled, is a compromise that is far better than having a "wrong" snapshot or suppress the full rsync mode at all.
On the contrary, as I said earlier I would prefer to have a snapshot preserved in any case, independently from this problem.

[Germar (germar)]

I added the above mentioned option (http://bazaar.launchpad.net/~bit-team/backintime/trunk/revision/1072). It is default off.

So with next release 1.1.4 you can use the 'take snapshot regardless of changes' option and I'll try to find a way to be noticed of deleted files for over-next release.

[Mauro (mauromol)]

Hi Germar,
please note that the English phrase for 'take snapshot regardless of changes' has a typo ("reagardless" instead of "regardless"). I couldn't find a way to change it myself through LaunchPad.

[Germar (germar)]

I didn't found any suitable way of detecting deleted files with 'Full rsync mode'. Only way would be to either go back and run a --dry-run first or to create hard-links with 'cp -aRl' again. Which both would make it very slow again.

There is already a bug report for rsync about this https://bugzilla.samba.org/show_bug.cgi?id=5263
and a discussion on mailing list https://<email address hidden>/msg25381.html
Hopefully they will add an option to get informed about deleted files some day.