exclude-globbing-filelist does not backup correct files with --progress option

Bug #1264744 reported by Jorman Franzini on 2013-12-28
30
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Duplicity
Medium
Unassigned

Bug Description

Hi, tnx for duplicity.

I'm trying to use duplicity with duply, and I want to use the exclude-globbing-filelist.

For testing purpose I created a bk dir, this is the structure
/home/jorman/Scrivania/bk
|-- file1
|-- file2
|-- file3
|-- subfolder1
| `-- file4
| `-- file5
| `-- file6
|-- subfolder2
| `-- file7
| `-- file8
| `-- file9

Each folder contains 3 files, so 9 in total

My exclusion list is that
- /home/jorman/Scrivania/bk/1

The command line is
export PASSPHRASE=XXX
duplicity --encrypt-key "YYY" --dry-run --verbosity 9 --full-if-older-than 1M --progress --exclude-globbing-filelist /home/jorman/.duply/test/exclude /home/jorman/Scrivania/bk file:///home/jorman/Scrivania/test

Now, the result log report that
--------------[ Statistiche del backup ]--------------
StartTime 1388264800.61 (Sat Dec 28 22:06:40 2013)
EndTime 1388264800.62 (Sat Dec 28 22:06:40 2013)
ElapsedTime 0.01 (0.01 seconds)
SourceFiles 21
SourceFileSize 12288 (12.0 KB)
NewFiles 12
NewFileSize 12288 (12.0 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 12
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 0 (0 bytes)
Errors 0
------------------------------------------------------

As you can see Duplicity try to add 12 files, but if I exclude the dir 1 the files must be only 8

Now I read the duplicity file selection, and if I decided to expand the argument so the command line is that
duplicity --encrypt-key "YYY" --dry-run --verbosity 9 --full-if-older-than 1M --progress --exclude /home/jorman/Scrivania/bk/1 /home/jorman/Scrivania/bk file:///home/jorman/Scrivania/test

And the result log is
--------------[ Statistiche del backup ]--------------
StartTime 1388265307.57 (Sat Dec 28 22:15:07 2013)
EndTime 1388265307.57 (Sat Dec 28 22:15:07 2013)
ElapsedTime 0.00 (0.00 seconds)
SourceFiles 14
SourceFileSize 8192 (8.00 KB)
NewFiles 8
NewFileSize 8192 (8.00 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 8
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 0 (0 bytes)
Errors 0
------------------------------------------------------

That is exact

What I wrong?

J

Related branches

Download full text (3.2 KiB)

Hi,
I think this problem is result from reading twice filelist file.

For example:
~/tmp/test-> tree -a in
in
├── .config
│   └── test1-in-config.txt
├── test2.txt
└── test3.txt

~/tmp/test-> cat filelist.txt
- /home/k/tmp/test/in/.config

duplicity -vi --progress --no-encryption --exclude-globbing-filelist filelist.txt /home/k/tmp/test/in/ file:///home/k/tmp/test/out/
Using archive dir: /home/k/.cache/duplicity/b5e2f4b829db69341672537d1ced3377
Using backup name: b5e2f4b829db69341672537d1ced3377
[....]
Reading globbing filelist filelist.txt

[******** First reading file list]

Main action: inc
================================================================================
duplicity 0.6.23 (January 24, 2014)
Args: /usr/bin/duplicity -vi --progress --no-encryption --exclude-globbing-filelist filelist.txt /home/k/tmp/test/in/ file:///home/k/tmp/test/out/
Linux kntbk 3.13-1-686-pae #1 SMP Debian 3.13.5-1 (2014-03-04) i686
/usr/bin/python 2.7.6 (default, Feb 26 2014, 06:42:04)
[GCC 4.8.2]
================================================================================
Using temporary directory /tmp/duplicity-iljo9e-tempdir
Temp has 47721136128 available, backup will use approx 34078720.
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
A .
A test2.txt
A test3.txt

[******** In this point file list is correct, below - second reading filelist ]

Reading globbing filelist filelist.txt
Using temporary directory /home/k/.cache/duplicity/b5e2f4b829db69341672537d1ced3377/duplicity-HXwYmn-tempdir
Using temporary directory /home/k/.cache/duplicity/b5e2f4b829db69341672537d1ced3377/duplicity-wbRfbF-tempdir
AsyncScheduler: instantiating at concurrency 0
A .
A .config
A .config/test1-in-config.txt
A test2.txt
A test3.txt

[******** in this point exclude list is empty]
[...]

IMHO - file with exclude list is simple not reopened.

Similar problem:
duplicity --progress --no-encryption --exclude-filelist filelist.txt /home/k/tmp/test/in/ file:///home/k/tmp/test/out/

Reading filelist filelist.txt
Sorting filelist filelist.txt
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Mar 13 18:26:27 2014
Reading filelist filelist.txt
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1495, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1489, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1338, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1471, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 623, in incremental_backup
    commandline.set_selection()
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 891, in set_selection
    sel.ParseArgs(select_opts, select_files)
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", line 235, in ParseArgs
    filelists[filelists_index], 0, arg))
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", line 313, in filelist_get_sf
    self.filelist_read(filelist_fp, inc_default, filelist_name)
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", l...

Read more...

Can you please check to see if there is a trailing space at the end of the line in your filelist?

Scott McKenzie (noizyland) wrote :
Download full text (7.1 KiB)

I am experiencing the same problem and I've found it is triggered by the --progress flag.

The following demonstrates the problem:
location=~/duplicity-test
mkdir -p "$location"/{.cache,src,dst}
mkdir -p "$location"/src/{dir1,dir2}/{subdir1,subdir2}
find "$location/src" -type d -exec touch {}/testfile \;
echo "- $location/src/dir2/subdir2" > "$location/exclude"
duplicity --archive-dir "$location/.cache" --name test_backup --no-encryption --verbosity 5 --exclude-globbing-filelist "$location/exclude" "$location/src" file://"$location/dst"
Using archive dir: /home/scott/duplicity-test/.cache/test_backup
Using backup name: test_backup
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.copycombackend Succeeded
Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Reading globbing filelist /home/scott/duplicity-test/exclude
Main action: inc
================================================================================
duplicity 0.7.0 ($reldate)
Args: /usr/bin/duplicity --archive-dir /home/scott/duplicity-test/.cache --name test_backup --no-encryption --verbosity 5 --exclude-globbing-filelist /home/scott/duplicity-test/exclude /home/scott/duplicity-test/src file:///home/scott/duplicity-test/dst
Linux ddsmash 3.16.0-28-generic #38-Ubuntu SMP Sat Dec 13 16:13:28 UTC 2014 x86_64 x86_64
/usr/bin/python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2]
================================================================================
Using temporary directory /tmp/duplicity-gzjWQ7-tempdir
Temp has 35840032768 available, backup will use approx 34078720.
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
Using temporary directory /home/scott/duplicity-test/.cache/test_backup/duplicity-xqRenW-tempdir
Using temporary directory /home/scott/duplicity-test/.cache/test_backup/duplicity-y6w8cZ-tempdir
AsyncScheduler: instantiating at concurrency 0
A .
A dir1
A dir1/subdir1
A dir1/subdir1/testfile
A dir1/subdir2
A dir1/subdir2/testfile
A dir1/testfile
A dir2
A dir2/subdir1
A dir2/subdir1/testfile
A dir2/testfile
A testfile
AsyncScheduler: running task synchronously (asynchronicity disabled)
Writing duplicity-full.20150108T185804Z.vol1.difftar.gz
Deleting /tmp/duplicity-gzjWQ7-tempdir/mktemp-LZ1B2R-2
AsyncScheduler: task completed successfully
Processed volume 1
--------------[ Backup...

Read more...

Thanks both for such a comprehensive report/comments. Much to my surprise, --progress does seem to cause the --exclude-globbing-filelist to backup the incorrect files. I have added an automated test showing this behaviour in the linked branch:
lp:~aaron-whitehouse/duplicity/progress_option_error

Jorman, can you please run your example again without the --progress option and see if this makes any difference?

summary: - exclude-globbing-filelist don't works
+ exclude-globbing-filelist does not backup correct files with --progress
+ option

This bug happens because in the first dry-run, Selection closes the filelist, so it can't read again in the second, real run.
I needed a quick fix for this, so I created a function that resets select_files in commandline.py and called it between each run in duplicity.

Changed in duplicity:
importance: Undecided → Medium
milestone: none → 0.7.04
status: New → Fix Committed
Changed in duplicity:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers