Globbing patterns fail to include some files if include contains "*" or "**"

Bug #884371 reported by Elifarley Cruz on 2011-10-31
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Duplicity
Medium
Unassigned

Bug Description

Duplicity doesn't include some files if "**" is used as prefix.

Duplicity 0.6.15
Python 2.7.2
ArchLinux x64
Linux filesystem

In the example below, I would expect to have all 3 files from folder-c included in the backup, but they are not:

--------------------------------------------------

BASE=/tmp
mkdir -p $BASE/test-dup/folder-{a,b,c}

touch $BASE/test-dup/file-{1.txt,2.jpg,3.tmp}
touch $BASE/test-dup/folder-a/file-{4.txt,5.jpg,6.tmp}
touch $BASE/test-dup/folder-b/file-{7.txt,8.jpg,9.tmp}
touch $BASE/test-dup/folder-c/file-{10.txt,11.jpg,12.tmp}

tree /tmp/test-dup/
# result:
/tmp/test-dup/
+-- file-1.txt
+-- file-2.jpg
+-- file-3.tmp
+-- folder-a
¦ +-- file-4.txt
¦ +-- file-5.jpg
¦ +-- file-6.tmp
+-- folder-b
¦ +-- file-7.txt
¦ +-- file-8.jpg
¦ +-- file-9.tmp
+-- folder-c
    +-- file-10.txt
    +-- file-11.jpg
    +-- file-12.tmp

cat my-file-list
# result:
- **/test-dup/folder-a
**/test-dup/folder-b/*.txt
- **/test-dup/folder-b

duplicity --name test-dup --include-globbing-filelist my-file-list
$BASE/test-dup/ file://$BASE/test-dup-backup

duplicity --name test-dup list-current-files file://$BASE/test-dup-backup
# result:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Oct 31 13:31:33 2011
Mon Oct 31 13:15:04 2011 .
Mon Oct 31 13:15:04 2011 folder-b
Mon Oct 31 13:15:04 2011 folder-b/file-7.txt

Related branches

description: updated
Elifarley Cruz (elifarley) wrote :

Here's some actual test code to show the bug.
This bug only happens when I use '**' as prefix, instead of 'testfiles'. Please see below for details.

This method already exists in selectiontest.py, and it works just fine:
    def test_globbing_filelist(self):
        """Filelist glob test similar to above testParse2"""
        self.ParseTest([("--include-globbing-filelist", "file")],
                       [(), ('1',), ('1', '1'), ('1', '1', '2'),
                        ('1', '1', '3')],
                       ["""
- testfiles/select/1/1/1
testfiles/select/1/1
- testfiles/select/1
- **
"""])

I have added this method, which also works:

    def test_globbing_filelist_bug_884371_a(self):
        """Filelist glob test related to bug 884371"""
        self.ParseTest([("--include-globbing-filelist", "file")],
                       [(), ('1',), ('1', '1'), ('1', '1', '1'),
                        ('1', '1', '2'), ('1', '1', '3')],
                       ["""
**/select/1/1
- **
"""])

The one below doesn't work, and it's simply a modified version of 'test_globbing_filelist' (with the string 'testfiles' replaced by '**'):

    def test_globbing_filelist_bug_884371_b(self):
        """
  Modified version of 'test_globbing_filelist'
  (with the string 'testfiles' replaced by '**')
  """
        self.ParseTest([("--include-globbing-filelist", "file")],
                       [(), ('1',), ('1', '1'), ('1', '1', '2'),
                        ('1', '1', '3')],
                       ["""
- **/select/1/1/1
**/select/1/1
- **/select/1
- **
"""])

Here are the results of the tests:

========================================
test_globbing_filelist (__main__.ParseArgsTest)
Filelist glob test similar to above testParse2 ...
self.Select:
()
('1',)
('1', '1')
('1', '1', '2')
('1', '1', '3')
ok
test_globbing_filelist_bug_884371_a (__main__.ParseArgsTest)
Filelist glob test related to bug 884371 ...
self.Select:
()
('1',)
('1', '1')
('1', '1', '1')
('1', '1', '2')
('1', '1', '3')
ok
test_globbing_filelist_bug_884371_b (__main__.ParseArgsTest)
Filelist glob test related to bug 884371 ...
self.Select:
()
End when i2 = ('1',)
FAIL
========================================

summary: - Globbing patterns fail to include some files
+ Globbing patterns fail to include some files if prefix is "**"
description: updated
Changed in duplicity:
importance: Undecided → Medium
milestone: none → 0.6.18
status: New → Triaged
Changed in duplicity:
milestone: 0.6.18 → none

I've just added a bunch of test cases to the linked branch to dig into this in more detail.

It looks to me as though neither * nor ** are processed correctly if they are included anywhere in an include line, whether this is in a globbing-filelist or in an --include on the commandline.

* and ** appear to work as expected if they are in an exclude line of a filelist or in an --exclude on the commandline.

summary: - Globbing patterns fail to include some files if prefix is "**"
+ Globbing patterns fail to include some files if include contains "**"
summary: - Globbing patterns fail to include some files if include contains "**"
+ Globbing patterns fail to include some files if include contains "*" or
+ "**"
Changed in duplicity:
assignee: nobody → Aaron Whitehouse (aaron-whitehouse)
Changed in duplicity:
assignee: Aaron Whitehouse (aaron-whitehouse) → nobody
milestone: none → 0.7.04
status: Triaged → Fix Committed
Changed in duplicity:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers