Filelist (non-globbing) should include a folder if it contains higher priority included files

[Aaron Whitehouse]

Using the --include/--exclude commandline options, or either an exclude or include globbing filelist, with the following instructions:
+ testfiles/select/1/2/1
- testfiles/select/1/2
- testfiles/select/1/1
- testfiles/select/1/3

will include the folder 1/2 because it contains an included folder (1/2/1).

By contrast, using --include-filelist or --exclude-filelist with the same comments will not include anything. I'll upload some test cases to show this issue.

[Aaron Whitehouse (aaron-whitehouse)]

I note that this was explicitly flagged in the manual (see point 2):
"The --include-filelist, --exclude-filelist, --include-filelist-stdin, and --exclude-filelist-stdin options also introduce file selection conditions. They direct duplicity to read in a file, each line of which is a file specification, and to include or exclude the matching files. Lines are separated by newlines or nulls, depending on whether the --null-separator switch was given. Each line in a filelist is interpreted similarly to the way extended shell patterns are, with a few exceptions:

1. Globbing patterns like *, **, ?, and [...] are not expanded.
2. Include patterns do not match files in a directory that is included. So /usr/local in an include file will not match /usr/local/doc. #
"

But I maintain that this should be changed. It seems a bizarre distinction to have between normal and globbing filelists. There is some risk that changing this could mean users have additional subfolders backed up that were not previously included, but this seems less of an issue than the potential that desired subfolders are mistakenly excluded.

[Kenneth Loafman (kenneth-loafman)]

I think you are correct. I thought we were going to fold the globbing and
non-globbing file lists into one.

On Thu, Mar 12, 2015 at 12:39 PM, Aaron Whitehouse <
<email address hidden>> wrote:

> I note that this was explicitly flagged in the manual (see point 2):
> "The --include-filelist, --exclude-filelist, --include-filelist-stdin, and
> --exclude-filelist-stdin options also introduce file selection conditions.
> They direct duplicity to read in a file, each line of which is a file
> specification, and to include or exclude the matching files. Lines are
> separated by newlines or nulls, depending on whether the --null-separator
> switch was given. Each line in a filelist is interpreted similarly to the
> way extended shell patterns are, with a few exceptions:
>
> 1. Globbing patterns like *, **, ?, and [...] are not expanded.
> 2. Include patterns do not match files in a directory that is included. So
> /usr/local in an include file will not match /usr/local/doc. #
> "
>
> But I maintain that this should be changed. It seems a bizarre
> distinction to have between normal and globbing filelists. There is some
> risk that changing this could mean users have additional subfolders
> backed up that were not previously included, but this seems less of an
> issue than the potential that desired subfolders are mistakenly
> excluded.
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1408411
>
> Title:
> Filelist (non-globbing) should include a folder if it contains higher
> priority included files
>
> Status in Duplicity - Bandwidth Efficient Encrypted Backup:
> New
>
> Bug description:
> Using the --include/--exclude commandline options, or either an exclude
> or include globbing filelist, with the following instructions:
> + testfiles/select/1/2/1
> - testfiles/select/1/2
> - testfiles/select/1/1
> - testfiles/select/1/3
>
> will include the folder 1/2 because it contains an included folder
> (1/2/1).
>
> By contrast, using --include-filelist or --exclude-filelist with the
> same comments will not include anything. I'll upload some test cases
> to show this issue.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1408411/+subscriptions
>

[Aaron Whitehouse (aaron-whitehouse)]

Yes, thanks Kenneth. I am about to submit a merge request doing that, but wanted to make sure you were aware that this will be a change of behaviour.