--exclude-filelist-stdin and gpg error with/without PASSPHRASE

[Raphaël Droz]

$ duplicity full --exclude-filelist-stdin ~/mydir file:///tmp/backup < /tmp/ex
Reading filelist standard input
Sorting filelist standard input
Warning: Problem with getpass. Passwords may be echoed.
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1236, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1229, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1121, in main
    globals.gpg_profile.passphrase = get_passphrase(1, action)
  File "/usr/bin/duplicity", line 128, in get_passphrase
    pass1 = getpass.getpass("GnuPG passphrase: ")
  File "/usr/lib/python2.5/getpass.py", line 30, in unix_getpass
    return default_getpass(prompt)
  File "/usr/lib/python2.5/getpass.py", line 71, in default_getpass
    return _raw_input(prompt, stream)
  File "/usr/lib/python2.5/getpass.py", line 82, in _raw_input
    line = sys.stdin.readline()
ValueError: I/O operation on closed file

-------------------------------
/tmp/ex contains a full path filename list

ii duplicity 0.6.06-2 encrypted bandwidth-efficient backup
ii gnupg 1.4.10-2 GNU privacy guard - a free PGP replacement
ii gnupg-agent 2.0.14-1 GNU privacy guard - password agent
ii gnupg-curl 1.4.10-2 GNU privacy guard - a free PGP replacement (cURL)
ii gnupg2 2.0.14-1 GNU privacy guard - a free PGP replacement (new v2.x)
ii python 2.5.4-9 An interactive high-level object-oriented language (default version)

Distributor ID: Debian
Description: Debian GNU/Linux unstable (sid)
Release: unstable
Codename: sid
----------------------------------------
with :
$ env PASSPHRASE=mypass duplicity full -v 9 --exclude-filelist-stdin ~/mydir file:///tmp/zob < /tmp/ex
Connecting with backend: LocalBackend
Archive dir: /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33

Found 0 secondary backup chains.
No backup chains with active signatures found
No orphaned or incomplete backup sets found.
Using temporary directory /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33/duplicity--kz87X-tempdir
Registering (mktemp) temporary file /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33/duplicity--kz87X-tempdir/mktemp-SrdnpI-1
Using temporary directory /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33/duplicity-lIeqQ6-tempdir
Registering (mktemp) temporary file /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33/duplicity-lIeqQ6-tempdir/mktemp-eEKSgM-1
AsyncScheduler: instantiating at concurrency 0
Registering (mktemp) temporary file /tmp/duplicity-80nCC_-tempdir/mktemp-A35XlL-2
Removing still remembered temporary file /tmp/duplicity-80nCC_-tempdir/mkstemp-p9T8lK-1
Selecting /home/user/mydir
Comparing () and None
Removing still remembered temporary file /tmp/duplicity-80nCC_-tempdir/mktemp-A35XlL-2
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1236, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1229, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1202, in main
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 416, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 294, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 275, in GPGWriteFile
    file = GPGFile(True, path.Path(filename), profile)
  File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 128, in __init__
    'logger': self.logger_fp})
  File "/usr/lib/python2.5/site-packages/duplicity/GnuPGInterface.py", line 365, in run
    create_fhs, attach_fhs)
  File "/usr/lib/python2.5/site-packages/duplicity/GnuPGInterface.py", line 415, in _attach_fork_exec
    if process.pid == 0: self._as_child(process, gnupg_commands, args)
  File "/usr/lib/python2.5/site-packages/duplicity/GnuPGInterface.py", line 437, in _as_child
    os.dup2( p.child, getattr(sys, "__%s__" % std).fileno() )
ValueError: I/O operation on closed file

Getting delta of (() /home/user/mydir dir) and None
A .
Removing still remembered temporary file /tmp/duplicity-80nCC_-tempdir/mkstemp-p9T8lK-1
Cleanup of temporary file /tmp/duplicity-80nCC_-tempdir/mkstemp-p9T8lK-1 failed
Removing still remembered temporary file /tmp/duplicity-80nCC_-tempdir/mktemp-A35XlL-2
Cleanup of temporary file /tmp/duplicity-80nCC_-tempdir/mktemp-A35XlL-2 failed
Cleanup of temporary directory /tmp/duplicity-80nCC_-tempdir failed - this is probably a bug.
Removing still remembered temporary file /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33/duplicity--kz87X-tempdir/mktemp-SrdnpI-1
Removing still remembered temporary file /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33/duplicity-lIeqQ6-tempdir/mktemp-eEKSgM-1
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1236, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1229, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1202, in main
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 416, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 294, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 278, in GPGWriteFile
    bytes_to_go = data_size - get_current_size()
  File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 270, in get_current_size
    return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: '/tmp/duplicity-80nCC_-tempdir/mktemp-A35XlL-2'

Removing still remembered temporary file /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33/duplicity--kz87X-tempdir/mktemp-SrdnpI-1
Cleanup of temporary file /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33/duplicity--kz87X-tempdir/mktemp-SrdnpI-1 failed
Cleanup of temporary directory /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33/duplicity--kz87X-tempdir failed - this is probably a bug.
Removing still remembered temporary file /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33/duplicity-lIeqQ6-tempdir/mktemp-eEKSgM-1
Cleanup of temporary file /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33/duplicity-lIeqQ6-tempdir/mktemp-eEKSgM-1 failed
Cleanup of temporary directory /home/user/.cache_xdg/duplicity/d42961f638a242a18f0b898e8969ae33/duplicity-lIeqQ6-tempdir failed - this is probably a bug

[Raphaël Droz (raphael-droz)]

Still reproducible with duplicity 0.6.11
AFAICT there should be two attempts to remove
/tmp/duplicity-v2lHq9-tempdir/mkstemp-oVge5k-1

gnupg 2.0.17
py-gnupg 0.3.2
pexpect 2.4
librsync 0.9.7
python 2.6.6-r2

[Andreas Rottmann (rotty)]

With duplicity 0.6.13, when --exclude-filelist-stdin is used,
duplicity still crashes with essentially the same backtrace as
originally reported in this bug. My take one this is the following:
since sys.stdin is closed by the filelist handling code, invoking GPG
subsequently fails. I've attached a patch that should fix this; in
the following is a short testcase exhibiting the symptoms of the bug
(for 0.6.13):

  $ PASSPHRASE=test /usr/bin/duplicity --exclude-filelist-stdin etc file:///tmp/test-duplicity
  Reading filelist standard input
  etc/foobar
  Sorting filelist standard input
  Local and Remote metadata are synchronized, no sync needed.
  Last full backup date: none
  No signatures found, switching to full backup.
  Cleanup of temporary directory /tmp/user/1000/duplicity-F5wmhD-tempdir failed - this is probably a bug.
  Traceback (most recent call last):
    File "/usr/bin/duplicity", line 1262, in <module>
      with_tempdir(main)
    File "/usr/bin/duplicity", line 1255, in with_tempdir
      fn()
    File "/usr/bin/duplicity", line 1233, in main
      full_backup(col_stats)
    File "/usr/bin/duplicity", line 417, in full_backup
      globals.backend)
    File "/usr/bin/duplicity", line 295, in write_multivol
      globals.gpg_profile, globals.volsize)
    File "/usr/lib/python2.6/dist-packages/duplicity/gpg.py", line 272, in GPGWriteFile
      file = GPGFile(True, path.Path(filename), profile)
    File "/usr/lib/python2.6/dist-packages/duplicity/gpg.py", line 125, in __init__
      'logger': self.logger_fp})
    File "/usr/lib/pymodules/python2.6/GnuPGInterface.py", line 357, in run
      create_fhs, attach_fhs)
    File "/usr/lib/pymodules/python2.6/GnuPGInterface.py", line 401, in _attach_fork_exec
      if process.pid == 0: self._as_child(process, gnupg_commands, args)
    File "/usr/lib/pymodules/python2.6/GnuPGInterface.py", line 423, in _as_child
      os.dup2( p.child, getattr(sys, "__%s__" % std).fileno() )
  ValueError: I/O operation on closed file

  Traceback (most recent call last):
    File "/usr/bin/duplicity", line 1262, in <module>
      with_tempdir(main)
    File "/usr/bin/duplicity", line 1255, in with_tempdir
      fn()
    File "/usr/bin/duplicity", line 1233, in main
      full_backup(col_stats)
    File "/usr/bin/duplicity", line 417, in full_backup
      globals.backend)
    File "/usr/bin/duplicity", line 295, in write_multivol
      globals.gpg_profile, globals.volsize)
    File "/usr/lib/python2.6/dist-packages/duplicity/gpg.py", line 275, in GPGWriteFile
      bytes_to_go = data_size - get_current_size()
    File "/usr/lib/python2.6/dist-packages/duplicity/gpg.py", line 267, in get_current_size
      return os.stat(filename).st_size
  OSError: [Errno 2] No such file or directory: '/tmp/user/1000/duplicity-F5wmhD-tempdir/mktemp-bkM8tn-2'

  Cleanup of temporary directory /home/rotty/.cache/duplicity/d6307cd60a297c3d0919e0fe183d8a27/duplicity-9uDzeS-tempdir failed - this is probably a bug.
  Cleanup of temporary directory /home/rotty/.cache/duplicity/d6307cd60a297c3d0919e0fe183d8a27/duplicity-LYkWP_-tempdir failed - this is probably a bug.

[Raphaël Droz (raphael-droz)]

Thanks a lot Andreas, your patch solved the bug in my case !
(tested against 0.6.13)