ResourceWarning for unclosed files in ssh backend

[François Marier]

Using the duplicity 0.8.04 ssh backend on Debian unstable (Python 3.7.5rc1), I get ResourceWarning messages about unclosed files.

Steps to reproduce:

$ mkdir a
$ touch a/file
$ PASSPHRASE=password duplicity --verbosity 1 --no-print-statistics a sftp://hostname/b
/usr/lib/python3/dist-packages/duplicity/backends/ssh_paramiko_backend.py:409: ResourceWarning: unclosed file <_io.TextIOWrapper name='/etc/ssh/ssh_config' mode='r' encoding='UTF-8'>
  sshconfig.parse(open(file))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3/dist-packages/duplicity/backends/ssh_paramiko_backend.py:409: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/francois/.ssh/config' mode='r' encoding='UTF-8'>
  sshconfig.parse(open(file))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3/dist-packages/duplicity/gpg.py:210: ResourceWarning: unclosed file <_io.BufferedWriter name=b'/tmp/user/1000/duplicity-ruepr3me-tempdir/mktemp-ba9ed_b4-2'>
  u'logger': self.logger_fp})
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3/dist-packages/duplicity/gpg.py:210: ResourceWarning: unclosed file <_io.BufferedWriter name=b'/home/francois/.cache/duplicity/8de96ae6a6809eaf1bf3083cfdd34b85/duplicity-full-signatures.20191007T163614Z.sigtar.gpg'>
  u'logger': self.logger_fp})
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3/dist-packages/duplicity/gpg.py:210: ResourceWarning: unclosed file <_io.BufferedWriter name=b'/home/francois/.cache/duplicity/8de96ae6a6809eaf1bf3083cfdd34b85/duplicity-full.20191007T163614Z.manifest.gpg'>
  u'logger': self.logger_fp})
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3/dist-packages/duplicity/dup_main.py:575: ResourceWarning: unclosed file <_io.BufferedReader name=b'/home/francois/.cache/duplicity/8de96ae6a6809eaf1bf3083cfdd34b85/duplicity-full.20191007T163614Z.manifest.part'>
  man_outfp.to_final()
ResourceWarning: Enable tracemalloc to get the object allocation traceback

[Kenneth Loafman (kenneth-loafman)]

Please do a couple of things, if possible:

- Check to see if Python is compiled with DEBUG
- Make sure that duplicity shebang does not have -Wall
- Check your environment for PYTHON* variables

You can check the cflags options with:
$ python-config --cflags

[edso (ed.so)]

Ken,

we should probably still strive to close the file instances. after all these are the underlying issue why we need so high ulimit values. do you agree?

..ede/duply.net

[François Marier (fmarier)]

I didn't see any PYTHON* environment variables (`set | grep PYTHON`).

Here are the Python command-line flags:

  $ python3-config --cflags
  -I/usr/include/python3.7m -I/usr/include/python3.7m -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.7-3r4wIA/python3.7-3.7.5~rc1=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O3 -Wall

The duplicity shebang line only consists of the Python 3 binary:

  $ head -1 /usr/bin/duplicity
  #!/usr/bin/python3

[Kenneth Loafman (kenneth-loafman)]

ede,

Because of the multiple levels of iterators some things don't get closed until the iterators complete. We really should rewrite using a cursor form (like databases), but we've never had the time. Since both of these are paramiko related I'm looking into that first.

...Ken

[Kenneth Loafman (kenneth-loafman)]

Please try pexpect+sftp:// instead of sftp://. That will rule out paramiko.

[Danilo G. Baio (dbaio)]

$ python3.6-config --cflags
-I/usr/local/include/python3.6m -I/usr/local/include/python3.6m -Wno-unused-result -Wsign-compare -Wunreachable-code -O2 -pipe -fstack-protector-strong -fno-strict-aliasing -DNDEBUG

$ head -1 /usr/local/bin/duplicity
#!/usr/local/bin/python3.6

No warnings with pexpect+sftp://

Regards.

[François Marier (fmarier)]

The following doesn't produce any output:

  PASSPHRASE=password duplicity --verbosity 1 --no-print-statistics a pexpect+sftp://hostname/b

Looking at the file timestamps on the remote host, it looks like it worked fine.

[Kenneth Loafman (kenneth-loafman)]

I am moving this over to paramiko since the ResourceWarning is not put out from any other backend.

[François Marier (fmarier)]

Kenneth, are you sure the warnings are coming from paramiko?

Looking at /usr/lib/python3/dist-packages/duplicity/backends/ssh_paramiko_backend.py:409: and /usr/lib/python3/dist-packages/duplicity/gpg.py:210 (both belonging to the duplicity package on my system), it looks like the files are opened by duplicity:

From /usr/lib/python3/dist-packages/duplicity/backends/ssh_paramiko_backend.py:409:

        sshconfig = paramiko.SSHConfig()
        try:
            sshconfig.parse(open(file))

From /usr/lib/python3/dist-packages/duplicity/gpg.py:210:

           p1 = gnupg.run(cmdlist, create_fhs=gnupg_fhs,
                           attach_fhs={u'stdout': encrypt_path.open(u"wb"),
                                       u'stderr': self.stderr_fp,
                                       u'logger': self.logger_fp})

[Kenneth Loafman (kenneth-loafman)]

This should fix the problem. A bug in warnings handling caused all Python warnings to be activated, including those that are disabled by default.