tmp directory was removed before duplicity was finished with it

[sirhc]

On Thu, Jan 26, 2012 at 8:04 AM, <email address hidden> <email address hidden> wrote:
Hello - I'm new to Duplicity and I'm likely making a user error, however I've been unable to find a path around this issue. I've built Duplicity 0.6.17 and continue to receive the error message below when I attempt my first full backup using the local file backend. Perhaps the real issue is that my backup config is invalid and is not generating a tmp file, but is exposing a minor bug. At any rate, the same backup with "--dry-run" does not produce the same error. Details below.

Any suggestions? Thanks!

-Chris

debussy> duplicity --verbosity debug /volume1/sirhc file:///volume2/backup/test
Using archive dir: /root/.cache/duplicity/353fa2a1d4f631ce7e04f457ad0ca9bc
Using backup name: 353fa2a1d4f631ce7e04f457ad0ca9bc
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ftpsbackend Succeeded
Import of duplicity.backends.u1backend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.botobackend Failed: No module named py
Import of duplicity.backends.giobackend Failed: No module named gio
Import of duplicity.backends.cloudfilesbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Main action: inc
================================================================================
duplicity 0.6.17 (November 25, 2011)
Args: /opt/bin/duplicity --verbosity debug /volume1/sirhc file:///volume2/backup/test
Linux debussy 2.6.32.12 #1955 SMP Sat Nov 26 14:52:27 CST 2011 x86_64
/opt/bin/python 2.6.7 (r267:88850, Jun 8 2011, 23:10:54)
[GCC 4.2.1]
================================================================================
Using temporary directory /tmp/duplicity-Q73Bx6-tempdir
Registering (mkstemp) temporary file /tmp/duplicity-Q73Bx6-tempdir/mkstemp-qrYQ7V-1
Temp has 520413184 available, backup will use approx 34078720.
Local and Remote metadata are synchronized, no sync needed.
0 files exist on backend
0 files exist in cache
Extracting backup chains from list of files: []
Last full backup date: none
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /root/.cache/duplicity/353fa2a1d4f631ce7e04f457ad0ca9bc

Found 0 secondary backup chains.
No backup chains with active signatures found
No orphaned or incomplete backup sets found.
PASSPHRASE variable not set, asking user.
GnuPG passphrase:
PASSPHRASE variable not set, asking user.
Retype passphrase to confirm:
No signatures found, switching to full backup.
Using temporary directory /root/.cache/duplicity/353fa2a1d4f631ce7e04f457ad0ca9bc/duplicity-sA6J8t-tempdir
Registering (mktemp) temporary file /root/.cache/duplicity/353fa2a1d4f631ce7e04f457ad0ca9bc/duplicity-sA6J8t-tempdir/mktemp-mj6VZN-1
Using temporary directory /root/.cache/duplicity/353fa2a1d4f631ce7e04f457ad0ca9bc/duplicity-2o67Sb-tempdir
Registering (mktemp) temporary file /root/.cache/duplicity/353fa2a1d4f631ce7e04f457ad0ca9bc/duplicity-2o67Sb-tempdir/mktemp-QkpxIF-1
AsyncScheduler: instantiating at concurrency 0
Registering (mktemp) temporary file /tmp/duplicity-Q73Bx6-tempdir/mktemp-bWLIa5-2
Selecting /volume1/sirhc
GPG process 16825 terminated before wait()
Comparing () and None
Getting delta of (() /volume1/sirhc dir) and None
A .
Removing still remembered temporary file /tmp/duplicity-Q73Bx6-tempdir/mktemp-bWLIa5-2
Cleanup of temporary file /tmp/duplicity-Q73Bx6-tempdir/mktemp-bWLIa5-2 failed
Removing still remembered temporary file /tmp/duplicity-Q73Bx6-tempdir/mkstemp-qrYQ7V-1
Cleanup of temporary file /tmp/duplicity-Q73Bx6-tempdir/mkstemp-qrYQ7V-1 failed
Cleanup of temporary directory /tmp/duplicity-Q73Bx6-tempdir failed - this is probably a bug.
Traceback (most recent call last):
 File "/opt/bin/duplicity", line 1377, in <module>
   with_tempdir(main)
 File "/opt/bin/duplicity", line 1370, in with_tempdir
   fn()
 File "/opt/bin/duplicity", line 1345, in main
   full_backup(col_stats)
 File "/opt/bin/duplicity", line 500, in full_backup
   globals.backend)
 File "/opt/bin/duplicity", line 378, in write_multivol
   globals.gpg_profile, globals.volsize)
 File "/opt/lib/python2.6/site-packages/duplicity/gpg.py", line 316, in GPGWriteFile
   bytes_to_go = data_size - get_current_size()
 File "/opt/lib/python2.6/site-packages/duplicity/gpg.py", line 308, in get_current_size
   return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: '/tmp/duplicity-Q73Bx6-tempdir/mktemp-bWLIa5-2'

This is a bug! The directory was removed before duplicity was finished with it.

Please report this to https://bugs.launchpad.net/duplicity.

...Thanks,
...Ken

[Phil Veal (pmveal)]

This still occurs intermittantly on Duplicity 0.6.21 -

Cleanup of temporary directory /tmp/duplicity-kp9c7e-tempdir failed - this is probably a bug.
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1411, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1404, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1374, in main
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 509, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 386, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 327, in GPGWriteFile
    bytes_to_go = data_size - get_current_size()
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 320, in get_current_size
    return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: '/tmp/duplicity-kp9c7e-tempdir/mktemp-lGZzMR-4'

[Phil Veal (pmveal)]

Another example -

Cleanup of temporary directory /tmp/duplicity-Tj_DsZ-tempdir failed - this is probably a bug.
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1411, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1404, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1374, in main
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 509, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 407, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 145, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 171, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 406, in <lambda>
    async_waiters.append(io_scheduler.schedule_task(lambda tdp, dest_filename, vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 307, in put
    tdp.delete()
  File "/usr/lib/python2.7/dist-packages/duplicity/dup_temp.py", line 111, in delete
    path.DupPath.delete(self)
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 568, in delete
    util.ignore_missing(os.unlink, self.name)
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 116, in ignore_missing
    fn(filename)
OSError: [Errno 2] No such file or directory: '/tmp/duplicity-Tj_DsZ-tempdir/mktemp-4j6bzo-3'

It always seems to occur on volume 2; i.e. a rerun shows -

Last full backup left a partial set, restarting.
Last full backup date: Sun Oct 6 09:20:57 2013
RESTART: Volumes 2 to 2 failed to upload before termination.
         Restarting backup at volume 2.

Specifying --asynchronous-upload appears to workaround the problem, or at least make it less likely to occur.

[Matt Giuca (mgiuca)]

I'm having this too on duplicity 0.7.06. I don't think this is a duplicate of #710198 (or if it is, that bug is mis-diagnosed). This doesn't seem to have anything to do with gpg (as shown by Comment #2 having no gpg in the stack trace).

My latest incident of this:

Cleanup of temporary directory /tmp/duplicity-c7h3vY-tempdir failed - this is probably a bug.
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1532, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1526, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1380, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1508, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 662, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 435, in write_multivol
    vi.set_hash("SHA1", gpg.get_hash("SHA1", tdp))
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 434, in get_hash
    fp = path.open("rb")
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 560, in open
    result = open(self.name, mode)
IOError: [Errno 2] No such file or directory: '/tmp/duplicity-c7h3vY-tempdir/mktemp-1BuqO1-6'

OK mine does have gpg.py in the stacktrace, but it's nothing to do with the gpg binary; it's crashing opening a temp file because the directory has been removed.

It looks like the top-level temporary directory (in my case, /tmp/duplicity-c7h3vY-tempdir) is being removed prematurely. I can't tell whether this is a bug in Duplicity or something weird in my system is removing temp directories spontaneously.

I've seen discussion about /tmp being deleted spontaneously by cronjobs, etc:
https://bbs.archlinux.org/viewtopic.php?id=170122
I haven't set up anything like that (in fact this is a brand new clean install of Ubuntu 16.04), but maybe something external to Duplicity is deleting these files.

[tz (te2et)]

Have the same bug on duplicity 0.7.06-2ubuntu2.

Cleanup of temporary directory /tmp/duplicity-z55rH2-tempdir failed - this is probably a bug.
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1532, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1526, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1380, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1501, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 567, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 425, in write_multivol
    at_end = gpg.GPGWriteFile(tarblock_iter, tdp.name, globals.gpg_profile, globals.volsize)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 336, in GPGWriteFile
    bytes_to_go = data_size - get_current_size()
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 328, in get_current_size
    return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: '/tmp/duplicity-z55rH2-tempdir/mktemp-BA1WCM-80'

It appears on every single run, meaning duplicity will never finish a single backup.
I also agree that this is unrelated to gpg not being found as suggested in #710198 and #532244. I have both gpg and gpg2 installed on the system.

$ gpg --version
gpg (GnuPG) 1.4.20

$ gpg2 --version
gpg (GnuPG) 2.1.11
libgcrypt 1.6.5

Ubuntu 16.04.01

[Jeroen (jeroenpraat-deactivatedaccount)]

I had this also and I know what it is.

In Ubuntu you can empty the /tmp dir automatically after ... hour or days. I had it set on 1 hour and that's why Duplicity/Deja Dup was failing for the same as in this bug report. Setting this setting (system config > privacy) to 1 or more days solved it.