When testing a restore, don't blindly use /tmp

[Kai Engert]

Fedora and other Linux distributions have started to use a "tmpfs" for the /tmp/ directory, which has limited space available.

The new cross-distribution rule seems to be: If your application needs to place large amount of data into a temporary area, then it should use /var/tmp

The proposal here is to change deja-dup to use /var/tmp

I just ran into this myself, when deja-dup attempted to test a restore, and storage in /tmp/ was full (deja-dup wrote 3.7 GB).

However, I have to say, I'm a bit surprised by duplicity's strategy. It attempts to download ALL duplicity-full-signatures files to /tmp, which in my case, is back to 2011 and has 4.8 GB in total. Why isn't it sufficient to download only the most recent full signatures file?

Should I file a deparate bug for duplicity?

[Kai Engert (kaie)]

And if I select "cancel" in deja-dup's progress of the verification - then deja-dup doesn't clean up the /tmp/deja-dup-*/deja-dup directory which has already downloaded 1.7 GB of data...!

[Kai Engert (kaie)]

my version: 24.0

BTW, background on tmpfs in fedora 18: http://fedoraproject.org/wiki/Features/tmp-on-tmpfs

[Kai Engert (kaie)]

I notice you already worked on tmp in 25.5
Also I just found https://bugzilla.redhat.com/show_bug.cgi?id=892063 - will test

[Michael Terry (mterry)]

I'll mark this as a dup of bug 1100092.

Please file a bug against duplicity about the signatures issue.

[Kai Engert (kaie)]

I disagree this bug is a dupe of 1100092.

That one is marked as fixed in 25.5, but I just tried 25.5 and my problem isn't fixed.

deja-dup starts duplicity with the following command:

/usr/bin/python /usr/bin/duplicity list-current-files --gio sftp://user@host-ip/remote-dir --verbosity=9 --gpg-options=--no-use-agent --archive-dir=/tmp/deja-dup-MNBIUW/deja-dup --tempdir=/home/kaie/.cache/deja-dup/tmp --log-fd=22

As you can see, deja dup 25.5 makes the decision to use directory /tmp/deja-dup-MNBIUW/deja-dup - which is the one that will get multiple gigabytes, and eventually abort with "no space left in /"

[Kai Engert (kaie)]

The only place that uses template deja-dup-XXXXXX is in DuplicityJob.vala, which calls DirUtils.make_tmp, which apparently is provided by vala, which is mapped to

http://valadoc.org/#!api=glib-2.0/GLib.DirUtils
"Creates a subdirectory in the preferred directory for temporary files (as returned by get_tmp_dir)"

http://valadoc.org/#!api=glib-2.0/GLib.Environment.get_tmp_dir
"Gets the directory to use for temporary files. This is found from inspecting the environment variables TMPDIR, TMP, and TEMP in that order. If none of those are defined "/tmp" is returned on UNIX and "C:\" on Windows. The encoding of the returned string is system-defined. On Windows, it is always UTF-8. The return value is never null or the empty string."

I assume the calls to DirUtils.make_tmp must be avoided, and deja-dup should set its own path.

[Michael Terry (mterry)]

You're right. While normal duplicity usage is fixed in 25.5, there is a very similar bug when going down the "testing a restore" code path. Working on a fix.

[Kai Engert (kaie)]

I found that get_tempdirs() returns the result of Environment.get_tmp_dir() as the first dir.

On my system, that returns "/tmp"

Did you assume this function only returns something useful if the user has explicitly set a variable? On my syste, no variable related to tmp is set in the environment, but nevertheless the function returns /tmp

[Michael Terry (mterry)]

get_tempdir is the interesting one. It picks the "best" tempdir from among the ones that get_tempdirs returns. What does get_tempdir return?

[Kai Engert (kaie)]

How about this patch?

[Kai Engert (kaie)]

I think you should consider to take the reordering I've proposed in the patch, because you have multiple scenarios in get_tempdirs where you fall back to the first entry.

If you want to allow the environment to override the temp directory, you cannot rely on it to be set, because by default that function will return /tmp - you would have to implement your own strategy to check the environment. Or, if the function uses /tmp/, you shouldn't use it, and use /var/tmp

[Kai Engert (kaie)]

To answer your question from comment 9:

I used a patched build with test output (without the reordering from my patch v2).

Result:
/home/kaie/.cache/deja-dup/tmp

[Kai Engert (kaie)]

Here is another detail for your consideration when working on this patch: Security

If the backup is encrypted - then you shouldn't do the test-restore on the backup media - I assume the test restore will leak unencrypted data into whatever filesystem is being used for testing.

At least on my system, where the backup is on a remote network server, the unencrypted signature files were leaked into the temporary file system.

You might consider to always use the user's home directory for the testing, because that's the only place you can assume to have sufficient protection. For example, on my system, /home is encrypted, but /tmp and /var/tmp are not.

[Michael Terry (mterry)]

get_tempdirs() is doing the right thing. All else considered, we prefer /tmp to /var/tmp because /tmp is cleaned automatically. "All else considered" means everything is on the same partition. The fallback cases you mention, where we throw up our hands and just pick /tmp regardless are really exceptional cases. Things like not being able to query the filesystem ID.

The fact that get_tempdir() returned the correct value for you shows things are working just fine. (it returned a path in your HOME)

Thank you for your patch! But I've already got a branch ready for review [1]. I don't want to re-order get_tempdirs(), but otherwise my patch is similar in spirit to yours.

As for security and restoring into /tmp, we should be fine. When restoring into a temporary directory, the directory should only be readable by the user doing the restore, using good old UNIX permissions. If that is not the case, please file a (separate) bug.

[1] https://code.launchpad.net/~mterry/deja-dup/tempdir-archive-dir

[Michael Terry (mterry)]

(Note that we do clean up after ourselves too. If an operation is stopped, there is code to cleanup the tempdirs used. That same cleanup step is run on startup, in case we crashed or something went terribly wrong last time.)

[Launchpad Janitor (janitor)]

This bug was fixed in the package deja-dup - 26.0-0ubuntu1

---------------
deja-dup (26.0-0ubuntu1) raring; urgency=low

  * New upstream release
    - Only use the GNOME Shell interface when actually in GNOME Shell,
      not something that is merely similar, like the Cinnamon shell
      (LP: #1155929)
    - When testing a restore, also use a temporary directory that is
      on the same partition as the include files (LP: #1159749)
    - Spread default backup time around a bit, from always starting at
      midnight UTC to starting between 2 and 4 AM local time
      (LP: #1154920)
    - Use more-aggressively-idle modes for ionice and nice if available
 -- Michael Terry <email address hidden> Thu, 28 Mar 2013 13:01:53 -0400