0.7.0 fails plain ftp on certificate validation error

Bug #1385599 reported by Remy van Elst on 2014-10-25
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Duplicity
Medium
Unassigned

Bug Description

    Duplicity 0.7.0 fails with a plain FTP backend on certificate validation. It worked with 0.6.24. I installed lftp because 0.7.0 does not use ncftp anymore.

    Command:

        duplicity -v 9 --asynchronous-upload --volsize 25 --tempdir="/tmp" --file-prefix="94e55ec9-1cd9-4988-bf86-68e12b24a935." --name="94e55ec9-1cd9-4988-bf86-68e12b24a935." --exclude-device-files --exclude-globbing-filelist=/etc/backup/exclude.conf --full-if-older-than="2D" --encrypt-key=B5EBDA9B --sign-key=8C12A683 / ftp://user@ipaddr

    Output:

        Using archive dir: /root/.cache/duplicity/94e55ec9-1cd9-4988-bf86-68e12b24a935.
        Using backup name: 94e55ec9-1cd9-4988-bf86-68e12b24a935.
        Import of duplicity.backends.botobackend Succeeded
        Import of duplicity.backends.cfbackend Succeeded
        Import of duplicity.backends.copycombackend Succeeded
        Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
        Import of duplicity.backends.ftpbackend Succeeded
        Import of duplicity.backends.ftpsbackend Failed: the scheme ftps already has a backend associated with it
        Import of duplicity.backends.gdocsbackend Succeeded
        Import of duplicity.backends.giobackend Succeeded
        Import of duplicity.backends.hsibackend Succeeded
        Import of duplicity.backends.imapbackend Succeeded
        Import of duplicity.backends.localbackend Succeeded
        Import of duplicity.backends.megabackend Succeeded
        Import of duplicity.backends.par2backend Succeeded
        Import of duplicity.backends.rsyncbackend Succeeded
        Import of duplicity.backends.sshbackend Succeeded
        Import of duplicity.backends.swiftbackend Succeeded
        Import of duplicity.backends.sxbackend Succeeded
        Import of duplicity.backends.tahoebackend Succeeded
        Import of duplicity.backends.webdavbackend Succeeded
        Import of duplicity.backends.~par2wrapperbackend Succeeded
        LFTP version is 4.3.3
        Using temporary directory /tmp/duplicity-Kvedf4-tempdir
        Registering (mkstemp) temporary file /tmp/duplicity-Kvedf4-tempdir/mkstemp-yOu6kX-1
        Reading globbing filelist /etc/backup/exclude.conf
        Main action: inc
        ================================================================================
        duplicity 0.7.0 (October 23, 2014)
        Args: /usr/local/bin/duplicity -v 9 --asynchronous-upload --volsize 25 --tempdir=/tmp --file-prefix=94e55ec9-1cd9-4988-bf86-68e12b24a935. --name=94e55ec9-1cd9-4988-bf86-68e12b24a935. --exclude-device-files --exclude-globbing-filelist=/etc/backup/exclude.conf --full-if-older-than=2D --encrypt-key=B5EBDA9B --sign-key=8C12A683 / ftp://user@ipaddr
        Linux 1204-php-sql-base-2 3.2.0-69-virtual #103-Ubuntu SMP Tue Sep 2 05:21:29 UTC 2014 x86_64 x86_64
        /usr/bin/python2 2.7.3 (default, Feb 27 2014, 19:58:35)
        [GCC 4.6.3]
        ================================================================================
        Registering (mkstemp) temporary file /tmp/duplicity-Kvedf4-tempdir/mkstemp-Cm3YqW-2
        Temp has 13807632384 available, backup will use approx 60293120.
        Reading results of 'lftp -c 'source /tmp/duplicity-Kvedf4-tempdir/mkstemp-yOu6kX-1;ls ''''
        Backtrace of previous error: Traceback (innermost last):
          File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 368, in inner_retry
            return fn(self, *args)
          File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 567, in list
            return [tobytes(x) for x in self.backend._list()]
          File "/usr/local/lib/python2.7/dist-packages/duplicity/backends/ftpbackend.py", line 104, in _list
            _, l, _ = self.subprocess_popen(commandline)
          File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 488, in subprocess_popen
            (private, result, stdout + '\n' + stderr))
         BackendException: Error running 'lftp -c 'source /tmp/duplicity-Kvedf4-tempdir/mkstemp-yOu6kX-1;ls '''': returned 1, with output:

        ls: Fatal error: Certificate verification: Not trusted

        Attempt 1 failed. BackendException: Error running 'lftp -c 'source /tmp/duplicity-Kvedf4-tempdir/mkstemp-yOu6kX-1;ls '''': returned 1, with output:

        ls: Fatal error: Certificate verification: Not trusted

        Reading results of 'lftp -c 'source /tmp/duplicity-Kvedf4-tempdir/mkstemp-yOu6kX-1;ls ''''
        Backtrace of previous error: Traceback (innermost last):
          File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 368, in inner_retry
            return fn(self, *args)
          File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 567, in list
            return [tobytes(x) for x in self.backend._list()]
          File "/usr/local/lib/python2.7/dist-packages/duplicity/backends/ftpbackend.py", line 104, in _list
            _, l, _ = self.subprocess_popen(commandline)
          File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 488, in subprocess_popen
            (private, result, stdout + '\n' + stderr))
         BackendException: Error running 'lftp -c 'source /tmp/duplicity-Kvedf4-tempdir/mkstemp-yOu6kX-1;ls '''': returned 1, with output:

        ls: Fatal error: Certificate verification: Not trusted

        Attempt 2 failed. BackendException: Error running 'lftp -c 'source /tmp/duplicity-Kvedf4-tempdir/mkstemp-yOu6kX-1;ls '''': returned 1, with output:

        ls: Fatal error: Certificate verification: Not trusted

    Attempts continue to go on and fail.

    The solution for me was to add the following to /etc/lftp.conf:

        set ssl:verify-certificate no

    Then the backup works without issues.

    OS: Ubuntu 12.04 LTS 64 bit
    Duplicity installed from source, 0.7.0
    LFTP version is 4.3.3
    [GCC 4.6.3]
    /usr/bin/python2 2.7.3 (default, Feb 27 2014, 19:58:35)

Remy van Elst (raymii) on 2014-10-25
description: updated
Download full text (20.0 KiB)

It looks like we need to add a manual entry about certificate verification
to the man page, otherwise, duplicity is doing what I think it should,
refusing to work with an untrusted certificate.

NOTE: "Import of duplicity.backends.ftpsbackend Failed: the scheme ftps
already has a backend associated with it". It looks like you may have
installed over the top of the old 0.6 release. Please manually delete
duplicity/backends/ftpsbackend.py to clear the error message.

On Sat, Oct 25, 2014 at 3:05 AM, Remy van Elst <email address hidden>
wrote:

> Public bug reported:
>
> Duplicity 0.7.0 fails with a plain FTP backend on certificate
> validation. It worked with 0.6.24. I installed lftp because 0.7.0 does
> not use ncftp anymore.
>
> Command:
>
> duplicity -v 9 --asynchronous-upload --volsize 25
> --tempdir="/tmp" --file-prefix="94e55ec9-1cd9-4988-bf86-68e12b24a935."
> --name="94e55ec9-1cd9-4988-bf86-68e12b24a935." --exclude-device-files
> --exclude-globbing-filelist=/etc/backup/exclude.conf --full-if-older-
> than="2D" --encrypt-key=B5EBDA9B --sign-key=8C12A683 /
> ftp://user@ipaddr
>
> Output:
>
> Using archive dir:
> /root/.cache/duplicity/94e55ec9-1cd9-4988-bf86-68e12b24a935.
> Using backup name: 94e55ec9-1cd9-4988-bf86-68e12b24a935.
> Import of duplicity.backends.botobackend Succeeded
> Import of duplicity.backends.cfbackend Succeeded
> Import of duplicity.backends.copycombackend Succeeded
> Import of duplicity.backends.dpbxbackend Failed: No module named
> dropbox
> Import of duplicity.backends.ftpbackend Succeeded
> Import of duplicity.backends.ftpsbackend Failed: the scheme ftps
> already has a backend associated with it
> Import of duplicity.backends.gdocsbackend Succeeded
> Import of duplicity.backends.giobackend Succeeded
> Import of duplicity.backends.hsibackend Succeeded
> Import of duplicity.backends.imapbackend Succeeded
> Import of duplicity.backends.localbackend Succeeded
> Import of duplicity.backends.megabackend Succeeded
> Import of duplicity.backends.par2backend Succeeded
> Import of duplicity.backends.rsyncbackend Succeeded
> Import of duplicity.backends.sshbackend Succeeded
> Import of duplicity.backends.swiftbackend Succeeded
> Import of duplicity.backends.sxbackend Succeeded
> Import of duplicity.backends.tahoebackend Succeeded
> Import of duplicity.backends.webdavbackend Succeeded
> Import of duplicity.backends.~par2wrapperbackend Succeeded
> LFTP version is 4.3.3
> Using temporary directory /tmp/duplicity-Kvedf4-tempdir
> Registering (mkstemp) temporary file
> /tmp/duplicity-Kvedf4-tempdir/mkstemp-yOu6kX-1
> Reading globbing filelist /etc/backup/exclude.conf
> Main action: inc
>
> ================================================================================
> duplicity 0.7.0 (October 23, 2014)
> Args: /usr/local/bin/duplicity -v 9 --asynchronous-upload
> --volsize 25 --tempdir=/tmp
> --file-prefix=94e55ec9-1cd9-4988-bf86-68e12b24a935.
> --name=94e55ec9-1cd9-...

Remy van Elst (raymii) wrote :

The point was that for a plain FTP server there is no certificate. I would expect this error when I connect to this server over FTPS, but this is plain FTP, no certificates involved.

uo (uo-li) wrote :

I have the same issue. 0.7.0 installed via PPA:
https://launchpad.net/~duplicity-team/+archive/ubuntu/ppa
Plain FTP.

@Kenneth
duplicity/backends/ftpsbackend.py does not exist

"sudo apt-get purge duplicity" and reinstall does not help.

NOTE:
Only one of my Ubuntu 12.04 machines is concerned (the one which does its backup to a QNAP). The other one (talking to ProFTP on a Raspberry Pi) still works fine.

uo (uo-li) wrote :

Contrary to ncftp, lftp wants to use ssl if it is offered by the server which makes some sense.
duplicity option "--ssl-no-check-certificate" does not make any change.

WORKAROUNDS:
-switch off ssl support in your FTP server if you can if you don't need it for anything else
-add your self-signed certificate to lftp via its config file:
http://www.versatilewebsolutions.com/blog/2014/04/lftp-ftps-and-certificate-verification.html (not tested)
-disable lftp certificate verification ("set ssl:verify-certificate no" in the config file)

Unfortunately I wasn't able to disable SSL in LFTP. "ftp:ssl-allow false" in lftp config file does not make any change.

Not a really bad bug but still unexpected behaviour with the change to lftp for people using plain ftp whose servers are offering ssl and have a self-signed certificate.

uo (uo-li) wrote :

SUGGESTION:
-create an duplicity option that tells the lftp backend to use plain FTP even if SSL/TLS is offered by the server
(some people may choose plain FTP in a safe environment because of the speed increase)

Yannick (yannick-molinet) wrote :

HI all,

I have the same issue.
in /etc/lftp.conf, I add set ssl:verify-certificate no

but I can see that adding set ftp:ssl-allow false in /etc/lftp.conf has no effect because the file command generate by duplicity contain s set ftp:ssl-allow true in all case (even if ftp address is ftp:// and not ftps://)

I'm trying to send file manually with lftp on a no ssl ftp server, and it's working as fine.

Yannick (yannick-molinet) wrote :

I'm suggest this modification in ftpbackend.py :

        self.tempfile, self.tempname = tempdir.default().mkstemp()
        if self.parsed_url.scheme == 'ftps':
            os.write(self.tempfile, "set ftp:ssl-allow true\n")
            os.write(self.tempfile, "set ftp:ssl-protect-data true\n")
            os.write(self.tempfile, "set ftp:ssl-protect-list true\n")
        else:
            os.write(self.tempfile, "set ftp:ssl-allow false\n")
        os.write(self.tempfile, "set net:timeout %s\n" % globals.timeout)
        os.write(self.tempfile, "set net:max-retries %s\n" % globals.num_retries)
        os.write(self.tempfile, "set ftp:passive-mode %s\n" % self.conn_opt)
        os.write(self.tempfile, "open %s %s\n" % (self.portflag, self.parsed_url.hostname))

it's working as fine.

Changed in duplicity:
milestone: none → 0.7.01
importance: Undecided → Medium
status: New → Fix Committed
Changed in duplicity:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers