rev 1515: TypeError: can't concat str to bytes

[Mark Grandi]

due to this bug (https://bugs.launchpad.net/duplicity/+bug/1852876) i tried to download the latest trunk and run that, but now i am getting "TypeError: can't concat str to bytes"

i am running the latest revision (as of this point), 1515, with the only thing changed being the version number manually changed to 0.8.07.1 and changing the shebang line for the duplicity script that gets put into ~/.local/bin/duplicity to be python3 instead of python (seems to be a bug since you changed it in a recent revision i noticed)

i installed duplicity by doing `bzr branch`, editing the files above, `python3 setup.py bdist_wheel` and then `pip3 install <whatever>.whl`

the command line is:

/usr/bin/time duplicity incremental --encrypt-sign-key "<KEY>" --log-file "/mnt/c/Users/auror/Temp/Logs/duplicity/duplicity_log_file_$(date --utc '+%FT%H_%M_%S.%N%z').log" --name "mgrandi_lenovo_x1_carbon_win10_20191120" --exclude-device-files --progress --progress-rate 60 --volsize 1024 --file-prefix "mgrandi_lenovo_x1_carbon_win10_20191120-" --exclude-filelist "/mnt/c/Users/auror/duplicity_exclude_list_linux_filepaths.txt" "/mnt/c/Users/auror/" "dpbx:///Backups/duplicity/mgrandi_lenovo_x1_carbon_win10_20191120/"

see attached log file

.

[Kenneth Loafman (kenneth-loafman)]

What OS version? What version of librsync?

$ apt show librsync1

-or-

$ apt show librsync2

-or-

your distro's equivalent.

[Mark Grandi (markgrandi)]

It is ubuntu 18.10 running on windows 10 WSL, the uname string:

Linux <HOSTNAME> 4.4.0-18362-Microsoft #476-Microsoft Fri Nov 01 16:53:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

this did work in a previous version of duplicity (0.8.06) that i downloaded from the website

python3 version:

~$ python3 -VV Python 3.6.8 (default, Oct 7 2019, 12:59:55) [GCC 8.3.0]

since i compiled it from the source code, i did not install librsync from apt, but it compiled it using the setup.py script, the location is:

/home/mgrandi/.local/lib/python3.6/site-packages/duplicity/_librsync.cpython-36m-x86_64-linux-gnu.so

i am testing a previously installed version (0.8.6) to make sure that works still, and the problem is just 0.8.7+

[Mark Grandi (markgrandi)]

and the backup just finished, version 0.8.6 works fine

[Kenneth Loafman (kenneth-loafman)]

Try this:

$ PYTHONPATH='.' duplicity incremental --encrypt-sign-key "<KEY>"

Looks like the current directory is not in the path. It should be.

BTW, the normal install is '$ sudo setup.py install'. I'm wondering if the bdist_wheel setup is causing a problem.

[Mark Grandi (markgrandi)]

i use bdist_wheel because whatever the normal install does is not hooking into the normal packaging framework , it shows up in `pip3 list` but then when you try and uninstall it, it doesn't remove the entire installation, which is annoying to manually go through and remove the files one by one, and has the benefit of having the librsync shared library included (for uploading onto pypi for example.)

Compare the removal when using a wheel vs the normal install: https://gist.github.com/mgrandi/c8d79e8e62de162c07fe6705d039c53a

anyway, i did some debugging and it seems that its encountering a locked file, or there is some interaction with a file being locked on windows and a linux process via WSL is trying to access it but can't

the full error message (i printed out the full exception):

Error Invalid argument getting delta for /mnt/c/Users/auror/.android/metrics/spool/3e68d120-cc5a-4d7c-a4b0-131782ec79bd.trk, [Errno 22] Invalid argument

i can see in Process Explorer that android studio indeed has that file open

so maybe duplicity is just expecting 1 type of exception but in this case its a different type, so it falls through and doesn't use the normal "file is locked" code path?

[Kenneth Loafman (kenneth-loafman)]

Looked at WSL. Not a virtual machine. Have you tried this under a VM? Docker? VirtualBox works well and provides a real Linux environment, not an emulation.

[Mark Grandi (markgrandi)]

WSL is not emulation, and is correctly running ELF binaries just fine, the only problem here is that duplicity is not handling an IOError exception properly

i would love to run this natively under windows but there is so much python code that uses linux specific stdlib stuff that it became too much work to just get a backup solution working

but i will say that if a file is not locked then duplicity works perfectly, i would not consider this a WSL issue at all.

[Andrew Smith (andrew-smith)]

At the risk of adding noise I'll observe that I think duplicity contains a number of bugs around string/byte handling when running under Python 3 instead of Python 2, e.g. I raised https://github.com/henrysher/duplicity/issues/23 upstream just now because the WebDAV backend is broken (using Basic auth) since Ubuntu 19.10 when Python default switched to 3.

[SchnippenSchnappen (snibbedisnaps)]

I run the current Snap version from Snapcraft: https://snapcraft.io/deja-dup on Ubuntu 19.10 and I get this bug. I opted for the Snap version because of this bug (https://bugs.launchpad.net/duplicity/+bug/1852876) as well.

[SchnippenSchnappen (snibbedisnaps)]

Sorry for the double post. There seems no way to edit posts though. So here is the full error message I get:

Traceback (innermost last):
  File "/snap/deja-dup/252/bin/duplicity", line 101, in <module>
    with_tempdir(main)
  File "/snap/deja-dup/252/bin/duplicity", line 87, in with_tempdir
    fn()
  File "/snap/deja-dup/252/lib/python3.6/site-packages/duplicity/dup_main.py", line 1539, in main
    do_backup(action)
  File "/snap/deja-dup/252/lib/python3.6/site-packages/duplicity/dup_main.py", line 1663, in do_backup
    full_backup(col_stats)
  File "/snap/deja-dup/252/lib/python3.6/site-packages/duplicity/dup_main.py", line 569, in full_backup
    globals.backend)
  File "/snap/deja-dup/252/lib/python3.6/site-packages/duplicity/dup_main.py", line 426, in write_multivol
    globals.volsize)
  File "/snap/deja-dup/252/lib/python3.6/site-packages/duplicity/gpg.py", line 393, in GPGWriteFile
    data = block_iter.__next__().data
  File "/snap/deja-dup/252/lib/python3.6/site-packages/duplicity/diffdir.py", line 541, in __next__
    result = self.process_continued() # pylint: disable=assignment-from-no-return
  File "/snap/deja-dup/252/lib/python3.6/site-packages/duplicity/diffdir.py", line 725, in process_continued
    data, last_block = self.get_data_block(self.process_fp)
  File "/snap/deja-dup/252/lib/python3.6/site-packages/duplicity/diffdir.py", line 709, in get_data_block
    buf = fp.read(read_size)
  File "/snap/deja-dup/252/lib/python3.6/site-packages/duplicity/diffdir.py", line 452, in read
    self.sig_gen.update(buf)
  File "/snap/deja-dup/252/lib/python3.6/site-packages/duplicity/librsync.py", line 224, in update
    self.buffer += buf
 TypeError: can't concat str to bytes

[Kenneth Loafman (kenneth-loafman)]

That contains, I think, the stable version of duplicity, 0.8.08. You want the edge version:

$ sudo snap refresh --classic --edge duplicity

[SchnippenSchnappen (snibbedisnaps)]

Thanks for your quick reply.

You are correct with your assumption that it deja-dup 40.6 contains duplicity 0.8.08. However I will have to wait for Michael Terry to update the deja-dup snap (at least the edge version of it) with duplicity 0.8.09 so I can fully replicate my issue and see if it is fixed. (Also I have a giant backup that takes almost 7 hours to complete which I want to manage with deja-dup. I'm honestly too scared to touch this with a "naked" duplicity.)

Maybe Mark Grandi can do a quick test of the edge version since he used duplicity without deja-dup before?

[Tim Starling (tstarling)]

This looks like a duplicate of #1855636. I came here because I'm seeing an identical backtrace to the one reported here, but on closer investigation it appears to be the same issue which was fixed by https://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/revision/1523