Failure due to _logger.log failure for content with special characters: TypeError decoding Unicode not supported

[Ralf Herold]

-- duplicity 0.6.04 (but error was also present with earlier version) from duplicity-team PPA
-- Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) [GCC 4.3.3] on linux2
-- Ubuntu 9.04
-- command line (please note "@" and ":" in part before the webdavs-server name): duplicity --log-file ~/tmp/dpl.log /home/abc/def/ webdavs://<email address hidden>:<email address hidden>/xyz
-- from dpl.log:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1241, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1234, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1135, in main
    sync_archive()
  File "/usr/bin/duplicity", line 914, in sync_archive
    remlist = globals.backend.list()
  File "/usr/lib/python2.6/dist-packages/duplicity/backends/webdavbackend.py", line 185, in list
    filename = self.__taste_href(href)
  File "/usr/lib/python2.6/dist-packages/duplicity/backends/webdavbackend.py", line 201, in __taste_href
    "%s -> %s" % (raw_filename, filename))
  File "/usr/lib/python2.6/dist-packages/duplicity/log.py", line 73, in Debug
    Log(s, DEBUG)
  File "/usr/lib/python2.6/dist-packages/duplicity/log.py", line 65, in Log
    _logger.log(DupToLoggerLevel(verb_level), unicode(s, "utf8", errors='ignore'))
TypeError: decoding Unicode is not supported

Please note that this error from _logger.log is triggered at several places; for example, the command line above would raise the error from line 201 in webdavbackend.py but also by line 810 in collections.py etc.

Many thanks for your attention - kind regards

[Larry Gilbert (l2g)]

It looks like Ken changed this:

    _logger.log(DupToLoggerLevel(verb_level), s)

to this:

    _logger.log(DupToLoggerLevel(verb_level), unicode(s, "utf8", errors='ignore'))

the purpose being to "ignore unicode() tignore unicode() translation errors in log messsages." But TypeError gets thrown by unicode() when it's given something that's already of type unicode.

I'm a little nervous about fixing this one myself because I don't understand the circumstances that would have caused the original bug that required the change above.

[Larry Gilbert (l2g)]

I meant to say that the log entry for that change was "ignore unicode() translation errors in log messsages." Sorry for the typo. (pasto?)

[Steve Pomeroy (xxv)]

I just encountered this problem as well. Debian just pushed out 0.6.04 and this seems to break duplicity entirely or at least the following command:

    duplicity collection-status s3+http://foo/bar

as well as incremental.

Is this affecting everyone or is there some strange set of state that one needs to trigger it?

[Kenneth Loafman (kenneth-loafman)]

I'm guessing here since I can't reproduce it, but I think its got to be locale and Unicode filenames. The original bug was an unadorned string that was throwing an exception because of Unicode being in the string, most likely from a Unicode filename since the bug was before we started translations and all the strings were ASCII.

Anyone know how to print a string where part of it is in ASCII and part in that abomination called Unicode? We have to be able to support strings like "Unable to read %s" where the string is ASCII and the %s replacement is Unicode. It appears that Python will happily format the two together, then complain about the result later.

[Kenneth Loafman (kenneth-loafman)]

I've attached a patch that may fix it. If you are getting this error now, please try this out and let me know if it works. If it does not, please let me know more that may help isolate this problem: language, locale, filesystem, any other options that you can isolate. I really am guessing on this problem, so any info may be relevant.

[Steve Pomeroy (xxv)]

Ken,

That patch worked for me, thanks!

For reference, I keep my locale in:

LANG=en_US.UTF-8
[everything else in en_US.UTF-8]
LC_TIME=en_DK.UTF-8
LC_ALL=

and this is an ext3 filesystem, probably with some files that have utf-8 in their names.

[Ralf Herold (ralf-herold)]

Thanks also from here, no problems with the command line above any more. Thanks!