--help command line option is broken

[Iakov Davydov]

In 0.3.1
$ exaile --help
Traceback (most recent call last):
  File "/usr/lib/exaile/exaile.py", line 52, in <module>
    main()
  File "/usr/lib/exaile/exaile.py", line 49, in main
    exaile = main.Exaile()
  File "/usr/lib/exaile/xl/main.py", line 56, in __init__
    (self.options, self.args) = self.get_options().parse_args()
  File "/usr/lib/python2.6/optparse.py", line 1394, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/usr/lib/python2.6/optparse.py", line 1434, in _process_args
    self._process_long_opt(rargs, values)
  File "/usr/lib/python2.6/optparse.py", line 1509, in _process_long_opt
    option.process(opt, value, values, self)
  File "/usr/lib/python2.6/optparse.py", line 788, in process
    self.action, self.dest, opt, value, values, parser)
  File "/usr/lib/python2.6/optparse.py", line 810, in take_action
    parser.print_help()
  File "/usr/lib/python2.6/optparse.py", line 1664, in print_help
    file.write(self.format_help().encode(encoding, "replace"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 102: ordinal not in range(128)

[reacocard (reacocard)]

Works for me

[Iakov Davydov (iakov-davydov)]

I am using Ubuntu Karmic, packages from official ppa.

$ locale
LANG=ru_RU.UTF-8
LC_CTYPE="ru_RU.UTF-8"
LC_NUMERIC="ru_RU.UTF-8"
LC_TIME="ru_RU.UTF-8"
LC_COLLATE="ru_RU.UTF-8"
LC_MONETARY="ru_RU.UTF-8"
LC_MESSAGES="ru_RU.UTF-8"
LC_PAPER="ru_RU.UTF-8"
LC_NAME="ru_RU.UTF-8"
LC_ADDRESS="ru_RU.UTF-8"
LC_TELEPHONE="ru_RU.UTF-8"
LC_MEASUREMENT="ru_RU.UTF-8"
LC_IDENTIFICATION="ru_RU.UTF-8"
LC_ALL=

[Iakov Davydov (iakov-davydov)]

$LANG="" exaile --help works fine.

[reacocard (reacocard)]

Hm, even if i change locale to ru_RU I still can't reproduce this. Can anyone else confirm this problem?

[Mathias Brodala (mathbr)]

Confirmed with German locale. This one is known to me for quite a while now but I wasn’t yet able to figure out the root of the issue.

[Mathias Brodala (mathbr)]

Most likely this one: http://bugs.python.org/issue4319

[Mathias Brodala (mathbr)]

Subclassing optparse.OptionParser and overriding the print_help() method to not do any encoding at all works for me.

[Iakov Davydov (iakov-davydov)]

Mathias, if you attach a patch I can test it.

[Mathias Brodala (mathbr)]

Here’s the patch. Mostly only a workaround until Python itself is fixed.

[Iakov Davydov (iakov-davydov)]

Hello,
Sorry I cannot test it right now (my exaile instalation is on the different pc).
But here is simple test for unicode and optparse:

davidov@theatre:~/tmp/optparse$ python test1.py --help
Traceback (most recent call last):
  File "test1.py", line 7, in <module>
    (options, args) = parser.parse_args()
  File "/usr/lib/python2.6/optparse.py", line 1394, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/usr/lib/python2.6/optparse.py", line 1434, in _process_args
    self._process_long_opt(rargs, values)
  File "/usr/lib/python2.6/optparse.py", line 1509, in _process_long_opt
    option.process(opt, value, values, self)
  File "/usr/lib/python2.6/optparse.py", line 788, in process
    self.action, self.dest, opt, value, values, parser)
  File "/usr/lib/python2.6/optparse.py", line 810, in take_action
    parser.print_help()
  File "/usr/lib/python2.6/optparse.py", line 1664, in print_help
    file.write(self.format_help().encode(encoding, "replace"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd1 in position 116: ordinal not in range(128)
davidov@theatre:~/tmp/optparse$ python test2.py --help
Usage: test2.py [options]

Options:
  -h, --help show this help message and exit
  -f FILE, --file=FILE файл

davidov@theatre:~/tmp/optparse$ diff -u test1.py test2.py
--- test1.py 2010-03-02 12:44:32.639580195 +0300
+++ test2.py 2010-03-02 12:45:06.829555252 +0300
@@ -2,6 +2,6 @@
 from optparse import OptionParser
 parser = OptionParser()
 parser.add_option("-f", "--file", dest="filename",
- help="файл", metavar="FILE")
+ help=u"файл", metavar="FILE")

 (options, args) = parser.parse_args()

So you have just to use unicode strings for uncode help messages; that seems not to be optparse bug.

[Mathias Brodala (mathbr)]

Please check first. It IS a bug within optparse since we’re not passing Unicode data directly to optparse, thus cannot control what kind of string is passed. We simply let gettext replace messages with translations before outputting which is why optparse fails unable to handle non-ASCII strings here.

[Iakov Davydov (iakov-davydov)]

Dear Mathias, AFAIR this is how encodings work in python. The good practice is to use unicode inside the application always and only encode everything during IO.

Of course gettext cat use unicode by default; it's just not enabled in exaile.

Here is the patch that seems to fix the issue.

I think it need more work (it seems to affect something else).

[Iakov Davydov (iakov-davydov)]

Here is upgraded patch (fixes get_title() not in unicode).

[Johannes Sasongko (sjohannes)]

> Here is upgraded patch (fixes get_title() not in unicode).
>
> ** Patch added: "gettext unicode fix v2"
>   http://launchpadlibrarian.net/39954036/exaile_gettext_unicode_v2.patch

The change in xlgui/playlist looks good to me.

Need Aren's review on the xl/nls change, since there might be a reason
it wasn't using gettext.install in the first place. (We're using
``from xl.nls import gettext as _``; I've never bothered asking why.
If we use gettext.install, those can go away.)

[reacocard (reacocard)]

gettext.install never worked reliably; we kept getting errors where _ would be inexplicably undefined. Hence the decision to switch to static imports of it in each module. Its not pretty, but it is reliable. Additionally, going through our own module allows us to override gettext with a custom function for systems built without NLS, which was requested by a maintainer of our BSD packages.

[reacocard (reacocard)]

Hm, if I understand the problem correctly, gettext is just returning a utf-8 string instead of a unicode obj? Is it not then sufficient to coerce it to unicode before returning? The attached patch attempts to do that and doesn't appear to break anything for me, can anyone confirm it fixes the issue?

[Iakov Davydov (iakov-davydov)]

gettext.patch seems to fix the issue. But there are a lot of lines like "if title == col.get_title():" in playlist.py.
I am not sure why pygtk returns raw strings insead of unicode strings (maybe this should be considered as a bug while python 3 is not here).

But anyway here is your patch + fix in playlist.py

[reacocard (reacocard)]

Since my patch and that fix are technically two separate issues, I've split the patch into two commits, r2888 and r2889.