Member names do not support ǧ

[Hendrik Maryns]

In the administrator area, the members page, I wanted to enter the name of a subscriber. Her name is Çelikoǧlu. Note that the Turkish ǧ is not supported by most encodings. It gave an error in that nothing happened: her name wasn’t changed.

?? Doesn’t Mailman work with utf-8 ??

http://mailman.biohostnet.de/mailman/admin/mitarbeiter/members

[Mark Sapiro (msapiro)]

If the list's preferred language is Turkish, the character set is iso-8859-9. The character set used depends on the language and is defined in the LC_DESCRIPTIONS dictionary which is initialized at the end of Defaults.py.

You can change the character set for Turkish to utf-8, but you also have to convert messages/tr/LC_MESSAGES/mailman.po and templates/tr/* to utf-8 and rebuild messages/tr/LC-Messages/mailman.mo.

Note however that in spite of any character set issues, you should have been able to enter a name with the ǧ and have it at least display as ǧ (it actually should display as a character, but xss protection goes overboard in escaping the '&').

[Hendrik Maryns (hamaryns)]

The list’s prefered language is German, since this is in Germany. However, in this globalized world, you just cannot assume all names are German, especially not names.

Why at all use iso-8859-9, or any other iso-xxxx for that matter? There is UTF-8 now, no need for it.

Note that I am not a sysadmin on the machine running Mailman, I am only a user administrator who is annoyed that I cannot enter the name of one of my collaborators. So thanks for your suggestions, but I won’t be bothering my web hoster with this stuff, unless you indicate that this indeed is due to a wrong setup on his part.

As I said: I was able to enter the name (of course, in Firefox), but submitting the changes did nothing, not even an error message. I do not understand your last sentence.

[Mark Sapiro (msapiro)]

The character set in Mailman for German is iso-8859-1. I agree with you that everything should be unicode (real names are currently stored in Mailman as unicodes) and character sets should be those such as utf-8 that can faithfully represent unicodes, and I think Mailman is moving in this direction, and the newer translations are utf-8 encoded, but there are still many languages that use iso-8859-x or other (euc-jp, euc-kr, koi8-r) encodings. It is primarily the translators who maintain these translations who determine the character set.

>As I said: I was able to enter the name (of course, in Firefox), but submitting the changes did nothing, not even an error message. I do not understand your last sentence.

There does seem to be a bug in that I can enter (on a German Language list) Çeliko-lu and submit changes and what I entered displays as Çeliko-lu. I can enter -elikoǧlu and submit changes and what I entered displays as -elikoǧlu, but if I enter Çelikoǧlu and submit changes I get a "Bug in Mailman version 2.1.12". I can work on fixing that, but that doesn't explain why your Mailman doesn't report the error unless your hosting service has somehow modified Mailman to supress this display.

And what I meant by the last sentence is that the display that I see (-eliko&#487;lu) should really display the browser's rendering of the HTML entity rather that the raw entity (&#487;), but there is code in Mailman to prevent this being used as an XSS attack (say by putting <script ... as part of the real name) and that code changes &#487; into &amp;#487; so you see &#487; instead of the browser's rendering of character 487.

[Hendrik Maryns (hamaryns)]

> There does seem to be a bug in that I can enter (on a German Language list) Çeliko-lu and submit changes and what I entered displays as Çeliko-lu. I can enter -elikoǧlu and submit changes and what I entered displays as -eliko&#487;lu, but if I enter Çelikoǧlu and submit changes I get a "Bug in Mailman version 2.1.12". I can work on fixing that, but that doesn't explain why your Mailman doesn't report the error unless your hosting service has somehow modified Mailman to supress this display.

This seems to indicate that it chooses an encoding depending on the characters inside. But since Ç and ǧ do not fit in the same iso-8859-xx encoding, it barfs. Strange, that it doesn’t consider UTF-8, then.

Strange that it doesn’t report anything, yes. Where are these reports to be found? Is there a log file?\

I found out it is version 2.1.9, btw.

[Mark Sapiro (msapiro)]

> This seems to indicate that it chooses an encoding depending on the characters inside. But since Ç and ǧ do not fit in the same iso-8859-xx encoding, it barfs. Strange, that it doesn’t consider UTF-8, then.

Not really. It tries to keep the name internally in unicode. The problem occurs in converting the name from the web browser to unicode when the input from the web browser contains an &#nnn; entity with nnn > 256 as well as other non-ascii characters. The code parses the input into pieces separated by HTML entities which it parses into characters, and then it uses the Python string join() method to put it all back together. join() has to coerce the string to unicode because of the >256 entity which has become a unicode. This coersion occurs using Python's default encoding which is normally 'ascii' and can only be changed in Python's site.py module.

I am currently testing the attached patch which I think will fix the bug.

>Strange that it doesn’t report anything, yes. Where are these reports to be found? Is there a log file?

It should report an error. The fact that it doesn't implies your hosting service has modified this mailman in some way. The error should also be logged in Mailman's 'error' log which could be anywhere, but in a default install is /usr/local/mailman/logs/error.

[Pander (pander)]

In mailman 1:2.1.14-1ubuntu2 I get ? for ž and ë when I do list_members -f.

[Pander (pander)]

Ah, this is because sys.getdefaultencoding() returns 'ascii' That should be 'utf-8' although env | grep LANG returns LANG=en_US.UTF-8

[Mark Sapiro (msapiro)]

This is a Python thing. You can create, for example, /usr/lib/pythonv.v/site-packages/sitecustomize.py containing

import sys
sys.setdefaultencoding('utf8')

and this will set the default unicode encoding to utf8 as long as Python is run without the -S option. The sitecustomize.py module can be anywhere in the sys.path buily by site.py.