Mismatch between declared and actual encoding in mail notifications and web pages

[Johannes Rohr]

This is about 2.1.22 in Ubuntu Yakkety. mailman is configured with German (de) as default language and the templates for German and English are generated.

For some reason, the notifications and web pages in German are all encoded in latin-1, while the declared encoding both in mail notification as well as in web pages served is utf-8. This result in broken display of umlauts until one manually changes the encoding.

See https://lists.uferwerk.org/cgi-bin/mailman/listinfo/

I have tried to manually enforce UTF-8 by adding

 add_language('en', 'English (USA)', 'utf-8', 'ltr')
 add_language('de', 'Deutsch', 'utf-8', 'ltr')

We see this misbehaviour since we migrated mailman from one server running Ubuntu 14.4 with mailman 1:2.1.14-3ubuntu0.4 to another server running 16.10 with mailman 1:2.1.22-1ubuntu0.1

On the old server, it worked just fine without any additional tweaking and on the new server, not tweaking seems to change anything at all.

In the mailman error log I see recurring error messages of the sort:

  Apr 24 15:10:19 2017 (898) list: mitglieder: can't decode "Liste f�r Mitglieder der Uferwerk e.G." as utf-8
 Apr 24 15:10:19 2017 (901) list: mitglieder: can't decode "Liste f�r Mitglieder der Uferwerk e.G." as utf-8
 Apr 24 15:10:20 2017 (901) list: mitglieder: can't decode "Liste f�r Mitglieder der Uferwerk e.G." as utf-8

I am not sure why these descriptions were suddenly in latin-1 but I manually converted them to utf-8 now, restarted mailman, and the situation remains unchanged: Both the overview page and the mail notifications are in latin-1 while the headers declare that they are in utf-8, which results in broken rendering of the text

[Mark Sapiro (msapiro)]

This is not a GNU mailman bug. It is a Debian/Ubuntu issue. See <https://bugs.launchpad.net/mailman/+bug/1462755> including comment #6 and the list posts linked from there.

Note that in current Debian/Ubuntu packages there is Debian 91_utf8.patch (available, e.g. in <http://http.debian.net/debian/pool/main/m/mailman/mailman_2.1.18-2+deb8u1.debian.tar.xz>)

Among other things, this patch redefines the add_language() function to ignore the third argument and force the encoding to UTF-8.

Your issue is one or both of two things:

You may have custom 'de' templates that are latin-1 encoded (Debian's patch recodes all the standard templates to utf-8).

You have strings in list attributes that are latin-1 encoded (Definitely in 'description' attributes. This is the part that Debian completely overlooked. The recode_list script mentioned at <https://bugs.launchpad.net/mailman/+bug/1462755/comments/6> and the mailman-users list thread also mentioned in that comment may help.

There also seems to be another issue in your case. The boilerplate at the top of <https://lists.uferwerk.org/cgi-bin/mailman/listinfo/> comes from translated strings in messages/de/LC_MESSAGES/mailman.mo. Debian's patch recodes all these to utf-8 in its distribution, but for some reason your compiled German message catalog contains latin-1 strings.

[Johannes Rohr (jorohr)]

Thanks Mark! It seems the mailman.po is in latin1

[Johannes Rohr (jorohr)]

Indeed, used recode to make it into utf-8 and regenerated mailman.mo, and now its fine! Thanks for your help!

[Joshua Powers (powersj)]

Hi! The original reporter appears to have the issue resolved and I do not see any current action required. Because the issue was related to local configuration of the message catalog in latin-1, I am marking this invalid.

If someone disagrees or feels that some additional work or documentation is required, please feel free to make a suggestion and add it to this bug report.