Broken encoding in Russian-translated HTML templates

[WGH]

Description: Ubuntu 18.04 LTS
Release: 18.04

mailman:
  Installed: 1:2.1.26-1
  Candidate: 1:2.1.26-1
  Version table:
 *** 1:2.1.26-1 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status

The Ubuntu package contains Russian-translated HTML templates in really curious text encoding: UTF-8 interpreted as KOI8-R and converted into UTF-8. Needless to say, the text looks like this: пёп╨п╟п╤п╦я┌п╣ п©п╟я─п╬п╩я▄:

The upstream mailman has been using UTF-8 for Russian HTML templates since 2015 (https://bugs.launchpad.net/mailman/+bug/1418448). My guess is that Ubuntu package scripts (?) has been converting KOI8-R to UTF-8 before upstream switched to UTF-8, and continued to do so after switch, resulting in broken files.

[WGH (wgh)]

This bug also occurs in Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901810

[Andreas Hasenack (ahasenack)]

Thanks for filing this bug in Ubuntu, and also in Debian. The mailman package in Ubuntu is a straight sync from the Debian one, so it's best to be fixed there first.

This might have something to do with debian/patches/91_utf8.patch. I see it doing some charset manipulation in the "install:" target, and this line looks suspiscious:
+ CHARSET_ru=koi8-r; \

[Andreas Hasenack (ahasenack)]

It does look correct when displayed in a web browser, or not? I confirm the actual html file looks like you said, but the browser gives me a page like the attached screenshot (never mind it says "xenial" there, it's a bionic container, I just made a mistake with the hostname).

[WGH (wgh)]

This is indeed correct.

In my case it didn't render correctly though. I think that was because the template also included parts in proper encoding (e.g. list description or localized messages from gettext), and the browser can't really recover from mixed incorrect encoding.

[WGH (wgh)]

I'll need to spin up my own test container to properly test why it ever renders correctly. I don't think browsers are actually able to figure out that the text should be decoded from UTF-8 to KOI8-R and then interpreted as UTF-8. My Firefox wasn't able to do that when I opened template source file directly.

Maybe default Debian mailman configuration somehow fixes the encoding back into something recoverable, I don't know.

Though, frankly, I don't really want to investigate why obviously broken configuration works in certain cases, when the fix is (more or less) apparent :)

[Andreas Hasenack (ahasenack)]

Debian's commit: https://salsa.debian.org/mailman-team/mailman2/commit/1b04eda09f630f06254fa314aac9331240e4539e