Auto-detect of Japanese Character Encoding does not work

[Jun Kobayashi]

Binary package hint: firefox-3.0

Auto detectiong for Japanese character encodings does not work on Firefox 3 Beta even if this function works well on Firefox 2.
This is hassle for users who browse Japanese web pages.

A patch to fix this problem is already in upstream bug tracker.

[Masa141421356 (masa141421356)]

At this case, both of HTTP response header and META in html contains only
"Content-Type: text/html", "charset" is not contained.

[Alice0775 (alice0775)]

Regression window is :
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5pre) Gecko/2008032512 :
work fine
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5pre) Gecko/2008032513 : NG

[M-wada (m-wada)]

Confirmed with Fx trunk 2008033105(Win-XP SP2).
No problem when Auto Detect/Universal, with all EUC-JP,UTF-8,Shift_JIS test pages.
> View/Character Encoding/Auto Detect/Universal
> (intl.charset.detector=universal_charset_detector)

[Masa141421356 (masa141421356)]

According to Bonsai, Bug 424916 may be related.
http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=PhoenixTinderbox&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2008-03-25+12%3A00%3A00&maxdate=2008-03-25+14%3A00%3A00&cvsroot=%2Fcvsroot

[hidenosuke (hidenosuke)]

I can reproduce with recent trunk Linux.
In the case of 2 URLs in comment 0 auto-detect doesn't work at all.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9pre) Gecko/2008040120 Firefox 3.0pre

[Vladimir Vukicevic (vvuk)]

Marking blocking, based on regression... need an automated test for this.

[hidenosuke (hidenosuke)]

I tried to back the patch out of bug 424916
and it seeme to have solved this problem.
I think this is a regression of bug 424916.

[Smontagu (smontagu)]

Created attachment 313966
Patch

So based on comment 3 I think the way to fix this is to get rid of the old CJK parallel state machine detectors in intl/chardet and just use the universal detector with a language filter. The universal detector has been much better maintained, and it will remove a lot of duplicated data.

The patch is a bit large and scary, but most of it is just moving XPCOM stuff around. Since time is short, I'm requesting code review already while I work on testcases.

The patch doesn't include the cvs removes:
intl/chardet/src/Big5Statistics.h
intl/chardet/src/EUCJPStatistics.h
intl/chardet/src/EUCKRStatistics.h
intl/chardet/src/EUCTWStatistics.h
intl/chardet/src/GB2312Statistics.h
intl/chardet/src/nsBIG5Verifier.h
intl/chardet/src/nsCP1252Verifier.h
intl/chardet/src/nsEUCJPVerifier.h
intl/chardet/src/nsEUCKRVerifier.h
intl/chardet/src/nsEUCTWVerifier.h
intl/chardet/src/nsGB18030Verifier.h
intl/chardet/src/nsGB2312Verifier.h
intl/chardet/src/nsHZVerifier.h
intl/chardet/src/nsISO2022CNVerifier.h
intl/chardet/src/nsISO2022JPVerifier.h
intl/chardet/src/nsISO2022KRVerifier.h
intl/chardet/src/nsPSMDetectors.cpp
intl/chardet/src/nsPSMDetectors.h
intl/chardet/src/nsPkgInt.h
intl/chardet/src/nsSJISVerifier.h
intl/chardet/src/nsUCS2BEVerifier.h
intl/chardet/src/nsUCS2LEVerifier.h
intl/chardet/src/nsUTF8Verifier.h
intl/chardet/src/nsVerifier.h

[Smontagu (smontagu)]

Created attachment 313968
Corrected some silly typos

[Smontagu (smontagu)]

Created attachment 314152
Mochitests

[L. David Baron (dbaron)]

I knew there was another review I really meant to get to yesterday... sorry about missing this one.

I'm having trouble understanding the filtering stuff. It seems when FILTER_JAPANESE, etc., is used, the universal detector will consider all Japanese *plus* all non-CJK encodings, when we really only want the Japanese ones (plus UTF-8), or something like that. When FILTER_NONE is used, it seems like we'll consider all encodings except for CJK, when we really want to consider all of them.

[L. David Baron (dbaron)]

Er, never mind about the first problem -- I didn't read the nsUniversalDetector.cpp changes closely enough. But I still think the second problem exists, and it might help to fix it by replacing FILTER_NONE by FILTER_ALL (with at least one additional constant for FILTER_NON_CJK) -- and then you could always do Latin1 and UTF8, regardless of filter (which looks like what you intended).

[Smontagu (smontagu)]

Created attachment 314495
Address David's comments

[Smontagu (smontagu)]

Created attachment 314496
Address David's comments

The right patch this time

[L. David Baron (dbaron)]

Comment on attachment 314496
Address David's comments

r+sr=dbaron

[Smontagu (smontagu)]

Checked in.

[Masa141421356 (masa141421356)]

Works fine. Thank you.
-> Verified.

[Jun Kobayashi (jkbys)]

Binary package hint: firefox-3.0

Auto detectiong for Japanese character encodings does not work on Firefox 3 Beta even if this function works well on Firefox 2.
This is hassle for users who browse Japanese web pages.

A patch to fix this problem is already in upstream bug tracker.

[Rolf Leggewie (r0lf)]

FWIW, the URLs given in the upstream bug report

http://space.geocities.jp/alice0775/STORE/UTF-8.html
http://space.geocities.jp/alice0775/STORE/EUC-JP.html

work fine here

[Rolf Leggewie (r0lf)]

Kobayashi-san, I am having some other trouble with fonts, though. Many times in Thunderbird and Firefox the display is not as good as in gutsy (mixing different fonts, for example). If you can see above pages (or force them to the correct encoding), do you get different fonts to display the page as in the attached screenshot? If this is not a configuration error on my part, I'd like to open a new bug report about that.

[Alexander Sack (asac)]

should be fixed everywhere now