[hardy] different font for Kanji and Kana

[Rolf Leggewie]

Binary package hint: thunderbird

Luckily, I found a Japanese mail I don't mind sharing with you in my spam folder to demonstrate the following bug. I have a three-pane layout like many other people, folders on the left, content of the selected folder on the top right and content of the selected mail on the bottom right portion.

Since I upgraded to hardy, Thunderbird is doing funny things with Japanese text. I have seen it at least the in "content of the selected folder" part, but there might be other areas. TB is mixing different fonts. In the screenshot, Kana (green underline) are displayed in a different font than Kanji (red underline). The Kana font is softer, not sure exactly which font is used.

Here is the header from that mail (some obfuscation against further spam)

Return-Path: <[e-mail]>
X-Original-To: $address1
Delivered-To: $address2
X-Greylist: delayed 3477 seconds by postgrey-1.31 at Rie; Sat, 26 Apr 2008 00:27:53 CEST
Received: from SPK70.localdomain (unknown [219.255.67.25])
 by $mail.mydomain.NIX (Postfix) with ESMTP id 4338313293
 for <$address1>; Sat, 26 Apr 2008 00:27:52 +0200 (CEST)
Received: by SPK70.localdomain (Postfix, from userid 48)
 id ADC98707E9; Sat, 26 Apr 2008 06:01:06 +0900 (KST)
To: $address1
Subject: =?ISO-2022-JP?B?GyRCOkc/N04iGyhCRFZEGyRCRn4yWSROJCpDTiRpJDshKhsoQg==?=
Date: Sat, 26 Apr 2008 06:01:06 +0900
From: [e-mail]
Message-ID: <email address hidden>
X-Priority: 3
X-Mailer: Microsoft Outlook Express 6.00.2900.3138
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="iso-2022-jp"

[Rolf Leggewie (r0lf)]

Just saw that FF3 is affected by the same problem

[Rolf Leggewie (r0lf)]

what's interesting is that the message window is displayed in only one font type, it seems, the round one.

[Rolf Leggewie (r0lf)]

bug 219931 confirms this problem (the two reports are likely to be about the same issue)

[Arne Goetje (arnegoetje)]

what is your system locale?

It's most likely not the same issue like #219931, as the text existed beforehand and was not typed by you. Therefor it is not logical to me why first a Chinese font was picked and then the Kanas are rendered in a different font. This should not be necessary as both, Kanji and Kana are present in both Chinese and Japanese fonts.

To reproduce this, I'd need to know your system locale and the list of installed font packages on your system ( dpkg -l | grep ttf- ).

[rogmorri (frontporsche)]

Actually, bug 219931 happens with pre-entered text as well. I first noticed it on a web page I happened to view.

[Saša Bodiroža (jazzva)]

Changing task from firefox to firefox-3.0, as it's reported in FF3.

[Alexander Sack (asac)]

arne, could you please do some initial triaging so we understand exactly what this issue is about?

Many thanks,

 - Alexander

[Arne Goetje (arnegoetje)]

I need more information to be able to triage this bug report:

1. your system locale

2. which font packages you have installed? ('dpkg -l | grep ttf-')

3. the output of 'fontconfig-voodoo -c'

[Rolf Leggewie (r0lf)]

$ set|egrep '(LC|LANG)'
GDM_LANG=de_DE.utf8
LANG=de_DE.utf8
LC_MESSAGES=en_US.utf8
MAILCHECK=60

$ dpkg -l | grep ttf-*
ii libsdl-ttf2.0-0 2.0.9-1 ttf library for Simple DirectMedia Layer with FreeType 2 support
ii ttf-arphic-ukai 0.2.20080216.1-0ubuntu1 "AR PL UKai" Chinese Unicode TrueType font collection Kaiti styl
ii ttf-arphic-uming 0.2.20080216.1-0ubuntu1 "AR PL UMing" Chinese Unicode TrueType font collection Mingti st
ii ttf-bitstream-vera 1.10-7 The Bitstream Vera family of free TrueType fonts
ii ttf-dejavu 2.23-1 Metapackage to pull in ttf-dejavu-core and ttf-dejavu-extra
ii ttf-dejavu-core 2.23-1 Vera font family derivate with additional characters
ii ttf-dejavu-extra 2.23-1 Vera font family derivate with additional characters
ii ttf-freefont 20060501cvs-12 Freefont Serif, Sans and Mono Truetype fonts
ii ttf-kochi-gothic-naga10 1.0.20030809-4 Kochi Subst Gothic Japanese TrueType font with naga10 (non-free)
ii ttf-kochi-mincho-naga10 1.0.20030809-4 Kochi Subst Mincho Japanese TrueType font with naga10 (non-free)
ii ttf-mgopen 1.1-3 Magenta MgOpen TrueType fonts
ii ttf-opensymbol 1:2.4.0-3ubuntu6 The OpenSymbol TrueType font
ii ttf-vlgothic 20071215-1 Japanese TrueType font from Vine Linux

$ fontconfig-voodoo -c
Unconfigured

[Rolf Leggewie (r0lf)]

"dpkg -l | grep ttf-*" will not cover all the fonts I have installed, though. I know at least about msttcorefonts which installs msgothic and msmincho.

[Arne Goetje (arnegoetje)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Rolf Leggewie wrote:
> "dpkg -l | grep ttf-*" will not cover all the fonts I have installed,
> though. I know at least about msttcorefonts which installs msgothic and
> msmincho.
>

ok, we don't support non-packaged fonts or msttcorefonts, therefor we
only provide configurations for the fonts we ship.
Since you use a non-Japanese locale, the system cannot determine which
font you prefer to display Japanese text. Since many Kanji are also used
in Chinese texts and the Chinese fonts we provide cover more glyphs than
the Japanese ones, fontconfig/pango will prefer the Chinese font over
the Japanese ones. An exception is made here for Kana, because those are
labeled as being exclusively Japanese within fontconfig/pango. Therefor
a Japanese font is used here.

If you want to tell the system that you prefer Japanese fonts, you can
use 'fontconfig-voodoo -f -s ja_JP'. But this will also force the Latin
glyphs to be rendered by Japanese fonts, since this is expected behavior
among Japanese users.

Other than that, the only way to 'fix' this would be for you to create a
custom ~/.fonts.conf file and specify the fonts you would like to use
and in which order there. You can take a look at
/etc/fonts/conf.avail/69-language-selector-$lang-$country.conf for examples.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIKULObp/QbmhdHowRAgZ1AKDFdLGi3yKCzHsdlWENCqd+rl7AVQCgn1IT
ZCRItcnAHorbeITUh1uf+bg=
=wv5E
-----END PGP SIGNATURE-----

[Rolf Leggewie (r0lf)]

Well, so I guess, I want to create a ~/.fonts.conf file. I have to say, I am already not too happy about this. IMHO, the goal should be for this to be supported OOTB, but I guess that is complicated. I am also not sure the Japanese users want Latin glyphs rendered by the Japanese fonts since they are usually quite ugly in those fonts.

Anyway, so I copied /etc/fonts/conf.avail/69-language-selector-ja-jp.conf to ~/.fonts.conf, but had to remove it right away since that broke display of Umlauts. I guess what I need is to specify that Latin glyphs be displayed the way they are being displayed right now, next in line is display with a Japanese font and only if that fails try a Chinese font (matching the Japanese type as closely as possible if possible to specify). Molding that into a ~/.fonts.conf file is unfortunately beyond my skill. Help would be appreciated.

[Arne Goetje (arnegoetje)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Rolf Leggewie wrote:
> Well, so I guess, I want to create a ~/.fonts.conf file. I have to say,
> I am already not too happy about this. IMHO, the goal should be for
> this to be supported OOTB, but I guess that is complicated. I am also

No, it's impossible! At least with the tools we currently have
available. Every user has different preferences when it comes to font
rendering. Providing one configuration that works for all is really
impossible.

> not sure the Japanese users want Latin glyphs rendered by the Japanese
> fonts since they are usually quite ugly in those fonts.

According to the Japanese LoCo team, they do prefer it this way.

> Anyway, so I copied /etc/fonts/conf.avail/69-language-selector-ja-
> jp.conf to ~/.fonts.conf, but had to remove it right away since that
> broke display of Umlauts. I guess what I need is to specify that Latin
> glyphs be displayed the way they are being displayed right now, next in
> line is display with a Japanese font and only if that fails try a
> Chinese font (matching the Japanese type as closely as possible if
> possible to specify). Molding that into a ~/.fonts.conf file is
> unfortunately beyond my skill. Help would be appreciated.

Please see the attached .fonts.conf I made for you. I just inserted
Bitstream Vera and DejaVu fonts to the top of the lists, means they get
the highest priority, followed by the Japanese fonts.

Is this what you want?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIKcANbp/QbmhdHowRAnRTAJ40Dv2Hskk2Z8abTY0we/TiIHjYsACcDRSs
067bMkzVwzhu/SWGLkQ8aIA=
=X+Vj
-----END PGP SIGNATURE-----

[Alexander Sack (asac)]

is this invalid or what? is that really properly filed against firefox? wherelese could we put this bug and forward upstream?

[Arne Goetje (arnegoetje)]

invalid, since this is a configuration issue and not a bug.