Microsoft Sans Serif maps to Thai font even on european setups

[Sebastian Bertho]

Binary package hint: fontconfig

Mono/WinForms uses "Microsoft Sans Serif" as it's default font in every form.

In Ubuntu default installation (at least with a german setup (DE_de)) this font maps to "Loma" which is a Thai font. "Loma" lacks our "umlauts" (Umlaute) - so every german Mono/WinForms Application will not display "umlauts" (Umlaute) like Ä,Ö,Ü,ß,ä,ö,ü in Ubuntu's default config.

Removing thai-ttf-tlg package will resolve the problem, because "Microsoft Sans Serif" is resolved to DejaVU Sans after this.

If it is possible to setup different font mappings on different locales, "Microsoft Sans Serif" should resolve to DejaVu Sans even if the package ttf-thai-tlg is installed.

[Novell-paul (novell-paul)]

Created an attachment (id=335483)
Screenshot with Unicode glyphs replaced by square boxes

[Novell-paul (novell-paul)]

Created an attachment (id=335487)
Test case using g.DrawString() directly

Per spouliot's request. This shows that CJK fails on the default font, and that non-ASCII fails with eg. Font("MS Sans Serif").

[Novell-paul (novell-paul)]

Created an attachment (id=335488)
Default Font("") on top three lines; Font("MS Sans Serif") on bottom three lines

[Spouliot-novell (spouliot-novell)]

Ok, if I got this right then I should be able to duplicate this with 2.4 (and 2.6 and HEAD) and get something different when I rebuild mono/mcs/libgdiplus 2.0, right ?

I'm updating my 2-0 branch right now and will try this tomorrow morning. Then I'll compare the results with 2.4.2.3 (default on my openSuSE 11.2 laptop), the Mono/mcs 2.6 (that I keep for moonlight development) and HEAD.

I'll attach the pictures when done.
Thanks for the test case!

[Spouliot-novell (spouliot-novell)]

Created an attachment (id=335582)
opensuse 11.2 (x86_64) using mono/libgdiplus 2.0.1

I made two change to your sample. First the background is white (instead of transparent, since it's easier with eog) and I also added a version string in the filename (to avoid mistakes between executions).

[mono] ~/src/bugzilla @ gmcs 569011.cs -pkg:dotnet
[mono] ~/src/bugzilla @ mono --version
Mono JIT compiler version 2.0.1 (/branches/mono-2-0/mono r149209)
Copyright (C) 2002-2008 Novell, Inc and Contributors. www.mono-project.com
 TLS: __thread
 GC: Included Boehm (with typed GC)
 SIGSEGV: altstack
 Notifications: epoll
 Architecture: amd64
 Disabled: none
[mono] ~/src/bugzilla @ mono 569011.exe 2.0.1
[mono] ~/src/bugzilla @ ego 569011-mono-2.0.1.png

From strace I made sure I was loading the 2.0.1 libgdiplus (not my distro one)
open("/opt/mono/lib/libgdiplus.so", O_RDONLY) = 3

[Spouliot-novell (spouliot-novell)]

Created an attachment (id=335583)
opensuse 11.2 (x86_64) using mono/libgdiplus 2.4.2.3 (distro)

Next I tried the mono version installed by my distro.

poupou@mizar:~/src/bugzilla> mono --version
Mono JIT compiler version 2.4.2.3 (tarball Sat Oct 24 05:46:54 UTC 2009)
Copyright (C) 2002-2008 Novell, Inc and Contributors. www.mono-project.com
 TLS: __thread
 GC: Included Boehm (with typed GC)
 SIGSEGV: altstack
 Notifications: epoll
 Architecture: amd64
 Disabled: none
poupou@mizar:~/src/bugzilla> mono 569011.exe 2.4.2.3

Using strace I made sure that I used my distro libgdiplus (2.4.2.3)
open("/usr/lib64/libgdiplus.so", O_RDONLY) = 3

[Spouliot-novell (spouliot-novell)]

As far as I can tell the two bitmaps are identical. This match the fact that there was very few changes in libgdiplus between both releases. In any case the behavior you described (font substitution) is not implemented in libgdiplus - any such occurrence would come from outside.

As you get different results I suspect they are due to system changes (e.g. different versions of fontconfig, cairo....) or configuration changes (e.g. installed fonts).

Good luck in finding the culprit!

[Novell-paul (novell-paul)]

Okay, thanks for that. So it's a change in the dependency chain that is also getting changed, but still something that (eg. use of a particular flag) that is only showing up via Mono.

At least it's documented and Googlable now.

[Sebastian Bertho (sbertho)]

Binary package hint: fontconfig

Mono/WinForms uses "Microsoft Sans Serif" as it's default font in every form.

In Ubuntu default installation (at least with a german setup (DE_de)) this font maps to "Loma" which is a Thai font. "Loma" lacks our "umlauts" (Umlaute) - so every german Mono/WinForms Application will not display "umlauts" (Umlaute) like Ä,Ö,Ü,ß,ä,ö,ü in Ubuntu's default config.

Removing thai-ttf-tlg package will resolve the problem, because "Microsoft Sans Serif" is resolved to DejaVU Sans after this.

If it is possible to setup different font mappings on different locales, "Microsoft Sans Serif" should resolve to DejaVu Sans even if the package ttf-thai-tlg is installed.

[Torsten Krah (tkrah)]

This does help to solve the problem described (add it to a new global system wide fonts conf file or to a custom user only place as described by fontconfig docs) - but having the thai package still there.

<match target="pattern">
                <test qual="any" name="family" compare="eq">
                        <string>MS Sans Serif</string>
                        <string>Microsoft Sans Serif</string>
                </test>
                <edit name="family" mode="append" binding="same">
                        <string>DejaVu Sans</string>
                </edit>
</match>

[Jean-Baptiste Lallement (jibel)]

Thanks for your report.

This is confirmed in maverick. This is caused by the mapping defined in /etc/fonts/conf.avail/89-ttf-thai-tlwg-synthetic.conf
Affecting to thaifonts-scalable

[Theppitak Karoonboonyanan (thep)]

Does this twist for 89-ttf-thai-tlwg-synthetic.conf help for your case?

        <match target="pattern">
                <test name="lang" compare="contains">
                        <string>th</string>
                </test>
                <test qual="any" name="family" compare="eq">
                        <string>MS Sans Serif</string>
                        <string>Microsoft Sans Serif</string>
                </test>
                <edit name="family" mode="append" binding="same">
                        <string>Loma</string>
                </edit>
        </match>

[Sebastian Bertho (sbertho)]

Yes, this twist fixes this issue for me.

[Theppitak Karoonboonyanan (thep)]

OK. Thanks. So, I'll consider applying this to upstream source.

I've tested it with HTML with 'lang' tag. It works, but I'm not sure if it's the case in general. German web sites without 'lang="de"' tag may still suffer from this issue, despite the twist.

[Sebastian Bertho (sbertho)]

I tested a web seite on a german setup and it seems that all common browsers (I checked firefox and chromium) consider the value of the LANG variable. So on all of my setups, "DejaVu Sans" is choosen for "MS Sans Serif" on web sites - with or without a 'lang="de"' tag.

But the real strange thing is:
Even if I choose "Loma" on a web site or in e.g. "Openoffice.org writer" umlauts are displayed correctly.
But if I choose "Loma" in Winforms/Mono applications, umlauts are replaced with boxes.

So there seems to be a problem with playing Winforms/Mono and Loma together.

But: This twist is a valid workaround on all of my systems.

I attach a sample program "UmlautsTest.cs" and a simple web-page "test.html" where you can confirm what I see here.

[Sebastian Bertho (sbertho)]

Compile the above with:
gmcs UmlautsTest.cs -r:System.Windows.Forms.dll

[Theppitak Karoonboonyanan (thep)]

Sorry that I can't verify this myself. My system is still mono-free. So, I'll just take your reports.

Loma doesn't provide accented Latin characters. So, it's up to applications how they handle the missing glyphs. Web browsers and OO.o take the glyphs from other fonts, while, according to your report, Winforms/Mono apps don't. That may explains.

OK, as the workaround is valid, I'll apply it upstream. Hope to release it before Maverick is frozen.

[Paul Sladen (sladen)]

This is fairly important. *Anyone* viewing to view basic ASCII characters (so all Latin-based languages other than straight English) and all other world languages (except Thai) display as square boxes for each of those characters.

[Novell-paul (novell-paul)]

This has eventually been traced back to a specific font package (providing Thai fonts) having a slightly too-broad match in:

  /etc/fonts/conf.avail/89-ttf-thai-tlwg-synthetic.conf

and thus catching /all/ explicit requests for "MS Sans Serif" and "Microsoft Sans Serif" ...which is what WinForms specifically requests.

Thus the explicit "grabbing" of the pseudo generic font family name causes everything to be sourced from that font, which has no coverage other than for basic (non-accented Latin) and for Thai.

[Theppitak Karoonboonyanan (thep)]

So, please sync thaifonts-scalable 0.4.14-1 from sid/squeeze which has been marked as addressing this bug. (Sorry, I noticed LP had already recognized this, so, I thought the sync would be done next. But that didn't happen.)

[Paul Sladen (sladen)]

Yup, Sebastian Bertho has also mailed to say that the version in Debian/sid fixes this.

[Paul Sladen (sladen)]

I've put in for a FFe and sync at bug #637131. I guess what's confusing is because this bug report (on Launchpad) was also acting as the upstream and Debian bugs, so Launchpad would normally watch the other bug reports and then update the status when they were fixed. You're too efficient!

The test Winforms application I've been using is 'openbve' (select options and there is a drop down of languages). The new Debian/upstream release fixes the display of Latin, but for some reason there is still another issue with CJK substitution not occuring, so it's possibly another font package making a similar land-grab (anyway, not Thaifonts).

[Launchpad Janitor (janitor)]

This bug was fixed in the package thaifonts-scalable - 1:0.4.14-1

---------------
thaifonts-scalable (1:0.4.14-1) unstable; urgency=low

  * New upstream bugfix release.
    - Fix Tahoma/MS Sans Serif fallback rules to not affect non-Thai
      environments. (LP: #539008, re-addresses LP #434054)
    - debian/patches/01_remove_tahoma_fallback.diff: Removed, addressed
      differently upstream.
  * Bump Standards-Version to 3.9.1 (no changes needed)
 -- Paul Sladen <email address hidden> Tue, 14 Sep 2010 10:22:19 +0000

[Martin Pitt (pitti)]

Accepted thaifonts-scalable into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[cablop (cablop)]

No fix package for Maverick?

[cablop (cablop)]

No thai fonts scalable here, but still font fails to fallback to a correct font, it is still not showing Japanese or Chinese characters in mono applications, just empty boxes/squares.

[Paul Sladen (sladen)]

cablop: this /should/ have been (at least partially) fixed in Ubuntu 10.10 ("natty") which is the upload in comment #13. Bug as noted in comment #12 there's still a (different, but related) issue with CJK.

[Neil Mayhew (neil.mayhew)]

@cablop: A fixed version was already in maverick. See bug #637131 for details.

[Launchpad Janitor (janitor)]

This bug was fixed in the package thaifonts-scalable - 1:0.4.13-4ubuntu0.1

---------------
thaifonts-scalable (1:0.4.13-4ubuntu0.1) lucid-proposed; urgency=low

  * Bugfix release (LP: #637131)
    - Fix Tahoma/MS Sans Serif fallback rules to not affect non-Thai
      environments. (LP: #539008, re-addresses LP #434054)
    - debian/patches/01_remove_tahoma_fallback.diff: Removed, addressed
      differently.
 -- Neil Mayhew <email address hidden> Mon, 08 Nov 2010 12:48:50 +0100

[cablop (cablop)]

There's no thai fonts scalable in my system. It is resolving the umlauts and accented characters now, but what it is ignoring is the CJK characters.

I tried the following

I opened a terminal, then

export LANG=ja

and then it shows this error

(KeePass.exe:30080): Gtk-WARNING **: Locale not supported by C library.
 Using the fallback 'C' locale.

(KeePass could be downloaded at their website http://keepass.info and get the portable one to do those tests).

I think it is related to the mono libraries.

[cablop (cablop)]

According to this https://bugzilla.novell.com/show_bug.cgi?id=569011, the issue is related with libgdiplus. It was not happening with the 2.0 version, but the 2.4.x throws the error.