[FFe] Demilight (OS/2 weight=350) confuses fontconfig

Bug #1556457 reported by Mingye Wang on 2016-03-12
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fontconfig
Fix Released
Medium
fontconfig (Debian)
New
Unknown
fontconfig (Ubuntu)
High
Unassigned

Bug Description

[FFe request comment]

Upgrading to fontconfig 2.11.94 is proposed as a fix of this bug, including a fix of certain font weight issues due to the switch from fonts-droid to fonts-noto-cjk for Chinese (discussed at bug #1468027) where we haven't found any way to work around the problem.

The proposed upload is available in this PPA:
https://launchpad.net/~gunnarhj/+archive/ubuntu/fontconfig-test2

Changes are listed in the attachment upstream_git-log.txt

[Original description including updates]

See https://bugs.freedesktop.org/show_bug.cgi?id=81453 and bug 1468027.

Fontconfig lacks support for many OpenType/TrueType OS/2 font weight values. This causes a bunch of problems, like mixing up Demilight (weight=350) and Regular (weight=400).

Although it's possible to write (dirty?) hacks for deb-packed fonts, this still causes problems for otherwise sourced fonts.

Archlinux: https://bugs.archlinux.org/task/48550 (fix released, upgraded to 2.11.94)

Recently Adobe & Google released a open-source pan-CJK font, Source Han Sans from Adobe or Noto Sans CJK from Google.

This font family features 7 font weights: ExtraLight, Light, Normal, Regular, Meidum, Bold, Heavy and their os2 weight are: 100, 300, 350, 400, 500, 700, 900 respectively.

However, in fontconfig, os2 weight class 350 and 400 both maps to weight 80 and I think this makes fontconfig or pango confuse about how to choose the default font.

In particular in the GktFontChooser user can't choose one of these fonts. To fix this we probably also have to fix Pango.

This issue is also reported to the source-han-sans project: https://github.com/adobe-fonts/source-han-sans/issues/5

I believe this is a Pango limitation, not fontconfig. I'll take a look.

My bad, this *is* a fontconfig issue. Investigating.

Fontconfig part fixed. Pango fix needed.

commit ffda7c0e8130eb107ecbb3bdc48043093b12dff9
Author: Behdad Esfahbod <email address hidden>
Date: Fri Jul 25 17:59:26 2014 -0400

    Linearly interpolate weight values

    Rest of Part of https://bugs.freedesktop.org/show_bug.cgi?id=81453

    Adds new API:

        FcWeightFromOpenType()
        FcWeightToOpenType()

commit bf9df5ada77469f57101851f6b4e279a4a5ea087
Author: Behdad Esfahbod <email address hidden>
Date: Fri Jul 25 18:07:10 2014 -0400

    Change DemiLight from 65 to 55

    Such that Regular is closer to Medium than to DemiLight

commit be6506ca04ccce10868a8cd51d89e586284d149b
Author: Behdad Esfahbod <email address hidden>
Date: Fri Jul 25 16:24:26 2014 -0400

    Add FC_WEIGHT_DEMILIGHT

    Part of https://bugs.freedesktop.org/show_bug.cgi?id=81453
    Also hooks up FC_WEIGHT_BOOK to fcfreetype.c.

(In reply to comment #3)
> Fontconfig part fixed. Pango fix needed.
>
> commit ffda7c0e8130eb107ecbb3bdc48043093b12dff9
> Author: Behdad Esfahbod <email address hidden>
> Date: Fri Jul 25 17:59:26 2014 -0400
>
> Linearly interpolate weight values
>
> Rest of Part of https://bugs.freedesktop.org/show_bug.cgi?id=81453
>
> Adds new API:
>
> FcWeightFromOpenType()
> FcWeightToOpenType()

Note that this fix introduced bug 82228 as lerp doesn't handle dy == 0.

*** Bug 94505 has been marked as a duplicate of this bug. ***

Mingye Wang (artoria2e5) wrote :
description: updated
description: updated
Mingye Wang (artoria2e5) on 2016-03-12
summary: - [Upstream] Demilight (OS/2 weight=350) confuses fontconfig
+ [Resolved Upstream] Demilight (OS/2 weight=350) confuses fontconfig
Mingye Wang (artoria2e5) on 2016-03-12
description: updated
Adolfo Jayme (fitojb) on 2016-03-13
Changed in fontconfig (Ubuntu):
importance: Undecided → High

Hi again, Mingye!

I think that fontconfig .conf files are effective whether the fonts were installed via .deb packages or in some other way, as long as the font files are placed in a directory which fontconfig looks at. With that said, if it's possible to make fontconfig pick the most sensible default font weight without extra configuration, it's indeed a better way.

Changed in fontconfig:
importance: Unknown → Medium
status: Unknown → Fix Released
Mingye Wang (artoria2e5) on 2016-04-02
description: updated

Hi Mingye,

I tried to build fontconfig in a PPA with your patch. The patch applies, but unfortunately the build failed.

"error: could not locate FcRangeCreateDouble in src/*.c"

https://launchpad.net/~gunnarhj/+archive/ubuntu/fontconfig-test/+packages

Mingye Wang (artoria2e5) wrote :

I may have missed something. You know, naive cherry-picking can go wrong...

Found the commit which added FcRangeCreateDouble and cherry-picked it to the result of applying my current patch. Note that this commit (3cd573f) is way before the ones in the patch (be6506c, bf9df5a, ffda7c0, 01bb697, 80edacc) and you may want to do the cherry-pick yourself in the correct order instead of using my shuffled crap....

Mingye Wang (artoria2e5) wrote :

So here is 3cd573f.

Gunnar Hjalmarsson (gunnarhj) wrote :

Thanks for your attempt to fix it. Haven't tried the new patch yet, but before I saw it, I created a new PPA for trying the Arch Linux way: A possible upgrade to fc 2.11.94:

https://launchpad.net/~gunnarhj/+archive/ubuntu/fontconfig-test2

Much better result. :) (The below is without the influence of the language-selector .conf files.)

$ fc-match -V
fontconfig version 2.11.1
$ fc-match -a | grep 'Noto Sans CJK SC'
NotoSansCJK.ttc: "Noto Sans CJK SC" "DemiLight"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Regular"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Medium"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Thin"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Light"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Bold"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Black"
$ fc-match -V
fontconfig version 2.11.94
$ fc-match -a | grep 'Noto Sans CJK SC'
NotoSansCJK.ttc: "Noto Sans CJK SC" "Regular"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Medium"
NotoSansCJK.ttc: "Noto Sans CJK SC" "DemiLight"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Light"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Thin"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Bold"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Black"

I can't tell if it would be possible to have this accepted for 16.04. It would be very interesting to know if fc 2.11.94 also fixes the Firefox issue, which has been reported at bug #1468027 (see my latest comment there).

Mingye Wang (artoria2e5) on 2016-04-03
description: updated
description: updated
summary: - [Resolved Upstream] Demilight (OS/2 weight=350) confuses fontconfig
+ [FFe] Demilight (OS/2 weight=350) confuses fontconfig
Gunnar Hjalmarsson (gunnarhj) wrote :
description: updated
Mingye Wang (artoria2e5) wrote :

A potential problem with 2.11.94 is that it is actually a devel snapshot for 2.12 (https://www.freedesktop.org/wiki/Software/fontconfig/Devel/), so there may be some new bugs (e.g. fd.o 94427) lying around, and similar efforts may be needed to bump fc@16.04 to 2.12.0 when 2.12 is out...

Gunnar Hjalmarsson (gunnarhj) wrote :

Even if 2.11.94 is a development release, the answer you got from a fontconfig developer (<https://bugs.freedesktop.org/94505#c5>) is striking. The alternative, i.e. patching the "ancient" 2.11.1 with multiple selected commits, is also risky business.

As regards <https://bugs.freedesktop.org/94427>, I'm not able to tell how big problem it is in practice. Wouldn't that one happen with your proposed patches?

I'll talk to someone on IRC (#ubuntu-desktop) tomorrow, and get their view.

Martin Pitt (pitti) wrote :

https://launchpadlibrarian.net/251449771/upstream_git-log.txt has a lot of changes which seem desirable, and not actually much in terms of new features (aside from documentation updates). However, the sheer number/size of the changes makes this risky, particularly as testing this is extremely difficult (e. g. how reliably can we test Ubuntu/Kubuntu/Xubuntu in Bengali, Thai, various CJK, or e. g. Georgian and are confident that fonts look right?).

I don't know much about fontconfig, but is there any better/simpler way of mass-testing with our fonts than visual inspection? I. e. can we compare which glyphs are being used from which font (for our default font selection) with the current and the new version, to ensure that we don't suddenly swap fonts around, or if that happens, that this is expected? Is there some way to iterate over the unicode glyphs for fonts that we have available, to ensure that the new fontconfig does not crash on those? Any other ideas for a test plan?

Does this need any other dependencies (freetype etc.)? I. e. if 2.11.94 causes trouble, how easily can we go back to 2.11.1?

Gunnar Hjalmarsson (gunnarhj) wrote :

Thanks for your comments, Martin.

I for one haven't considered extensive testing yet. The reason for the proposal is to prevent a regression wrt Chinese rendering (due to new default font), and we have a positive report from a Chinese user at bug #1468027 (comment #105).

As regards your last question, no changes were made to debian/control, so AFAICT going back to 2.11.1 if necessary would be easy.

The reporter of this bug report, Mingye Wang, has identified a bunch of upstream commits (attached as patches to this bug report), and patching 2.11.1 with them *might* be an alternative to upgrading. We haven't tested that approach successfully yet. However, given the number of commits that would need to be cherry picked, I'm not sure it would be a safer solution.

Mingye Wang (artoria2e5) wrote :

> Is there some way to iterate over the unicode glyphs for fonts that we have available, to ensure that the new fontconfig does not crash on those?

fc-match doesn't seem to provide glyph-level lists, but anyway `diff`-ing over some fallback order dump should help:

#!/bin/bash
FC_MATCH=/usr/bin/fc-match
# use dictionary keys as a string-set
declare -A locales

# grab a list-of-locales
for i in /var/lib/locales/supported.d/*; do
  mapfile -t arr < "$i"
  # kill the longest trailing [. ]*, so what we got are basically the lang_LOC part
  for l in "${arr[@]%%[. ]*}"; do
    locales["$l"]='' # make it defined, that's all we need to snap the key in
  done
done

for loc in "${!locales[@]}"; do
  echo "# $loc:"
  export LC_ALL=${loc}.UTF-8
  for font in Sans Serif Monospace Ubuntu; do
    for weight in regular bold; do
      # more inner loops for styles.......
      echo "* $font:weight=$weight:"
      fc-match -a "$font:weight=$weight:"
      echo
    done
  done
  echo
done

> Does this need any other dependencies (freetype etc.)? if 2.11.94 causes trouble, how easily can we go back to 2.11.1?

No.

2.11.94 seems to be 100% ABI and API compatible with 2.11.1.
Archlinux and AOSC OS survived the update, and Fedora has been using 2.11.94 for quite a while.

Mingye Wang (artoria2e5) wrote :

Oops, oops, I meant to use "$FC_MATCH" -a in the script posted.

Gunnar Hjalmarsson (gunnarhj) wrote :

Considering Martin's hesitation, I completed the attempt to fix it using the other approach: Patching a bunch of cherry picked upstream commits:

https://launchpad.net/~gunnarhj/+archive/ubuntu/fontconfig-test

Seems to work. I'll ask at the other bug report too.

Martin Pitt (pitti) on 2016-04-05
Changed in fontconfig (Ubuntu):
status: New → Fix Committed
Gunnar Hjalmarsson (gunnarhj) wrote :

Building right now. Thanks for your persistence on this matter, Mingye. ;)

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package fontconfig - 2.11.1-0ubuntu9

---------------
fontconfig (2.11.1-0ubuntu9) xenial; urgency=medium

  * debian/control:
    - Add gperf to build dependencies
    - Bump FreeType build dependency version to 2.5.1

  [ Mingye Wang ]
  * debian/patches/0002-demilight.patch:
    - Handle Demilight sensibly (LP: #1556457)

 -- Gunnar Hjalmarsson <email address hidden> Mon, 04 Apr 2016 21:45:00 +0200

Changed in fontconfig (Ubuntu):
status: Fix Committed → Fix Released
Mingye Wang (artoria2e5) on 2016-04-05
no longer affects: fontconfig (Arch Linux)
Ivan Larionov (xeron-oskom) wrote :

Gunnar, could you please check https://bugs.launchpad.net/ubuntu/+source/fontconfig/+bug/1566651

I believe it's related to this fontconfig update.

Gunnar Hjalmarsson (gunnarhj) wrote :

Reopening due to regression report.

Changed in fontconfig (Ubuntu):
status: Fix Released → In Progress
Martin Pitt (pitti) wrote :

I discussed that with Gunnar and Sèbastien on IRC. Given the known regressions with 2.11.1 and the backported patch and that .94 has been used for a while in other distros and that most changes are bug fixes, we agreed to go ahead with the update. Still risky, but Gunnar will send out a call for testing, and if we find regressions we can revert back to 2.11.1 and revisit that.

Martin Pitt (pitti) on 2016-04-06
Changed in fontconfig (Ubuntu):
status: In Progress → Fix Committed
Changed in fontconfig (Debian):
status: Unknown → New
Amr Ibrahim (amribrahim1987) wrote :

Version 2.11.95 (2.12 RC5) was just released 6 hours ago.
https://cgit.freedesktop.org/fontconfig/tree/README

Amr Ibrahim (amribrahim1987) wrote :

2.11.95 (2.12 RC5)

Akira TAGOH (22):
      Add one more debugging option to see transformation on font-matching
      Fix a crash when no objects are available after filtering
      No need to be public
      mark as private at this moment
      Don't return FcFalse even when no fonts dirs is configured
      Add a warning for blank in fonts.conf
      Fix a memory leak in FcFreeTypeQueryFace
      Update CaseFolding.txt to Unicode 8.0
      Bug 90867 - Memory Leak during error case in fccharset
      Fix the broken cache more.
      Fail on make runtime as needed instead of configure if no python installed
      Use long long to see the same size between LP64 and LLP64
      Fix build issue on MinGW
      Use int64_t instead of long long
      Fix compiler warnings on MinGW
      Fix assertion on 32bit arch
      remomve unnecessary code
      Bug 93075 - Possible fix for make check failure on msys/MinGW...
      Avoid an error message on testing when no fonts.conf installed
      Add hintstyle templates and make hintslight default
      Revert "Workaround another race condition issue"
      Update libtool revision

Behdad Esfahbod (6):
      Revert changes made to FcConfigAppFontAddDir() recently
      Call FcFreeTypeQueryFace() from fcdir.c, instead of FcFreeTypeQuery()
      [GX] Support instance weight, width, and style name
      [GX] Enumerate all named-instances in TrueType GX fonts
      Improve OpenType to Fontconfig weight mapping
      [GX] Improve weight mapping

Patrick Haller (1):
      Optimizations in FcStrSet

Gunnar Hjalmarsson (gunnarhj) wrote :

On 2016-04-06 19:53, Amr Ibrahim wrote:
> Version 2.11.95 (2.12 RC5) was just released 6 hours ago.

Right. The reasons why 2.11.94 was still uploaded for now is that we are very late in the 16.04 development cycle, and 2.11.94 has been in use for a while by some distros.

Gunnar Hjalmarsson (gunnarhj) wrote :

Since fontconfig 2.11.94 is on its way into the archive, I'd like to mention this "call for testing" message I posted to the ubuntu-devel mailing list:

https://lists.ubuntu.com/archives/ubuntu-devel/2016-April/039303.html

Martin Pitt (pitti) wrote :

> Version 2.11.95 (2.12 RC5) was just released 6 hours ago.

Right, we started with .94 as we already had a tested package with that. But we indeed should track the .9x upstream releases now, changelog looks fine.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package fontconfig - 2.11.94-0ubuntu1

---------------
fontconfig (2.11.94-0ubuntu1) xenial; urgency=medium

  * New upstream release (LP: #1556457)
    - Fixes blurry fonts regression from previous upload (LP: #1566651)
  * d/p/0001-Revert-Bug-73291-poppler-does-not-show-fl-ligature.patch,
    d/p/0002-demilight.patch:
    - Dropped, applied in new release
  * Bump freetype build dep to 2.5.1 as per configure.ac.
  * Drop gperf build dep again, not necessary any more.

 -- Gunnar Hjalmarsson <email address hidden> Sun, 03 Apr 2016 02:54:00 +0200

Changed in fontconfig (Ubuntu):
status: Fix Committed → Fix Released
Amr Ibrahim (amribrahim1987) wrote :

Thanks Gunnar,

2.11.9x are release candidates on the road to 2.12. I just wanted to mention that.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.