snapd generates incomplete fontconfig caches, result in emoji rendering issue in Chromium (and non-snap browsers too)

Bug #1858636 reported by Marius Gedminas
90
This bug affects 23 people
Affects Status Importance Assigned to Milestone
chromium-browser (Ubuntu)
Invalid
Medium
Unassigned
snapd (Ubuntu)
Fix Released
High
Unassigned

Bug Description

I’ve recently noticed that Chromium no longer can display emoji. All I get are square boxes and some black & white symbols. E.g. https://getemoji.com renders like the screenshot I posted to https://discourse.ubuntu.com/t/call-for-testing-chromium-browser-deb-to-snap-transition/11179/237.

I'm like 95% sure that this is a recent problem, and that I didn't have it when I first upgraded to Ubuntu 19.10 and to the snapified Chromium.

Note that I don't have fonts-symbola installed, and I do have fonts-not-color-emoji installed. I've also noticed a bunch of

saus. 07 14:08:40 blynas chromium_chromium.desktop[4802]: Fontconfig error: Cannot load default config file
saus. 07 14:08:40 blynas chromium_chromium.desktop[4802]: Fontconfig error: Cannot load default config file
saus. 07 14:08:41 blynas chromium_chromium.desktop[4802]: Fontconfig error: Cannot load default config file
saus. 07 14:08:43 blynas chromium_chromium.desktop[4802]: Fontconfig error: Cannot load default config file

messages in journalctl when I first launch chromium.
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu8.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DRM.card0-DP-1:
 enabled: disabled
 dpms: Off
 status: disconnected
 edid-base64:
 modes:
DRM.card0-DP-2:
 enabled: disabled
 dpms: Off
 status: disconnected
 edid-base64:
 modes:
DRM.card0-DP-3:
 enabled: enabled
 dpms: On
 status: connected
 edid-base64: AP///////wAebfl2IdYEAAkbAQOAUCJ46sqVplVOoSYPUFQla4BxQIGAgcCpwLMA0cCBANHPzUYAoKA4H0AwIDoAHk4xAAAaKVkAoKA4J0AwIDoAHk4xAAAaAAAA/QA4Sx5aGAAKICAgICAgAAAA/ABMRyBVTFRSQVdJREUKAZACAxrxIwkHB0cQBAMBHxMSgwEAAGUDDAAQAIwK0Iog4C0QED6WAB5OMQAAGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP8ANzA5TlRXRzlBOTYxCgAAAAAAAAAAAAAAQg==
 modes: 2560x1080 2560x1080 1920x1080 1920x1080 1920x1080 1920x1080 1920x1080 1680x1050 1600x900 1280x1024 1280x1024 1280x800 1152x864 1152x864 1280x720 1280x720 1280x720 1280x720 1024x768 1024x768 832x624 800x600 800x600 720x576 720x480 720x480 720x480 720x480 640x480 640x480 640x480 640x480
DRM.card0-DP-4:
 enabled: disabled
 dpms: On
 status: disconnected
 edid-base64:
 modes:
DRM.card0-DP-5:
 enabled: disabled
 dpms: On
 status: disconnected
 edid-base64:
 modes:
DRM.card0-HDMI-A-1:
 enabled: disabled
 dpms: Off
 status: disconnected
 edid-base64:
 modes:
DRM.card0-HDMI-A-2:
 enabled: disabled
 dpms: Off
 status: disconnected
 edid-base64:
 modes:
DRM.card0-eDP-1:
 enabled: enabled
 dpms: On
 status: connected
 edid-base64: AP///////wAGry1aAAAAAAAcAQSlHRF4Au6Vo1RMmSYPUFQAAAABAQEBAQEBAQEBAQEBAQEBtDeAoHA4PkA6KjUAJaUQAAAYAAAADwAAAAAAAAAAAAAAAAAgAAAA/gBBVU8KICAgICAgICAgAAAA/gBCMTMzSEFOMDUuQSAKAHA=
 modes: 1920x1080
DiskUsage:
 Filesystem Type Size Used Avail Use% Mounted on
 /dev/nvme0n1p5 ext4 284G 241G 29G 90% /
 tmpfs tmpfs 7,7G 164M 7,6G 3% /dev/shm
 /dev/nvme0n1p5 ext4 284G 241G 29G 90% /
DistroRelease: Ubuntu 19.10
EcryptfsInUse: Yes
InstallationDate: Installed on 2019-06-12 (208 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
MachineType: LENOVO 20Q0CTO1WW
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
Package: chromium-browser 77.0.3865.120-0ubuntu1.19.10.1
PackageArchitecture: amd64
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.3.0-24-generic root=UUID=ad2946f7-6a38-471f-b7d1-779e8e9fd109 ro quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 5.3.0-24.26-generic 5.3.10
Snap.ChromeDriverVersion: ChromeDriver 79.0.3945.79 (29f75ce3f42b007bd80361b0dfcfee3a13ff90b8-refs/branch-heads/3945@{#916})
Snap.ChromiumVersion: Chromium 79.0.3945.79 snap
Tags: eoan wayland-session snap
Uname: Linux 5.3.0-24-generic x86_64
UpgradeStatus: Upgraded to eoan on 2019-10-18 (80 days ago)
UserGroups: adm cdrom dip docker libvirt lpadmin lxd plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 07/18/2019
dmi.bios.vendor: LENOVO
dmi.bios.version: N2JET73W (1.51 )
dmi.board.asset.tag: Not Available
dmi.board.name: 20Q0CTO1WW
dmi.board.vendor: LENOVO
dmi.board.version: SDK0J40697 WIN
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: None
dmi.modalias: dmi:bvnLENOVO:bvrN2JET73W(1.51):bd07/18/2019:svnLENOVO:pn20Q0CTO1WW:pvrThinkPadX390:rvnLENOVO:rn20Q0CTO1WW:rvrSDK0J40697WIN:cvnLENOVO:ct10:cvrNone:
dmi.product.family: ThinkPad X390
dmi.product.name: 20Q0CTO1WW
dmi.product.sku: LENOVO_MT_20Q0_BU_Think_FM_ThinkPad X390
dmi.product.version: ThinkPad X390
dmi.sys.vendor: LENOVO

Revision history for this message
Marius Gedminas (mgedmin) wrote : CurrentDmesg.txt

apport information

tags: added: apport-collected eoan snap wayland-session
description: updated
Revision history for this message
Marius Gedminas (mgedmin) wrote : Dependencies.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : Lspci.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : Lsusb.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : ProcEnviron.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : ProcModules.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : Snap.ChromiumPrefs.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : Snap.Connections.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : Snap.Info.chromium.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : Snap.Info.core.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : Snap.Info.core18.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : Snap.Info.gtk-common-themes.txt

apport information

Revision history for this message
Marius Gedminas (mgedmin) wrote : UdevDb.txt

apport information

Revision history for this message
Launchpad Janitor (janitor) wrote : Re: [snap] color emoji not rendered; many bw emoji rendered as boxes

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in chromium-browser (Ubuntu):
status: New → Confirmed
Changed in chromium-browser (Ubuntu):
importance: Undecided → High
Revision history for this message
Olivier Tilloy (osomon) wrote :

(cross-posting my comment on the forum thread)

I’m seeing the same in a clean eoan VM (the chromium snap version doesn’t matter, all channels exhibit the problem). Interestingly my work laptop with the chromium snap from the stable channel doesn’t exhibit the problem, emojis are rendered correctly.
In both cases I’m seeing several instances of this error message:

[ERROR:child_process_sandbox_support_impl_linux.cc(79)] FontService unique font name matching request did not receive a response.

but it’s probably unrelated, given that emojis are rendered correctly and coloured on my laptop.

description: updated
Revision history for this message
Janick Martinez Esturo (ph03) wrote :

I was affected as well, solved by removing and re-installing the `fonts-noto-color-emoji` pacakge

Revision history for this message
Marius Gedminas (mgedmin) wrote :

The workaround worked for me too. I used the simplified version

    sudo apt install --reinstall fonts-noto-color-emoji

Revision history for this message
Marius Gedminas (mgedmin) wrote :

Unfortunately the workaround is temporary. I rebooted and the emoji are gone again.

Revision history for this message
Marius Gedminas (mgedmin) wrote :

I tired to figure out what exactly happens when I reinstall fonts-noto-color-emoji. According to dpkg's output, fontconfig's triggers run. fontconfig's postinst runs

  # Force regeneration of all fontconfig cache files.
  mkdir -p /var/cache/fontconfig
  fc-cache -s -v 1>/var/log/fontconfig.log 2>&1 || printf "fc-cache failed.\nSee /var/log/fontconfig.log for more information.\n"

when triggered. I looked at /var/log/fontconfig.log, and I see a lot of "skipped" messages for various directories (because "existing cache is valid" or "looped directory detected"), and then I see

  /var/cache/fontconfig: cleaning cache directory
  /var/cache/fontconfig: invalid cache file: 0bd3dc0958fa2205aaaa8ebb13e2872b-le64.cache-7
  fc-cache: succeeded

Now I'm wondering how a fontconfig cache file becomes invalid, and how do other applications cope.

Revision history for this message
Marius Gedminas (mgedmin) wrote :

Yesterday I saw a notification about chromium being auto-refreshed, restarted it, and today I noticed that color emoji are gone again.

I suspect the two facts are related.

Revision history for this message
Marius Gedminas (mgedmin) wrote :

Another experiment: while emoji are broken, I can do sudo fc-cache -s -v, and it doesn't find any invalid cache files, and when I restart chromium, I don't get color emoji.

But then I do sudo apt install --reinstall fonts-noto-color-emoji, and I see a /var/log/fontconfig.log with an up-to-date timestamp, and it says, again,

    /var/cache/fontconfig: invalid cache file: 0bd3dc0958fa2205aaaa8ebb13e2872b-le64.cache-7

and I restart chromium again, and now I have working color emoji. (Until the next snap refresh at least).

I don't understand anything.

Revision history for this message
Olivier Tilloy (osomon) wrote :

Thanks for your findings Marius. I can reliably reproduce the problem in a clean and up-to-date focal VM with the following steps:

 - snap install chromium
 - snap run chromium # color emojis are rendered correctly
 - snap refresh chromium --beta
 - snap run chromium # color emojis are gone
 - sudo apt reinstall fonts-noto-color-emoji
 - snap run chromium # color emojis are back

Olivier Tilloy (osomon)
Changed in chromium-browser (Ubuntu):
importance: High → Medium
tags: added: focal
Revision history for this message
Marius Gedminas (mgedmin) wrote :

FWIW rebooting also makes the problem reappear. snap info shows that I last refreshed chromium 13 days ago, but I rebooted today to install a kernel update, and color emoji disappeared.

Revision history for this message
Marius Gedminas (mgedmin) wrote :

This time I successfully resurrected color emoji by manually removing the file that fc-cache used to complain about. Or, rather, I moved it aside, for study:

sudo mv /var/cache/fontconfig/0bd3dc0958fa2205aaaa8ebb13e2872b-le64.cache-7{,.bork}

Then I restarted Chromium and emoji are now colorful again.

Revision history for this message
Marius Gedminas (mgedmin) wrote :

FWIW 0bd3dc0958fa2205aaaa8ebb13e2872b is MD5 of "/usr/share/fonts/truetype/noto". fontconfig keeps one cache file per font directory.

The timestamp of the broken fontconfig cache file is May 15, 10:37. I wonder what my computer was doing at the time.

'last reboot' indicates that I rebooted on May 12, and then today, May 20. I'm pretty sure I launched Chromium right after booting and never quit it until the next reboot.

journalctl shows me what my computer was doing on May 15 at 10:36:

geg. 15 10:36:59 blynas lxd.daemon[365715]: => Stop reason is: snap refresh
geg. 15 10:36:59 blynas lxd.daemon[365715]: => Stopping LXD
geg. 15 10:36:59 blynas lxd.daemon[1225]: => LXD exited cleanly

so it is snap refresh, not reboot, that creates the broken fontconfig cache file! It's just that when I snap refresh something that is not chromium, I don't restart chromium!

Revision history for this message
Marius Gedminas (mgedmin) wrote :

Curiously, fc-cat /var/cache/fontconfig/0bd3dc0958fa2205aaaa8ebb13e2872b-le64.cache-7.bork does not complain about the cache file being invalid. It shows information about a bunch of Noto fonts BUT! not the ColorEmoji one!

diff -u <(fc-cat /var/cache/fontconfig/*.bork|cut -d'"' -f2|sort) <(ls /usr/share/fonts/truetype/noto/|sort)

shows that NotoColorEmoji.ttf is the only font missing from the font cache file.

Now it's time for baseless speculation: could it be that snap refresh somehow runs fc-cache inside a container where /var/cache/fontconfig is bind-mounted from the host OS, but /usr/share/fonts/truetype/ comes from some base snap image?

Revision history for this message
Marius Gedminas (mgedmin) wrote :

Maybe not.

ls -ld /snap/*/current/usr/share/fonts/truetype/noto shows that only one snap has that directory (gnome-3-34-1804), and instead of containing all the fonts but NotoColorEmoji.ttf it's the exact opposite: it contains one font only, and that is NotoColorEmoji.ttf.

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

We could probably remove "[snap]" from the bug title. I experience the same bug in Chrome, which is a deb.

Revision history for this message
Olivier Tilloy (osomon) wrote :

That's interesting information Daniel. Have you observed a pattern in how/when the problem occurs? Does reinstalling fonts-noto-color-emoji "fix" the problem (like in comment #25)?

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

It's completely random. Works most days but fails once or twice a week. Even though I open a fresh Chrome every day.

And yes reinstalling fonts-noto-color-emoji fixes the problem every time.

summary: - [snap] color emoji not rendered; many bw emoji rendered as boxes
+ color emoji not rendered; many bw emoji rendered as boxes
Revision history for this message
Marius Gedminas (mgedmin) wrote : Re: color emoji not rendered; many bw emoji rendered as boxes

I ended up adding

if [ -n "$PS1" ]; then
    # only for interactive sessions please
    if [ -f /var/cache/fontconfig/0bd3dc0958fa2205aaaa8ebb13e2872b-le64.cache-7 ]; then
       echo "color emoji will be broken in Chromium again (LP: #1858636) unless you"
       echo "sudo rm /var/cache/fontconfig/0bd3dc0958fa2205aaaa8ebb13e2872b-le64.cache-7"
    fi
fi

to my ~/.bashrc to warn me when this problem is coming back. So far the appearance of this broken font cache file correlates 100% with some snap (any snap! refreshing _lxd_ which has nothing to do with GUI apps or fonts causes this!) getting refreshed, according to the timestamp reported by snap changes.

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Yes I was thinking this sounds like a font cache problem. Perhaps unique to Ubuntu? I couldn't find anything like it reported against Chromium upstream.

Revision history for this message
Olivier Tilloy (osomon) wrote :

Marius, that last comment is a very interesting data point. I'm adding a snapd task to the bug.

Changed in snapd (Ubuntu):
importance: Undecided → High
summary: - color emoji not rendered; many bw emoji rendered as boxes
+ snapd generated incomplete fontconfig caches, result in emoji rendering
+ issue in chromium
summary: - snapd generated incomplete fontconfig caches, result in emoji rendering
+ snapd generates incomplete fontconfig caches, result in emoji rendering
issue in chromium
tags: added: rls-ff-incoming
Revision history for this message
Sebastien Bacher (seb128) wrote : Re: snapd generates incomplete fontconfig caches, result in emoji rendering issue in chromium

Confirmed by doing those steps on a focal system

($ sudo apt install fonts-noto-color-emoji
if you don't have it installed)

$ fc-cat /var/cache/fontconfig/* | grep NotoColorE
there is a result listed

"NotoColorEmoji.ttf" 0 "Noto Color Emoji:familylang=en:style=Regular:stylelang=en:fullname=Noto Color ...

$ sudo rm -rf /var/cache/fontconfig/
$ sudo snap refresh --edge core
(there is probably a smarter way to trigger a fontconfig cache refresh from snapd but that did it)

$ fc-cat /var/cache/fontconfig/* | grep NotoColorE
-> not NotoColorEmoji.ttf result anymore

Revision history for this message
Ian Johnson (anonymouse67) wrote :

I can confirm I see the bug with seb128's reproducer. First I will propose a spread test which demonstrates the issue so we don't see regressions on this front, but also it's very curious to me that this specific font:

fonts-noto-color-emoji

does not show up in the cache we generate but this font:

fonts-kiloji

do show up in the cache we generate. The spread test we have for this functionality was using the latter so we didn't see this bug til now.

Changed in snapd (Ubuntu):
status: New → Confirmed
assignee: nobody → Ian Johnson (anonymouse67)
Revision history for this message
Ian Johnson (anonymouse67) wrote :

Also just to clarify, we will regenerate fontconfig cache's on any snap refresh/install during the link-snap task.

Changed in snapd (Ubuntu):
assignee: Ian Johnson (anonymouse67) → nobody
Revision history for this message
Ian Johnson (anonymouse67) wrote :

Unfortunately we do not have a clear solution to this yet, so I have unassigned myself from the snapd task and we need more help before we can continue on this. Although, we have a few options as I detail here: https://github.com/snapcore/snapd/pull/8856#issuecomment-644784642

I'm not sure what the best solution is, but definitely this will need some guidance from the desktop team or others more knowledgable about fonts about how to proceed here. Some questions I have include:

* Can someone explain the difference between fonts-noto-color-emoji and fonts-kiloji to try and understand why one shows up and the other doesn't with our current fontcache mechanism in snapd?
* Can someone explain exactly what things the `dpkg-reconfigure fontcache` command calls to build the cache? is it sufficient to just call fc-cache from the fontconfig package or do we need to be doing something else too?
* Would it be possible for the fontcache to be run from inside confinement? I.e. what set of directories from the host would the fontcache binary from the fontconfig package need to see in order to build a fontcache?
* If we had to ship fontconfig in the base snaps, is it possible for us to just ship the fontconfig binary like we do today in the snapd snap, or do we really need the full package in order to have full compatibility with all fonts in Ubuntu?

Revision history for this message
Sebastien Bacher (seb128) wrote :

Hey Ian, thanks for the updating the report. I don't have the answers to all your question and I'm unsure we have someone in desktop who is expert with fontconfig or fonts to reply to e.g the first one

I can reply to at least one though!

> * Can someone explain exactly what things the `dpkg-reconfigure fontcache` command calls to build the cache? is it sufficient to just call fc-cache from the fontconfig package or do we need to be doing something else too?

$ cat /var/lib/dpkg/info/fontconfig.postinst
...
  # Force regeneration of all fontconfig cache files.
  mkdir -p /var/cache/fontconfig
  printf "Regenerating fonts cache... "
  fc-cache -s -f -v 1>/var/log/fontconfig.log 2>&1 || (printf "failed.\nSee /var/log/fontconfig.log for more information.\n"; exit 1)
  printf "done.\n"

that's what the fontconfig package is doing

Revision history for this message
Sebastien Bacher (seb128) wrote :

(closing the chromium part since it's not an issue with that component)

Changed in chromium-browser (Ubuntu):
status: Confirmed → Invalid
Revision history for this message
James Henstridge (jamesh) wrote :

I've just been reading through the code used to build the bin/fc-cache-v6 and bin/fc-cache-v7 binaries in the core snap:

https://github.com/snapcore/fc-cache-static-builder

I think I understand what the problem is: while it is rebuilding copies of xenial's and bionic's fontconfig tools that statically link to libfontconfig, it's using the xenial version of libfreetype2 for both.

I suspect that this old freetype version does not understand all the features of the font (e.g. the scalable colour emoji), so the generated cache also misses them.

Revision history for this message
Ken VanDine (ken-vandine) wrote :
Revision history for this message
Sebastien Bacher (seb128) wrote :

@mvo built a snap from the PR, wiping the cache and installing it gave me a working cache listing the emoji font so it seems to resolve the issue

summary: snapd generates incomplete fontconfig caches, result in emoji rendering
- issue in chromium
+ issue in Chromium (and non-snap browsers too)
Revision history for this message
Ian Johnson (anonymouse67) wrote :

The mentioned fix has been released for some time in stable snapd now, are folks still seeing invalid caches created by snapd when any snap refreshes?

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Feels fixed to me. If the bug was still present I would have seen it every week, but I haven't for months.

Revision history for this message
Marius Gedminas (mgedmin) wrote :

Feels fixed to me too!

Changed in snapd (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers