External dictionaries are not loaded

[Jean-Louis Dupond]

On Ubuntu Eoan and most recent Thunderbird, the external Hunspell dictionaries are not available anymore on Thunderbird for spellcheck.

The symlink is still in place:
lrwxrwxrwx 1 root root 20 dec 15 2017 dictionaries -> ../../share/hunspell

But data doesn't get loaded for some reason.

NOTE TO SRU TEAM: Please do not release this package without consulting first. This is a security update pushed via -proposed to get a bit more testing. Will be released via it's own processes.

[Olivier Tilloy (osomon)]

It seems the ability to load external hunspell dictionaries was removed in https://hg.mozilla.org/mozilla-unified/rev/8810007550b1fdeeae0f5d797c4a511818a57be7.

The tracking bug upstream is https://bugzilla.mozilla.org/show_bug.cgi?id=1457321.

[Olivier Tilloy (osomon)]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) snap Chromium/77.0.3865.90 Chrome/77.0.3865.90 Safari/537.36

Steps to reproduce:

This bug was initially reported against the thunderbird 68.1.1 package in Ubuntu (https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/1847247).

Thunderbird 68 is not able to make use of external hunspell dictionaries provided by the distribution in /usr/lib/thunderbird/dictionaries.

Actual results:

This is a regression compared to thunderbird 60.9.0 and earlier, which locates and uses hunspell dictionaries installed in /usr/lib/thunderbird/dictionaries.

As far as I can tell, other distributions (such as Fedora 30) are similarly affected.

Expected results:

It looks like https://hg.mozilla.org/mozilla-unified/rev/8810007550b1fdeeae0f5d797c4a511818a57be7 is the revision that removed this ability. The tracking bug with the rationale is https://bugzilla.mozilla.org/show_bug.cgi?id=1457321, but it doesn't mention anywhere the way Linux distributions install external hunspell dictionaries and make them available to thunderbird. I wonder whether this was an oversight, or whether that use case was indeed considered but deemed unimportant when compared to the performance improvements that this change brought.

Either way, would it be possible to consider adding back the ability to load external dictionaries from the file system, maybe behind a setting that would be off by default?

[Olivier Tilloy (osomon)]

I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1587101 to track the bug upstream.

[Olivier Tilloy (osomon)]

@kmag @ehsan @aswan: can you please comment on the above request?

At the moment I'm holding back on updating the Ubuntu packages to version 68 in stable Ubuntu releases (it's only available in version 19.10 that was released yesterday) because from a user perspective this is a regression.

[Ehsan-mozilla (ehsan-mozilla)]

Hi Olivier,

Thanks for the bug report. It has been a while since that patch was landed so my memory may be a bit rusty, but I don't believe we considered the use case of Linux distributions side-loading dictionaries in this way...

Would it be a useful work-around to convert these dictionaries to the [WebExtension dictionary format](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/manifest.json/dictionaries)? I haven't tested this myself, but at least in Firefox it is [possible to side-load extensions](https://extensionworkshop.com/documentation/publish/distribute-sideloading/#standard-extension-folders), so perhaps installing such extensions to a location like `/usr/lib/thunderbird/extensions/{uuid}` may be a useful workaround?

[Kmaglione+bmo (kmaglione+bmo)]

Yes, we considered it. That directory was never intended for third-party use. If you want to install distribution dictionaries specifically for Thunderbird, you need to bundle them as extensions.

[Olivier Tilloy (osomon)]

Thanks for the feedback.

Having /usr/lib/thunderbird/dictionaries be a symlink to /usr/share/hunspell was an elegant way to reuse dictionaries installed system-wide instead of re-packaging them specifically for thunderbird.

It's a bit of a shame to loose this functionality, but I understand this wasn't designed specifically with this in mind anyway.

I'll experiment with a distro patch to restore this functionality, and if I come up with something that works and that's not too ugly I'll share it here.

Until then stable releases of Ubuntu are stuck with thunderbird 60.9.0.

[Kmaglione+bmo (kmaglione+bmo)]

If you want Thunderbird to use system Hunspell dictionaries, you can set the `DICPATH` environment variable to point to the system dictionary directory. I'm not particularly happy with that arrangement given how expensive scanning and loading dictionary directories at startup, but I recognize how convenient it is, so I've left that functionality in place.

[Olivier Tilloy (osomon)]

Setting `DICPATH` to `/usr/share/hunspell` works indeed, thanks for the tip Kris.

I understand the startup performance concern, but I think the trade off is worth it, at least for users of the Ubuntu packages who are used to having access to system dictionaries without the need to install additional extensions.

[Olivier Tilloy (osomon)]

As pointed out by Kris Maglione in the upstream bug report, setting the DICPATH environment variable to /usr/share/hunspell fixes the regression.

[Olivier Tilloy (osomon)]

Fixed with https://bazaar.launchpad.net/~mozillateam/thunderbird/thunderbird.eoan/revision/492.

[Olivier Tilloy (osomon)]

Given that there is indeed a way to load external dictionaries (the `DICPATH` environment variable), I'm closing this bug.
Thanks Kris and Ehsan for your help.

[Launchpad Janitor (janitor)]

This bug was fixed in the package thunderbird - 1:68.1.2+build1-0ubuntu2

---------------
thunderbird (1:68.1.2+build1-0ubuntu2) focal; urgency=medium

  * Ensure that external dictionaries are still loaded (LP: #1847247)
    - debian/thunderbird.sh.in
  * Update the Vcs-Bzr URL for Ubuntu 20.04 (the Focal Fossa)
    - update debian/control{,.in}

 -- Olivier Tilloy <email address hidden> Mon, 21 Oct 2019 15:22:34 +0200

[Fiona Klute (fiona-klute)]

Is the fix going to be available in Eoan? I can confirm that setting DICPATH works, but it'd be nice to have that handled automatically.

[Olivier Tilloy (osomon)]

Yes, I'm preparing an update (68.2.1) which includes this fix.

[Fiona Klute (fiona-klute)]

Great, thank you!

[Łukasz Zemczak (sil2100)]

Hello Jean-Louis, or anyone else affected,

Accepted thunderbird into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/thunderbird/1:68.2.1+build1-0ubuntu0.19.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Please note that normally this package goes via the security workflow, but this time we decided to push it through -proposed. Since this is not a regular SRU (and is targeted to go into -security in the end), the verification and release should not be performed without a +1 from the uploader and security team.

SRU Team: please ignore this bug while performing package releases for now. I'll take care of it once we deem it good to go. Thank you.

[Łukasz Zemczak (sil2100)]

Hello Jean-Louis, or anyone else affected,

Accepted thunderbird into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/thunderbird/1:68.2.1+build1-0ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Fiona Klute (fiona-klute)]

After installing 68.2.1+build1-0ubuntu0.19.10.1 from eoan-proposed the system dictionaries show up as expected.

[Olivier Tilloy (osomon)]

I installed thunderbird 1:68.2.1+build1-0ubuntu0.18.04.1 from bionic-proposed, and the system dictionaries in /usr/share/hunspell were loaded.

[Launchpad Janitor (janitor)]

This bug was fixed in the package thunderbird - 1:68.2.1+build1-0ubuntu0.19.10.1

---------------
thunderbird (1:68.2.1+build1-0ubuntu0.19.10.1) eoan; urgency=medium

  * New upstream stable release (68.2.1build1)

  * Ensure that external dictionaries are still loaded (LP: #1847247)
    - debian/thunderbird.sh.in
  * Export MOZ_SOURCE_REPO and MOZ_SOURCE_CHANGESET during the configure step
    as this is now mandatory for the build system to determine and expose the
    comm repository and revision used to generate the source tarball
    (see https://bugzilla.mozilla.org/show_bug.cgi?id=1507754)
    - debian/build/create-tarball.py
    - debian/build/rules.mk
  * Add an explicit build dependency on libclang-dev
    - debian/control{,.in}
  * Fix the repacking of lightning with all locales (the artifact was renamed
    to lightning.xpi, see https://bugzilla.mozilla.org/show_bug.cgi?id=1588659)
    - debian/build/rules.mk

 -- Olivier Tilloy <email address hidden> Thu, 31 Oct 2019 21:51:55 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package thunderbird - 1:68.2.1+build1-0ubuntu0.18.04.1

---------------
thunderbird (1:68.2.1+build1-0ubuntu0.18.04.1) bionic; urgency=medium

  * New upstream stable release (68.2.1build1) (LP: #1850064)

  * Vendor cbindgen and its dependencies in the source tarball, because it's
    not available in Debian/Ubuntu yet
    - update debian/build/create-tarball.py
    - update debian/build/rules.mk
    - update debian/config/mozconfig.in
    - update debian/config/tarball.conf
  * Bump build dependency on rustc >= 1.34.0 and cargo >= 0.35
    - update debian/control{,.in}
  * Add a build dependency on Node.js 8.11
    - update debian/config/mozconfig.in
    - update debian/control{,.in}
  * Add a build dependency on nasm 2.13.02
    - update debian/control{,.in}
  * Update exclusion patterns to remove outdated entries and to trim down the
    final size of the source tarball
    - update debian/config/tarball.conf
  * Do not install obsolete extension
    - update debian/thunderbird.install.in
  * Update patches
    - update debian/patches/reduce-rust-debuginfo-on-32bit-architectures.patch
    - update debian/patches/rust-drop-dll-checksums.patch
    - update debian/patches/support-coinstallable-trunk-build.patch
    - update debian/patches/unity-menubar.patch
  * Remove obsolete patches
    - remove debian/patches/dont-treat-tilde-as-special.patch
    - remove debian/patches/fix-missing-docs-error.patch
    - remove debian/patches/skia-big-endian.patch
  * Stop building and remove thunderbird-globalmenu (LP: #1834866)
  * Fix build on armhf by disabling unaligned FP accesses emulation
    - add debian/patches/armhf-disable-unaligned-fp-access-emulation.patch
  * Reduce the memory used by the linking process on armhf, to work around
    build failures on launchpad
    - add debian/patches/armhf-reduce-linker-memory-use.patch
  * Update shipped locales (adding cak, ka, uz)
    - update debian/config/locales.{all,shipped}
    - update debian/control
  * Build with clang
    - update debian/config/mozconfig.in
    - update debian/control{,.in}
    - add debian/patches/armhf-clang-no-integrated-as-for-neon.patch
    - add debian/patches/ppc64el-workaround-bug-1555531.patch
    - add debian/patches/s390x-fix-hidden-symbol.patch
  * Ensure that external dictionaries are still loaded (LP: #1847247)
    - debian/thunderbird.sh.in
  * Export MOZ_SOURCE_REPO and MOZ_SOURCE_CHANGESET during the configure step
    as this is now mandatory for the build system to determine and expose the
    comm repository and revision used to generate the source tarball
    (see https://bugzilla.mozilla.org/show_bug.cgi?id=1507754)
    - debian/build/create-tarball.py
    - debian/build/rules.mk
  * Fix the repacking of lightning with all locales (the artifact was renamed
    to lightning.xpi, see https://bugzilla.mozilla.org/show_bug.cgi?id=1588659)
    - debian/build/rules.mk

 -- Olivier Tilloy <email address hidden> Thu, 31 Oct 2019 19:23:11 +0100

[Crxssi (crxssi)]

I am running vanilla Firefox 78.3.0 ESR under Linux and neither $DICPATH nor the spellchecker.dictionary_path preference is working (also not working in Firefox 68ESR). I have a customized medical dictionary that I have used for many years under older versions of Firefox that I now cannot use. I assume it is supposed to be as simple as:

$ ls /opt/dictionary
$ file *
en-US.aff: ASCII text
en-US.dic: UTF-8 Unicode text

$ export DICPATH="/opt/dictionary"
$ firefox

Yet it is not using the external dictionary I defined (showing words in that external dictionary as misspelled, still). I tried also adding export DICTIONARY="en-US" before launching and that didn't help. I am hoping someone might have some idea what is going on. Is this option now broken? And I using it wrong? I would open a new bug report if I were confident I am not doing something wrong, myself. Thanks

[Olivier Tilloy (osomon)]

@crxssi: I suggest opening a new bug report. The issue discussed here was closed as resolved, it would be better to start a new conversation in a separate bug report.

[Crxssi (crxssi)]

Thanks
Created new bug 1674520