Web Extension API, `browser.search.get` throws error `An unexpected error occurred`

[Anoop Elias]

Web Extension API, `browser.search.get` throws error `An unexpected error occurred` on ubuntu's version of firefox 68

Steps to reproduce

1. Extract attached ff-extensions.zip
2. In the extracted folder, run `web-ext run`
3. In address bar type about:debugging. Temporary extensions 'Test Extension' will be loaded.
4. Click 'Debug'

It gives an error in console tab, Error: "An unexpected error occurred"

This issue is reproducible only with the Firefox installations from 'Ubuntu Software', not with the Firefox downloads from Mozilla's page.

This is the possible root cause for "Tridactyl is broken on ubuntu's version of firefox 68", see https://github.com/tridactyl/tridactyl/issues/1670

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: firefox 68.0.1+build1-0ubuntu0.16.04.1
ProcVersionSignature: Ubuntu 4.15.0-55.60~16.04.2-generic 4.15.18
Uname: Linux 4.15.0-55-generic x86_64
AddonCompatCheckDisabled: False
ApportVersion: 2.20.1-0ubuntu2.19
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: anoop 2183 F.... pulseaudio
BuildID: 20190718135155
Channel: Unavailable
CurrentDesktop: Unity
Date: Mon Aug 19 01:09:47 2019
DefaultProfileExtensions: extensions.sqlite corrupt or missing
DefaultProfileIncompatibleExtensions: Unavailable (corrupt or non-existant compatibility.ini or extensions.sqlite)
DefaultProfileLocales: extensions.sqlite corrupt or missing
DefaultProfilePrefErrors: Unexpected character ',' before close parenthesis @ /usr/lib/firefox/omni.ja:greprefs.js:1141
DefaultProfilePrefSources: prefs.js
DefaultProfileThemes: extensions.sqlite corrupt or missing
ForcedLayersAccel: False
IfupdownConfig:
 # interfaces(5) file used by ifup(8) and ifdown(8)
 auto lo
 iface lo inet loopback
InstallationDate: Installed on 2018-01-08 (587 days ago)
InstallationMedia: Ubuntu 16.04.3 LTS "Xenial Xerus" - Release amd64 (20170801)
IpRoute:
 default via 192.168.1.1 dev wlp2s0 proto static metric 600
 169.254.0.0/16 dev docker0 scope link metric 1000 linkdown
 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
 192.168.1.0/24 dev wlp2s0 proto kernel scope link src 192.168.1.100 metric 600
ProcEnviron:
 LANGUAGE=en_IN:en
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_IN
 SHELL=/bin/bash
Profile1Extensions: extensions.sqlite corrupt or missing
Profile1IncompatibleExtensions: Unavailable (corrupt or non-existant compatibility.ini or extensions.sqlite)
Profile1Locales: extensions.sqlite corrupt or missing
Profile1PrefErrors: Unexpected character ',' before close parenthesis @ /usr/lib/firefox/omni.ja:greprefs.js:1141
Profile1PrefSources: prefs.js
Profile1Themes: extensions.sqlite corrupt or missing
Profiles:
 Profile1 - LastVersion=69.0/20190807220259 (Out of date)
 Profile0 (Default) - LastVersion=68.0.1/20190718135155 (In use)
RunningIncompatibleAddons: False
SourcePackage: firefox
UpgradeStatus: Upgraded to xenial on 2018-09-10 (342 days ago)
dmi.bios.date: 08/07/2017
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.5.3
dmi.board.name: 0G0G6Y
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 10
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr1.5.3:bd08/07/2017:svnDellInc.:pnPrecision3520:pvr:rvnDellInc.:rn0G0G6Y:rvrA00:cvnDellInc.:ct10:cvr:
dmi.product.family: Precision
dmi.product.name: Precision 3520
dmi.sys.vendor: Dell Inc.

[Ssborbis (ssborbis)]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

browser.search.get() run on ubuntu 18.04 64bit, FF 68.0.1 (debug console, web extension background page, etc )

Actual results:

An unexpected error occured

Expected results:

Promise resolves with array of one-click search engines

[Ehumphries (ehumphries)]

~~I need to check on FFx build vs Canonical build, Canonical may be removing it.~~

Edited: reporter did confirm it's a distro issue.

[Ssborbis (ssborbis)]

Confirmed it only occurs in the Canonical build. Tarball 68.0.1 works as expected

[Ehumphries (ehumphries)]

Thank you for checking that!

This is always frustrating with distro builds, they may disable features for any number of reasons, which can cause things like this.

One more step, if you have time. Can you install this sample extension on the Canonical build and see if it works or not?

https://github.com/mdn/webextensions-examples/tree/master/menu-search

https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Temporary_Installation_in_Firefox

[Ssborbis (ssborbis)]

Using menu-search as temp add-on

Canonical - fail ( Error: An unexpected error occurred .......... undefined )
Official - works as expected

[Mozilla-kaply (mozilla-kaply)]

This is working with other distros. Maybe something Snap related?

[Ssborbis (ssborbis)]

(In reply to Mike Kaply [:mkaply] from comment #5)
> This is working with other distros. Maybe something Snap related?

I had an addon user report this bug on ubuntu 18.04 64bit, confirmed on my vbox install. I'll test snap vs apt

[R-rob-c (r-rob-c)]

Cannot reproduce on Ubuntu 18.04, Firefox 68.0.1 (Canonical, installed via `apt`, Dutch locale).

If you are able to reproduce, could you check whether the global JavaScript console (Ctrl-Shift-J) contains any errors?

The error could be caused by https://searchfox.org/mozilla-central/rev/b38e3beb658b80e1ed03e0fdf64d225bd4a40327/browser/components/extensions/parent/ext-search.js#56
... when a search engine is defined with a non-existing icon URL.

[R-rob-c (r-rob-c)]

I went to https://packages.ubuntu.com/bionic/web/firefox
and downloaded the source package from http://archive.ubuntu.com/ubuntu/pool/main/f/firefox/firefox_68.0.1+build1-0ubuntu0.18.04.1.debian.tar.xz

Inside that package, there is `debian/searchplugins/amazondotcom.xml` (and many other `amazon*.xml` files). They reference `resource://search-plugins/images/amazon.ico`.
This file was removed in part 3 of bug 1496075, so any request to fetch that image URL will fail, and hence break `browser.search.get`.

[Ssborbis (ssborbis)]

(In reply to Rob Wu [:robwu] from comment #8)
> I went to https://packages.ubuntu.com/bionic/web/firefox
> and downloaded the source package from http://archive.ubuntu.com/ubuntu/pool/main/f/firefox/firefox_68.0.1+build1-0ubuntu0.18.04.1.debian.tar.xz
>
> Inside that package, there is `debian/searchplugins/amazondotcom.xml` (and many other `amazon*.xml` files). They reference `resource://search-plugins/images/amazon.ico`.
> This file was removed in part 3 of bug 1496075, so any request to fetch that image URL will fail, and hence break `browser.search.get`.

Yep, simply deleting the Amazon engine without an icon fixed the issue. I'll let the user know.

[R-rob-c (r-rob-c)]

Created attachment 9082493
Bug 1570700 - Drop support for resource/chrome icons in search engines

All built-in engines have migrated from OpenSearch to WebExtensions.
WebExtensions do not support resource: or chrome:-URLs in their
`favicon_url` field, so the "resource:" and "chrome:" URLs can only be
used by external opensearch XML files. These should not rely on internal
resources from omni.ja, as the bug shows. So just drop support for
"chrome:" and "resource:"-URLs, as we don't need them any more.

Current OpenSearch engines that relied on chrome/resource:-URLs can
either replace the URL with a data:-URL, or migrate to WebExtensions.

[R-rob-c (r-rob-c)]

(linking to meta bug 1517486, because the proposed patch removes a feature that is not supported by the WebExtension format)

[Ehumphries (ehumphries)]

Thanks everyone for their help getting to next steps on this bug! Y'all rock. 🙌

[Mozilla-kaply (mozilla-kaply)]

I'll fix the URLs in the partner distro repository.

[Pulsebot (pulsebot)]

Pushed by <email address hidden>:
https://hg.mozilla.org/integration/autoland/rev/778471f56de7
Drop support for resource/chrome icons in search engines r=mkaply,daleharvey,mixedpuppy

[Ncsoregi (ncsoregi)]

https://hg.mozilla.org/mozilla-central/rev/778471f56de7

[Ryanvm (ryanvm)]

Is this something we should consider for uplift or can it ride Fx70 to release?

[Mozilla-kaply (mozilla-kaply)]

> Is this something we should consider for uplift or can it ride Fx70 to release?

Getting it on 69 would be nice, but the Snap problem specifically will be fixed by a change to their distro.

[R-rob-c (r-rob-c)]

Let's ride the train. The number of users that are affected by this is probably small. A work-aroumd for them is to delete the (opensearch) file/engine.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mcurtean (mcurtean)]

Created attachment 9088118
console browser search.png

Hello,
I've tried to reproduce this on Ubuntu 18.04.2 LTS 64-bit on Firefox Release 68.0.1 by downloading and temporarily adding in about:debugging the "menu-search" extension manifest from comment 3.
After which debugging mode was enabled, a Debug performed and also I used the the browser.search.get() function in the console.
No error was received; for more details please check the "console browser search" attachment.
If manual QA is still needed on this issue, could you please add the "qe+" verify flag and also provided more steps or details in order to test?
Thank you

[R-rob-c (r-rob-c)]

There is automated test coverage.

[Olivier Tilloy (osomon)]

I can confirm, I am able to reproduce the issue with the firefox ubuntu packages in a fully up-to-date eoan VM, whereas the official firefox builds don't exhibit the problem.

This requires further investigation.

[Olivier Tilloy (osomon)]

Reference documentation for that Web Extension API: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/search/get

[Olivier Tilloy (osomon)]

Upstream bug report: https://bugzilla.mozilla.org/show_bug.cgi?id=1570700

[Olivier Tilloy (osomon)]

I am testing a backport of https://hg.mozilla.org/mozilla-central/rev/778471f56de7

[Launchpad Janitor (janitor)]

This bug was fixed in the package firefox - 69.0+build2-0ubuntu0.16.04.4

---------------
firefox (69.0+build2-0ubuntu0.16.04.4) xenial; urgency=medium

  * Backport an upstream commit to fix the browser.search.get()
    Web Extension API (LP: #1840597)
    - add fix-browser-search-get-api.patch

 -- Olivier Tilloy <email address hidden> Wed, 28 Aug 2019 12:49:47 +0200

[Seth Arnold (seth-arnold)]

Thanks osomon! This is so much better. Fixed in 69.0+build2-0ubuntu0.18.04.1.