[snap] chrome-gnome-shell extension fails to detect native host connector

[Olivier Tilloy]

(initially reported at https://forum.snapcraft.io/t/chrome-gnome-shell-does-not-work-with-chromium-snap/3377)

See attached screenshot.

[Workaround]
If you're using Ubuntu 22.04 LTS, you can install GNOME Shell extensions with this app.

sudo apt install gnome-shell-extension-manager

[Olivier Tilloy (osomon)]

stracing the snap reveals that /usr/bin/chrome-gnome-shell cannot be found in the context of the snap:

  15760 access("/usr/bin/chrome-gnome-shell", F_OK) = -1 ENOENT (No such file or directory)

This is not an apparmor denial. Inside the snap’s execution environment, /usr/bin is in fact /snap/core/current/usr/bin.

[Olivier Tilloy (osomon)]

Similar report for the Plasma/Browser Integration (https://community.kde.org/Plasma/Browser_Integration): https://forum.snapcraft.io/t/call-for-testing-chromium-62-0-3202-62/2569/65.

[Emanuele (emanuc)]

Is there any news about this bug?
To this day I still can't use plasma-browser-integration

[Olivier Tilloy (osomon)]

I've toyed with an unpacked example extension that talks to a native host connector (a simple python script that listens to messages and echoes all it receives).

This works with the following caveats:

 - the connector has to be installed under $SNAP_USER_DATA/.config/chromium/NativeMessagingHosts/

 - the path to the connector in the manifest needs to be an absolute path (as documented at https://developers.chrome.com/extensions/nativeMessaging), e.g.:
    "path": "/home/osomon/snap/chromium/current/.config/chromium/NativeMessagingHosts/native-messaging-example-host"

[Olivier Tilloy (osomon)]

chrome-gnome-shell wouldn't work like that unmodified, because the connector has external dependencies (it's a python3 script that imports GLib and Gio).

[Olivier Tilloy (osomon)]

Also, the connector talks to the org.gnome.Shell.Extensions D-Bus API, for which there doesn't exist a snapd interface, so one would need to be written, and I'm pretty sure it wouldn't qualify for auto-connection given the level of privilege it would expose (listing installed extensions, install and uninstall extensions).

[Russ Dill (russ-dill)]

This is really frustrating and 19.10 is coming up fast which I understand will switch everyone over to snap based chromium making extensions.gnome.org unusable for everyone. Any chance of switching back to non-snap based chromium if this issue isn't resolved before then?

[Olivier Tilloy (osomon)]

Unlikely. There are a number of higher priority issues with the chromium snap that we're focusing on now, and the plan to transition from the deb to the snap in 19.10 still holds.

I know this isn't a proper fix, merely a workaround, but you can use gnome-tweaks to manage your gnome-shell extensions.

[zebul666 (zebul666)]

How and why that snap has been rolled out in 19.10 while we have less feature with it ?

Note: I don't like snap.
This bug is another example why. Moreover nobody cares to fix this. If it is even possible...

[Paul White (paulw2u)]

zebul666, please take time to read:

https://discourse.ubuntu.com/t/intent-to-provide-chromium-as-a-snap-only/5987
https://discourse.ubuntu.com/t/please-test-chromium-snap-as-a-replacement-for-the-deb/7978
https://discourse.ubuntu.com/t/call-for-testing-chromium-browser-deb-to-snap-transition/11179

Those threads should answer your questions as to why the Chromium browser will only be provided as a snap in Ubuntu 19.10 and eventually in all supported releases of Ubuntu.

[sbaragnaus (sbaragnaus)]

This bug is still present in Kubuntu 19.10. Poor Chromium...

[Proton Pony (protonpony)]

Chromium is now incapable of managing extensions for Ubuntu's primary desktop environment? This issue is tolerable only because Google Chrome is not affected. Is Google somehow behind this apparent reluctance to remedy the problem?

[Olivier Tilloy (osomon)]

It's been reported that the GooPG extension doesn't work because it uses native messaging (https://discourse.ubuntu.com/t/call-for-testing-chromium-browser-deb-to-snap-transition/11179/207).

[Olivier Tilloy (osomon)]

And from duplicate bug #1853835, the KeePassXC-Browser extension (https://chrome.google.com/webstore/detail/keepassxc-browser/oboonakemofpalcgghocfoadofidjkkk) doesn't work for the same reason: https://github.com/keepassxreboot/keepassxc-browser/issues/405.

[Coeur Noir (coeur-noir)]

2020-01-06 : https://forum.ubuntu-fr.org/viewtopic.php?pid=22202596#p22202596

It seems still broken in not-yet-released ubuntu 20.04.

[Avamander (avamander)]

KDE Connect is also affected by this. I'm wondering though, unable to find the answer online, why was the transition to snap-based chromium made?

[Efthimios Chaskaris (echaskaris)]

(I think) It makes development easier, you don't have to build it for every Ubuntu version in support, or any Linux OS, it contains specific libraries you know won't cause issues with the app.

[Horst Baerbel (horstbaerbel)]

I want to add this affects Firefox + the KeepassXC extension too. I agree that NativeMessaging is kind of a crutch, but if there is no way to make this work safely a snap interface needs to be implemented that applications from inside / outside of snap can use. There is demand for this kind of interface, so maybe devising a "correct" way of doing this and communicating it to the application / plugin developers would be a good thing...
At the moment I work around by installing firefox from repo and ignore the issue, but some day this might not work anymore. I'm kind of worried that I upgrade to Ubuntu 20.04 and Firefox is not available from repo anymore :/

[Balazs Gyurak (ba32107)]

Hi! Any update on this please? This is an extremely frustrating issue that makes me consider switching from Ubuntu to something else.

[Alberto Donato (ack)]

This is still an issue on 20.04

[Jeffrey Bouter (jbouter)]

This also affects the browserpass extension for the pass (passwordstore.org) password manager

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ian Johnson (anonymouse67)]

@osomon, could you point me to an existing forum post or start a new one summarizing what accesses would be necessary to manage gnome shell extensions? I couldn't find one searching, but if there's not one we should open one to try and discuss with snapd architects. Thanks

[Olivier Tilloy (osomon)]

@anonymouse67: see comments #2, #5, #6 and #7 in this bug.

[Farab (farabf33)]

This "connector" issue is also presents on firefox snap v77.0b2 (64-bit) Ubuntu 20.04 and has not addressed yet.

[Bruce Crowther (bwucie)]

Harrumph: not impressed. Was working fine this morning, then an update landed me with the snap version.

Back to gnome-tweaks...

[Bruce Crowther (bwucie)]

Nope, never mind gnome-tweaks. Removed Chromium snap and installed Google Chrome. Problem goes away.

[Marcin (mwoz123)]

Any changes on this bug (since 2018)? Maybe at least workaround?

I confirm same problem on Focal 20.04 LTS with snap Opera (chrome based)

[Efthimios Chaskaris (echaskaris)]

Download firefox from its website, just run it from it's unzipped directory

[Olivier Tilloy (osomon)]

One suggestion discussed in https://forum.snapcraft.io/t/allow-classic-confinement-for-postman-agent/19228 is to use a content interface to allow third-party snaps to write to ~/snap/chromium/common/chromium/NativeMessagingHosts/. Whether it’s a good idea from a security perspective is open for discussion, and it wouldn't solve all the problems discussed here, but it could be part of the solution to enable third-party native host connectors.

[lerra (lerra82)]

Hi team, is there any news about this 3y old bug? It would be nice to know if it is something that will be worked on or not even if the timeline indicates towards not.

[udo (udopton)]

Although GNOME Shell integration extension is running, native host connector is not detected. Refer documentation for instructions about installing connector.

[SunBear (sunbear-c22)]

For Ubuntu 20.04, I have also noticed that both snap installed Chromium and Brave Web browsers fails to work with the gnome-shell-integration extension.

In the case of Brave, this issue can be overcome by following their instruction on installing the .deb versions of brave-browser and brave-keyring files.
$ sudo apt install apt-transport-https curl
$ sudo curl -fsSLo /usr/share/keyrings/brave-browser-archive-keyring.gpg https://brave-browser-apt-release.s3.brave.com/brave-browser-archive-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/brave-browser-archive-keyring.gpg arch=amd64] https://brave-browser-apt-release.s3.brave.com/ stable main"|sudo tee /etc/apt/sources.list.d/brave-browser-release.list
$ sudo apt update
$ sudo apt install brave-browser

However, the chromium-browser does not have a .deb version to fix this issue. If one does issue the command:
sudo apt install chromium-browser,
this command will instead instruct snap to install chromium.

Hope the Chromium developer community can fix this issue given it was first reported in 2018-01-03. Chroumium adoption would be made easier too. Thank you.

[Merlijn Sebrechts (merlijn-sebrechts)]

I want to make this a bit more clear:

Every single Belgian needs this functionality in order to do basic stuff like:

* File their taxes
* Read communication from the government
* Check their medical history
* Register for a (covid) vaccine
* Rent stuff from the government
* etc.

It is vital that this issue is fixed before phasing out the Firefox deb package. With the Chromium apt package gone, there will be no alternative in the Ubuntu repositories to do these basic things in Belgium.

It’s really nice that our government is supporting Ubuntu using open-source software (https://github.com/Fedict/eid-mw) using common standards and APIs. Breaking this vital functionality looks like a big middle-finger to them.

[Olivier Tilloy (osomon)]

Thanks for these useful data points Merlijn. That is certainly an important use case that we want to continue supporting. I had a quick look at both the github repository and the "Belgium eID" firefox extension (https://addons.mozilla.org/en-US/firefox/addon/belgium-eid/), and I can't find any references to nativeMessaging there. Am I missing another piece of the puzzle? Or could it be that this is a different problem altogether?

[Olivier Tilloy (osomon)]

And for future reference, here is Mozilla's documentation on native messaging: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Native_messaging.

[Olivier Tilloy (osomon)]

I'm quoting James Henstridge on a possible approach to tackle the problem:

« The design of the native messaging system for extensions is something that could theoretically fit into a confinement system like snapd or flatpak. Instead of executing the native messaging server directly, the browser could ask something outside the sandbox to execute the server and hand it the communication pipes.

That's quite a large project though: it'd require modifications to the browser(s), design and implementation of an API (maybe in xdg-desktop-portal?), evaluation of the security implications, etc. »

[Sebastien Bacher (seb128)]

> there will be no alternative in the Ubuntu repositories to do these basic things in Belgium.

using epiphany (the GNOME browser) should work no?

[Renosifana-paksi (renosifana-paksi)]

(In reply to David D Lowe from comment #3)
> This bug is also being tracked at https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/1741074
>
> Ubuntu 21.10 ships the snap version of Firefox by default, (instead of the APT version), so we can expect more users to experience this bug.

I have same problem. Few days ago i have installed Kubuntu 23.10 "Mantic Minotaur" replacing Linux Mint 21.2 "Victoria". the extension doesn't work as well (Including KDE Browser Integration, a Download Manager (Persepolis Download Manager Integration, Aria-NG Integration)). I have grant Permission but nothing happen lol.

[Renosifana-paksi (renosifana-paksi)]

Created attachment 9365105
Screenshot_20231123_073703.png

KDE Plasma Integration on Firefox 120.

[2-david-u (2-david-u)]

Same probleme here with ubuntu 24.04.1, and firefox snap.

"Failed to connect to the native host."

[Amin Bandali (bandali)]

(In reply to david from comment #22)
> Same probleme here with ubuntu 24.04.1, and firefox snap.
>
> "Failed to connect to the native host."

Hello, could you please provide more information about this?

- Is this a fresh Ubuntu 24.04 LTS installation? Or did you upgrade from an existing Ubuntu release to this one? (If yes, from which version?)
- Please open a terminal, run the following commands, and provide the output here:
```
apt list --installed | grep xdg-desktop-portal
apt-cache policy xdg-desktop-portal
snap version
```
- Which Firefox version are you using?
- Which Firefox add-on are you trying to use here? And did you add a manifest for it under the `~/.mozilla/native-messaging-hosts/` directory?
- Are there any more error or warning messages in the logs beyond the one line you provided?

Thanks.

[2-david-u (2-david-u)]

(In reply to Amin Bandali [:bandali] from comment #23)
> (In reply to david from comment #22)
> > Same probleme here with ubuntu 24.04.1, and firefox snap.
> >
> > "Failed to connect to the native host."
>
> Hello, could you please provide more information about this?
>
> - Is this a fresh Ubuntu 24.04 LTS installation? Or did you upgrade from an existing Ubuntu release to this one? (If yes, from which version?)
> - Please open a terminal, run the following commands, and provide the output here:
> ```
> apt list --installed | grep xdg-desktop-portal
> apt-cache policy xdg-desktop-portal
> snap version
> ```
> - Which Firefox version are you using?
> - Which Firefox add-on are you trying to use here? And did you add a manifest for it under the `~/.mozilla/native-messaging-hosts/` directory?
> - Are there any more error or warning messages in the logs beyond the one line you provided?
>
> Thanks.

Hi,

- This is not a fresh ubuntu install (its a upgraded version from ubuntu 22.04 LTS), BUT, I'm using a fresh account to test (new user, new /home).
- This is the output of the commands:

apt list --installed | grep xdg-desktop-portal

    xdg-desktop-portal-gtk/noble,now 1.15.1-1build2 amd64 [installed,automatic]
    xdg-desktop-portal-kde/noble,now 5.27.11-0ubuntu3 amd64 [installed,automatic]
    xdg-desktop-portal/noble-updates,now 1.18.4-1ubuntu2 amd64 [installed,automatic]

apt-cache policy xdg-desktop-portal

    xdg-desktop-portal:
      Installed: 1.18.4-1ubuntu2
      Candidate: 1.18.4-1ubuntu2
      Version table:
     *** 1.18.4-1ubuntu2 500
            500 http://fr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages
            100 /var/lib/dpkg/status
         1.18.3-1ubuntu1 500
            500 http://fr.archive.ubuntu.com/ubuntu noble/main amd64 Packages

snap version

    snap 2.63.1+24.04
    snapd 2.63.1+24.04
    series 16
    ubuntu 24.04
    kernel 6.8.0-41-generic

- Firefox version is 130.0 (64-bit)
- The firefox addon I'm trying to use is: https://addons.mozilla.org/en-US/firefox/addon/plasma-integration/. I did **not** add a manifest for it under ~/.mozilla/native-messaging-hosts/ directory.
- On the extension console, I can see the messages:

    Invalid url undefined action_popup.js:86:33
    Failed to check for whether media controls are blocked NO_ORIGINS action_popup.js:279:17

and in about:debugging#/runtime/this-firefox i see:

    Warning details
    Reading manifest: Warning processing key: An unexpected property was found in the WebExtension manifest.

if I try to reload the addon I get:

    Host disconnected An unexpected error occurred extension.js:204:17
    Not auto-restarting host as we haven't received any message from it before. Check that it's working/installed correctly extension.js:226:21
    Background event page was not terminated on idle because a DevTools toolbox is attached to the extension. _generated_background_page.html

[Lissyx+mozillians (lissyx+mozillians)]

```
$ pip install --no-use-pep517 python-dbusmock==0.32.2
Collecting python-dbusmock==0.32.2
  Using cached python-dbusmock-0.32.2.tar.gz (104 kB)
ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml
```

[Lissyx+mozillians (lissyx+mozillians)]

Kagami in https://bugzilla.mozilla.org/show_bug.cgi?id=1868690#c10 you mentionned `--no-use-pep517` as temporary? Can we remove it?

[Lissyx+mozillians (lissyx+mozillians)]

```
(venv-python-dbusmock) alex@portable-alex:~/tmp/mozillasupport$ pip install --upgrade setuptools==70.0.0 wheel==0.43.0
Collecting setuptools==70.0.0
  Using cached setuptools-70.0.0-py3-none-any.whl.metadata (5.9 kB)
Collecting wheel==0.43.0
  Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
Using cached setuptools-70.0.0-py3-none-any.whl (863 kB)
Using cached wheel-0.43.0-py3-none-any.whl (65 kB)
Installing collected packages: wheel, setuptools
  Attempting uninstall: wheel
    Found existing installation: wheel 0.44.0
    Uninstalling wheel-0.44.0:
      Successfully uninstalled wheel-0.44.0
  Attempting uninstall: setuptools
    Found existing installation: setuptools 75.2.0
    Uninstalling setuptools-75.2.0:
      Successfully uninstalled setuptools-75.2.0
Successfully installed setuptools-70.0.0 wheel-0.43.0
(venv-python-dbusmock) alex@portable-alex:~/tmp/mozillasupport$ pip install python-dbusmock==0.32.2
Collecting python-dbusmock==0.32.2
  Using cached python-dbusmock-0.32.2.tar.gz (104 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: dbus-python in ./venv-python-dbusmock/lib/python3.12/site-packages (from python-dbusmock==0.32.2) (1.3.2)
Building wheels for collected packages: python-dbusmock
  Building wheel for python-dbusmock (pyproject.toml) ... done
  Created wheel for python-dbusmock: filename=python_dbusmock-0.32.2-py3-none-any.whl size=73663 sha256=838a8a57b4240b19003b00188a1e63c90f6b4f5f19d0ca8e7c0803ffec0dcc95
  Stored in directory: /home/alex/.cache/pip/wheels/fd/95/73/9cc598f34c15d81a7a4b400328064875b72d637c10ab236ae7
Successfully built python-dbusmock
Installing collected packages: python-dbusmock
Successfully installed python-dbusmock-0.32.2
```

Not sure why it fails on CI: https://treeherder.mozilla.org/logviewer?job_id=478870258&repo=try

[Krosylight (krosylight)]

(In reply to :gerard-majax from comment #26)
> Kagami in https://bugzilla.mozilla.org/show_bug.cgi?id=1868690#c10 you mentionned `--no-use-pep517` as temporary? Can we remove it?

That would require https://bugzilla.mozilla.org/show_bug.cgi?id=1868690#c3 option 2. I don't know whether anyone did it, maybe ahal knows.

[Ahal (ahal)]

No work has been done or planned here afaik,

But yeah, if I'm understanding the situation, `--no-use-pep517` is just a bandaid to allow our outdated infrastructure to chug along a little while longer. But at some point someone is going to have to do the actual work of updating our tooling to modern Python standards.

[Lissyx+mozillians (lissyx+mozillians)]

(In reply to Andrew Halberstadt [:ahal] from comment #29)
> No work has been done or planned here afaik,
>
> But yeah, if I'm understanding the situation, `--no-use-pep517` is just a bandaid to allow our outdated infrastructure to chug along a little while longer. But at some point someone is going to have to do the actual work of updating our tooling to modern Python standards.

Thanks. Does it means we cannot install this package until that work is done?

[Ahal (ahal)]

I think so.. I don't remember why we added `--no-use-pep517` in the first place, it looks like it might have been to improve performance? If so we could alternatively back that patch out.

[Krosylight (krosylight)]

Wasn't `--no-use-pep517` to skip building wheel? It still builds wheel here and then fails, so not sure why the flag is relevant here.

[Lissyx+mozillians (lissyx+mozillians)]

So, how should such a package installed ?

[Krosylight (krosylight)]

All I can say is that `pip install --timeout 120 --no-index --find-links https://pypi.pub.build.mozilla.org/pub/ python-dbusmock==0.32.2` (the command from the CI log in comment #27) fails on my local machine too with the same error. Not sure why it behaves differently from `pip install python-dbusmock==0.32.2`.

[Lissyx+mozillians (lissyx+mozillians)]

interesting, it also fails for me, that's not something i would have expected

[Lissyx+mozillians (lissyx+mozillians)]

When it works i can see:
```
  Using cached setuptools-75.2.0-py3-none-any.whl (1.2 MB)
  Using cached setuptools_scm-8.1.0-py3-none-any.whl (43 kB)
  Using cached packaging-24.1-py3-none-any.whl (53 kB)
  Installing collected packages: setuptools, packaging, setuptools-scm
  Successfully installed packaging-24.1 setuptools-75.2.0 setuptools-scm-8.1.0
```

When it does not we install other versions:
```
  Looking in links: https://pypi.pub.build.mozilla.org/pub/
  Collecting setuptools>=45
    Using cached https://pypi.pub.build.mozilla.org/pub/setuptools-69.0.3-py3-none-any.whl (819 kB)
  Collecting setuptools-scm
    Using cached https://pypi.pub.build.mozilla.org/pub/setuptools_scm-3.3.3-py2.py3-none-any.whl (23 kB)
  Installing collected packages: setuptools-scm, setuptools
  Successfully installed setuptools-69.0.3 setuptools-scm-3.3.3
  Installing build dependencies: finished with status 'done'
```

So maybe the setuptools version we have on our Pypi mirror is not recent enough? Or it's setuptools-scm?

[Lissyx+mozillians (lissyx+mozillians)]

Making a local dumb pypi mirror i could confirm:
 - unable to install with `setuptools-69.0.3 setuptools-scm-3.3.3`
 - able to install with ` packaging-23.1 setuptools-75.2.0 setuptools-scm-8.1.0`

Maybe we could have the wheel instead of the tar of dbusmock a wheel on our pypi mirror?

[Lissyx+mozillians (lissyx+mozillians)]

Thanks to both `:ahal` and `:saschanaz` this seems to work now on this push https://treeherder.mozilla.org/jobs?repo=try&author=alissy%40mozilla.com&selectedTaskRun=LwPGCLVrQuyRDIDVvuJHVg.0

[Lissyx+mozillians (lissyx+mozillians)]

Created attachment 9433833
Bug 1661935 - Native Messaging WebExtension documentation r?robwu!

[Lissyx+mozillians (lissyx+mozillians)]

Created attachment 9433834
Bug 1661935 - Native Messaging WebExtensions portal tests r?robwu!

[Lissyx+mozillians (lissyx+mozillians)]

Created attachment 9433859
GitHub Pull Request

Upstream removal of the patch PR against nightly

[Lissyx+mozillians (lissyx+mozillians)]

*** Bug 1621763 has been marked as a duplicate of this bug. ***

[Lissyx+mozillians (lissyx+mozillians)]

Created attachment 9434051
GitHub Pull Request

Correct PR to remove the patch on nightly branch

[Pulsebot-d (pulsebot-d)]

Pushed by <email address hidden>:
https://hg.mozilla.org/integration/autoland/rev/594beeb5e424
Integration with a new WebExtensions XDG desktop portal for native messaging on Linux r=emilio,robwu,rpl,Gijs,andi,mach-reviewers
https://hg.mozilla.org/integration/autoland/rev/7b10b4e274db
Native Messaging WebExtension documentation r=robwu,rpl
https://hg.mozilla.org/integration/autoland/rev/620790051502
Native Messaging WebExtensions portal tests r=robwu,ahal

[Agoloman (agoloman)]

Backed out for causing xpcshell failures @@CycleCollectedJSContext.
* [Backout link](https://hg.mozilla.org/integration/autoland/rev/601c211c7757ae72b443570b566af254ced28415)
* [Push with failures](https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Csuccess%2Crunning%2Cpending%2Crunnable&searchStr=os%2Cx%2C10.15%2Cwebrender%2Copt%2Cxpcshell%2Ctests%2Ctest-macosx1015-64-qr%2Fopt-xpcshell%2Cx1&tochange=a5b67007fef784f72829151e51325f6ff335bce0&fromchange=25138ac6e597c2fe5a90a99ef6d62b92f235f1ab)
* [Failure Log](https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Csuccess%2Crunning%2Cpending%2Crunnable&searchStr=os%2Cx%2C10.15%2Cwebrender%2Copt%2Cxpcshell%2Ctests%2Ctest-macosx1015-64-qr%2Fopt-xpcshell%2Cx1&tochange=a5b67007fef784f72829151e51325f6ff335bce0&fromchange=25138ac6e597c2fe5a90a99ef6d62b92f235f1ab)

[Pulsebot-d (pulsebot-d)]

Backout by <email address hidden>:
https://hg.mozilla.org/integration/autoland/rev/59ce3da9e504
Backed out 3 changesets for causing xpc failures @test_subprocess.js.

[Lissyx+mozillians (lissyx+mozillians)]

(In reply to Pulsebot from comment #46)
> Backout by <email address hidden>:
> https://hg.mozilla.org/integration/autoland/rev/59ce3da9e504
> Backed out 3 changesets for causing xpc failures @test_subprocess.js.

So it looks like we are facing some bad case of a race condition here. We create `running` spawning a process and then we `connectRunning` with its FDs and get a second `proc` process. There `running.stdout` is `id: 1` and had `fd: 16` when `proc.stdin` is `id: 5` with `fd: 16` which seems to be consistent with the expected behavior of those pipes. It means we have two `InputPipe` that shares the same FD. Upon creation they `fillBuffer` https://searchfox.org/mozilla-central/rev/b1cd7e191ac57f8dfe31806be2539e51843c03eb/toolkit/modules/subprocess/subprocess_common.sys.mjs#295. So both enqueues a `read ` call. Somehow the `pollfd.revents` https://searchfox.org/mozilla-central/rev/b1cd7e191ac57f8dfe31806be2539e51843c03eb/toolkit/modules/subprocess/subprocess_unix.worker.js#569 turns `!= 0`for the `InputPipe` of `id:5` before the `id:1` and so we trigger reading the content of the pipe, which is then not available anymore to `id:1`.

[Lissyx+mozillians (lissyx+mozillians)]

Created attachment 9437056
Bug 1661935 - Introduce ManagedProcess and connectRunning()

[Lissyx+mozillians (lissyx+mozillians)]

Created attachment 9437156
Bug 1661935 - Properly close process from connectRunning() r?mossop!

[Lissyx+mozillians (lissyx+mozillians)]

Created attachment 9438554
Bug 1661935 - Introduce Subprocess.connectRunning() and ManagedProcess r?robwu!

[Pulsebot-d (pulsebot-d)]

Pushed by <email address hidden>:
https://hg.mozilla.org/integration/autoland/rev/a1bfd8d6189b
Introduce Subprocess.connectRunning() and ManagedProcess r=robwu

[R-rob-c (r-rob-c)]

(adding leave-open keyword to prevent the bug from being marked as RESOLVED FIXED when the above patch reaches Nightly. The above patch is a dependency of the full implementation, but not the feature tracked by this bug)

[Pulsebot-d (pulsebot-d)]

Pushed by <email address hidden>:
https://hg.mozilla.org/integration/autoland/rev/0533ca878434
Integration with a new WebExtensions XDG desktop portal for native messaging on Linux r=emilio,robwu,rpl,andi,mach-reviewers
https://hg.mozilla.org/integration/autoland/rev/e46ea28ef686
Native Messaging WebExtension documentation r=robwu,rpl
https://hg.mozilla.org/integration/autoland/rev/f1cfeb19c642
Native Messaging WebExtensions portal tests r=robwu,ahal

[Sstanca (sstanca)]

https://hg.mozilla.org/mozilla-central/rev/a1bfd8d6189b

[Lissyx+mozillians (lissyx+mozillians)]

(In reply to Rob Wu [:robwu] from comment #52)
> (adding leave-open keyword to prevent the bug from being marked as RESOLVED FIXED when the above patch reaches Nightly. The above patch is a dependency of the full implementation, but not the feature tracked by this bug)

Thanks, I got logged out by Lando when pushing and after reauth it subtly selected back the first patch of the stack, but the rest was pushed now

[Nfay (nfay)]

https://hg.mozilla.org/mozilla-central/rev/0533ca878434
https://hg.mozilla.org/mozilla-central/rev/e46ea28ef686
https://hg.mozilla.org/mozilla-central/rev/f1cfeb19c642

[Jhorak (jhorak)]

I wonder if having the GetManifest() and Start() actually makes any sense in the Firefox. Could be sufficient enough to pass the manifest lookup and parsing to the portal and from the Firefox side only use CreateSession(requested_webextension_name, extension_id)->GetPipes()->Close() methods?

We've been discussing this with swick regarding to https://github.com/flatpak/xdg-desktop-portal/pull/705 - because it also have to parse the webextension json files.

He also brought idea to have webextension itself in the flatpak where we hit that the json files won't be accessible from the host system. The webextensions then will have to implement dbus interface to pass pipes (as it works with this patch) and the flatpak portal could stay in between to make it happen. From the Firefox point of view it won't require any additional changes. Only portal have to do all the lifting.

[R-rob-c (r-rob-c)]

Hi Jan, I am also involved as a reviewer in the portal PR you referenced, so we can continue the discussion there (my Github handle is `Rob--W` ).

> I wonder if having the GetManifest() and Start() actually makes any sense in the Firefox. Could be sufficient enough to pass the manifest lookup and parsing to the portal and from the Firefox side only use CreateSession(requested_webextension_name, extension_id)->GetPipes()->Close() methods?

It is an intentional design decision to forward the full native messaging manifest from the portal to the browser/Firefox, because it offers all relevant information to Firefox. The portal's primary responsibility is to do things that Firefox cannot, which is launching applications outside of the confinement. The fact that the portal should and does also perform additional validation does not preclude Firefox from doing the same. The relevant discussion is in this thread on an old revision of the patch: https://phabricator.services.mozilla.com/D140803?id=626421#inline-868567

> He also brought idea to have webextension itself in the flatpak where we hit that the json files won't be accessible from the host system. The webextensions then will have to implement dbus interface to pass pipes (as it works with this patch) and the flatpak portal could stay in between to make it happen. From the Firefox point of view it won't require any additional changes. Only portal have to do all the lifting.

I assume that by "webextension itself" you meant the native messaging host application (i.e. the native application outside of Firefox that an extension wishes to launch)? At some point, the host system needs to be able to discover which applications are able to talk with Firefox. The standard format for that is the native messaging manifest. It is up to the portal to pass a valid manifest to a confined Firefox, whether it is a real one, or a fake one that fits the required properties.

Since this discussion is only relevant to the portal and independent of Firefox, let's not comment on a closed bug in Firefox and continue the discussion after https://github.com/flatpak/xdg-desktop-portal/pull/705#issuecomment-2531614055

[Aglavic-c (aglavic-c)]

(In reply to Pulsebot from comment #53)
> Pushed by <email address hidden>:
> https://hg.mozilla.org/integration/autoland/rev/0533ca878434
> Integration with a new WebExtensions XDG desktop portal for native messaging
> on Linux r=emilio,robwu,rpl,andi,mach-reviewers
> https://hg.mozilla.org/integration/autoland/rev/e46ea28ef686
> Native Messaging WebExtension documentation r=robwu,rpl
> https://hg.mozilla.org/integration/autoland/rev/f1cfeb19c642
> Native Messaging WebExtensions portal tests r=robwu,ahal

Your patch has improved performance, thank you for all your work!

Perfherder has detected a mozperftest performance change from push [f1cfeb19c64250aa762b439736966d81a9a2ecbb](https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=f1cfeb19c64250aa762b439736966d81a9a2ecbb).

### Improvements:

| **Ratio** | **Test** | **Platform** | **Options** | **Absolute values (old vs new)** |
|--|--|--|--|--|
| [25%](https://treeherder.mozilla.org/perfherder/graphs?timerange=1209600&series=autoland,5137798,1,15) | browser_ml_summarizer_perf.js SUM-XENOVA-DISTILBART-CNN-12-6_MEDIUM-total-memory-usage | linux1804-64-shippable | | 765.54 -> 573.50 |
| [13%](https://treeherder.mozilla.org/perfherder/graphs?timerange=1209600&series=autoland,297636,1,15) | browser_ml_engine_perf.js EXAMPLE-cold-start-model-run-latency | linux1804-64-shippable | | 43.92 -> 38.00 |
| [12%](https://treeherder.mozilla.org/perfherder/graphs?timerange=1209600&series=autoland,283844,1,15) | browser_ml_engine_multi_perf.js suggest-pipeline-ready-latency | linux1804-64-shippable | | 24,217.38 -> 21,336.08 |
| [12%](https://treeherder.mozilla.org/perfherder/graphs?timerange=1209600&series=autoland,283845,1,15) | browser_ml_engine_multi_perf.js suggest-initialization-latency | linux1804-64-shippable | | 24,238.54 -> 21,357.83 |
| [11%](https://treeherder.mozilla.org/perfherder/graphs?timerange=1209600&series=autoland,297634,1,15) | browser_ml_engine_perf.js EXAMPLE-cold-start-pipeline-ready-latency | linux1804-64-shippable | | 8,340.92 -> 7,384.17 |
|...|...|...|...|...|
| [8%](https://treeherder.mozilla.org/perfherder/graphs?timerange=1209600&series=autoland,5138103,1,15) | browser_ml_suggest_feature_perf.js INTENT-initialization-latency | linux1804-64-shippable | | 13,501.83 -> 12,356.92 |

Details of the alert can be found in the [alert summary](https://treeherder.mozilla.org/perfherder/alerts?id=43028), including links to graphs and comparisons for each of the affected tests.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run these tests on try with `./mach try perf --alert 43028`

For more information on performance sheriffing please see our [FAQ](https://wiki.mozilla.org/TestEngineering/Performance/FAQ).