Onboard doesn't work in Wayland

Bug #1672465 reported by Jeremy Bícha
36
This bug affects 7 people
Affects Status Importance Assigned to Milestone
Onboard
Won't Fix
Undecided
Unassigned
OnboardOSK
Confirmed
High
Unassigned
onboard (Ubuntu)
Confirmed
High
Unassigned

Bug Description

Onboard 1.4.1 on Ubuntu GNOME 17.04

Onboard at least shows the keyboard now in GNOME on Wayland. But it doesn't send key presses to gedit or other native Wayland apps.

Tags: wayland
Revision history for this message
marmuta (marmuta) wrote :

Yes, that's still an open issue. The last I wrote about it was here:
https://bugs.launchpad.net/onboard/+bug/1366421/comments/22

We probably need a different strategy too, though, and add support for wl_text_input.

Changed in onboard:
status: New → Confirmed
Revision history for this message
Jeremy Bícha (jbicha) wrote :

Have you tried talking to GNOME Shell developers directly about this or other Onboard integration issues?

Revision history for this message
Jeremy Bícha (jbicha) wrote :

There is some interest from Canonical in making GNOME on Wayland the default for 18.04 LTS but there will need to more investigation of what won't work right if that happens.

Revision history for this message
marmuta (marmuta) wrote :

Yes, thanks for the reminder, we heard the news and already shifted focus to GNOME Shell. I think we can be ready in time for 18.04 LTS, I'm actually cautiously aiming for 17.10.

There's a local branch that I had worked on before. The idea is to embed directly into GNOME Shell via extension and a Clutter frontend.

JavaScript shell extension -> GObject introspection shared library + typelib -> custom
> ClutterActor -> Python calls into Onboard.

Most of the pieces are working in a proof of concept kind of way. The Clutter part still needs a lot of work, though.

gdk_wayland_window_set_accept_focus won't be needed then anymore. It was never enough to get into Activities and password views anyway. The Clutter approach would solve that too.

Also we want to switch to Wayland interfaces for text input. zwp_text_input_unstable_v2 might be the one, but I haven't looked into it much beyond Weston samples.

Jeremy Bícha (jbicha)
tags: added: wayland
Changed in onboard (Ubuntu):
importance: Undecided → High
Jeremy Bícha (jbicha)
Changed in onboard (Ubuntu):
status: New → Confirmed
Changed in onboard:
status: Confirmed → In Progress
Revision history for this message
Sebastien Bacher (seb128) wrote :

Could somebody from the onboard team gives a status update there of how onboard work with gnome-shell and under wayland?

Did anyone try to replace caribou with onboard in gnome-shell, is that doable/easy? How do they compare usability/feature wise?

With the switch from Unity to GNOME we need to have a look at what osk we can/should use, the onboard team has been great over the years and it would be nice if we could keep working together. That discussion might be a bit off topic for this bug report though so let me know if you would prefer to move it to another report, a mailing list, direct email, ...

Revision history for this message
Francesco Fumanti (frafu) wrote :

Hi Sebastien,

First of all, thanks for you appreciating our team and we also hope to continue the cooperation with you.

As Onboard has more features than Caribou, there already has been some demand for its integration into GNOME and people have also done some work on it; for example:
https://stackoverflow.com/questions/33065735/replacing-gnomes-virtual-keyboard-with-onboard-hide-show-via-dbus-doesnt-wor

In the meantime, Onboard is shipping an own extension for its integration into GNOME shell.
https://launchpad.net/ubuntu/xenial/amd64/gnome-shell-extension-onboard
https://bugs.launchpad.net/onboard/+bug/1538627

The release for zesty does not fully support GNOME shell yet, but since Ubuntu has decided to switch to GNOME Shell and Wayland, these two items have become high priority for us and marmuta is currently actively working on the port. He told me that he made some progress, but he is better placed than me to give you details on it, so I leave it up to him.

As far as I know, Lightdm is also being replaced by GDM, which will also be relevant for Onboard...

Thanks again for contacting us.

Revision history for this message
marmuta (marmuta) wrote :

> Did anyone try to replace caribou with onboard in gnome-shell, is that doable/easy?
I did, and came to the conclusion that fully replacing caribou with the existing Onboard is close to impossible. That's mainly because applications don't have access to the activities and authentication views. Can't search applications or enter passwords with Onboard. In Wayland there was the additional problem of gdk_wayland_window_set_accept_focus not being implemented and Onboard uselessly sending key-strokes to itself.

There is an alternative approach, though, which I tried to outline in #4. The idea was to embed Onboard directly into gnome-shell via shell extension and custom clutter actors.
The good news is, that yes, it works. See attached screen-shot with the new Onboard in the Activities view. It handles similar to the stand-alone application, can be docked to screen-edge or left floating, resized and moved, etc. Focusing is not an issue and it can be shown anywhere at any time.

There's plenty of work left to do, though. My goal has been to get something usable in X and Wayland before Artful feature freeze. It is already starting to get there in X, but problems remain in Wayland.

Wayland's input separation makes it hard to find replacements for anything that generates input events, key-strokes, clicks or wants to receive global clicks and key-presses. Accessibility features will probably take a hit. Mousetweaks isn't getting ported to Wayland for the same reasons, so no hover-click support for now. It's unclear if we will find replacements for the click mapping buttons (right-, middle-, double-, drag-click).
Write access to /deb/uinput would help a lot in that regard, but this undermines Wayland's security.

Other than that, we'll probably get most of the old Onboard's features ported, including the new emoji palette, which hadn't been released yet.

Enabling Onboard in gnome-shell will mean having the Onboard_Indicator extension installed and enabled. If disaster strikes, all that's needed to return to the shell's built-in keyboard is to disable the extension.

Revision history for this message
Jeremy Bícha (jbicha) wrote :

Personally, I would vote for Onboard to be integrated into GNOME upstream instead of Caribou.

Caribou is very deficient compared to Onboard and has not really been actively developed much. (There was a GSOC proposal to do some things but I am guessing the student could probably end up working just as easily on Onboard in GNOME instead.)

Two prominent examples: I believe Caribou only supports a very small number of hard-coded keyboard layouts. Caribou is a fairly simple onscreen keyboard that does not have as many accessibility features as Onboard.

So Caribou can work if you just want a simple English onscreen keyboard, but if you need a different keyboard layout or more accessibility options, it's not very helpful at all.

Therefore, I recommend that the Onboard team open a discussion with the GNOME Shell developers about using Onboard's core to replace Caribou. (That could help with the Wayland difficulties since the necessary integration could be built directly into GNOME Shell.)

Revision history for this message
marmuta (marmuta) wrote :

Let me give you an update on the current state of Onboard in gnome-shell.

First, things changed - a lot. The work is happening in a new independent project OnboardOSK at
https://launchpad.net/onboardosk
This is basically a fork of Onboard, written mainly in C++. There's no Python involved apart from supporting tools.

At the project's core is the (largely) toolkit-independent shared library libonboardosk.so. It contains an onscreen keyboard closely resembling the original Onboard in appearance and, well eventually, features.

A second shared library libonboardoskgnomeshell.so implements the toolkit-dependent parts, here by integrating with Clutter and GObject introspection.

A new GNOME Shell extension then embeds OnboardOSK into the shell and provides the shell's services to the shared libraries below.

The end result is an onscreen keyboard that is already quite indistinguishable from the old standalone Onboard, but, and that's a big but, it's not done and realistically won't be before Artful feature freeze.

So what works, what's missing? The majority of the old Onboard's features are ported by now, e.g., key logic, word suggestions, emoji palette, themes, color schemes, etc. are there. "window" (actor) handling has become more complete and smoother. Hover-click does something even without mousetweaks (for Wayland), but is limited to the keyboard itself.

The main missing things are
- long-press popups (in progress currently)
- click-mapping buttons (right-, middle-, double-, drag-click)
- modifier+click for accessibility
- keyboard scanning
- a bunch of dialogs: snippets, language model error recovery

Most worryingly, however, Wayland support is still lagging behind X. Developing in Wayland is a bit of a pain because there is no easy way to restart GNOME Shell (that I'm aware of). Reloading extensions is insufficient as this won't reload shared libraries. Therefore my strategy has been to get as much as possible done outside of GNOME Shell in Wayland, replacing with GNOME Shell in X and test_stage.js in Wayland and switch back later. Well, that's going to be soon, but the time is up.

So, apologies, I tried to make it happen for 17.10, but it'll take more time. We'll very likely make the 18.10 release, however, as default or not.

FWIW, this message was typed with with OnboardOSK in GNOME Shell (X). What is there seems to work alright. No crashes lately, no noticeable leaking (unlike Python Onboard).

marmuta (marmuta)
Changed in onboardosk:
status: New → Confirmed
importance: Undecided → High
Changed in onboard:
status: In Progress → Won't Fix
Revision history for this message
marmuta (marmuta) wrote :

> We'll very likely make the 18.10 release
I meant to write:
We'll very likely make the 18.04 release

Revision history for this message
Joe Linux (gbodley) wrote :

I don't know if I'm running Wayland, but it is the most recent version of Kubuntu as it comes out of the box. Unfortunately Onboard crashes the entire desktop. If you try to type one Onboard french character, the desktop turns black and you eventually are brought to the login screen and have to login with a password. I hope this can be fixed as Onboard is very important to me.

Revision history for this message
A Raghuram (araghuramindia) wrote :

Onboard doesn't send key presses to gedit or any other desired window and instead closes down. This happens only on Wayland windowing system. It works fine in X11. I am working on Ubuntu 20.04.3 Gnome 3.36.8. I have been using Onboard since Ubuntu 12.04 stable release. In release 18.04, I had disabled Wayland, as it had other issues too. However, in 20.04, Wayland is working reasonably well, except for supporting Onboard

Onboard is extremely useful to me and I don't want to give up Wayland too, since it loads faster than X11. I want Onboard to work on Wayland too, so that users can get benefits of both Onboard and Wayland. I request some one to work on resolving this.

Revision history for this message
sudodus (nio-wiklund) wrote :

@ Jeremy Bicha,

Is "Won't fix" for Onboard in Wayland still the policy, or is there some development?

What about the current (in Ubuntu 22.04.1 LTS) screen keyboard that we get via gnome-control-center? Is it still based on Caribou? Are there features from Onboard in it? (It looks like Caribou to me.)

Revision history for this message
Jeremy Bícha (jbicha) wrote :

I wouldn't say it's "policy" but it is the current situation.

The GNOME screen keyboard does not use Caribou. It is built into GNOME Shell. It has far less features than Onboard. The GNOME implementation is currently more about basic keyboard access on a touch-only device instead of as a full-featured accessibility aid.

Revision history for this message
Evan (eschox) wrote :

I created an account just to squeak a little at this wheel.

Whatever is being used in Ubuntu 22 is far inferior to Onboard.

X11 is being deprecated.

I imagine that devs knew that Wayland would replace X11 in Ubuntu LTS

Time to decide what you will all do.

in 4 years Will Onboard be abandoned or will it support wayland?

this took8 minutes to type in ubuntu 22 lts keyboard

Revision history for this message
A Raghuram (araghuramindia) wrote :

The situation is the same as mentioned in my post of 25Aug21. Onboard OSK works in X11 but not in Wayland. This time, though, the Onboard window doesn't close down in Wayland.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.