ibus in 19.10 misinterprets ~/.XCompose, somehow maps to the Delete key

[Steve Langasek]

After upgrade to 19.10, I found that my Delete key was not working as a Delete key, but that instead if I hit Delete twice it would print the character ☭.

Since others were not reporting this issue, I had a look around at my input config and remembered that I had ibus configured from a long time ago in order to support Chinese input.

If I disable ibus (either by unsetting the environment variables; or by killing ibus-daemon), then the Delete key works again as expected.

This is a regression in behavior since Ubuntu 19.04, where I had the same input setup on my desktop but the Delete key worked without problems.

I'm also not sure how to disable ibus, now that I am in this situation; or if ibus is expected to always be running.

The problem persists if I run ibus-setup and remove Chinese SunPinyin from the list of input methods, leaving only "English - English (US)".

[Gunnar Hjalmarsson (gunnarhj)]

Hi Steve,

I can't reproduce it here (with a Swedish keyboard). Some questions to start with:

* Which values do the environment variables have?

  env | grep -E 'XMOD|_IM'

* Are you possibly on Wayland?

* Can you reproduce it on a fresh 19.10 (or with a newly created user)?

Assuming that you are on standard Ubuntu, using ibus-setup for adding/removing input sources is not the 'right' way - it's supposed to be done in Settings -> Region & Language (probably not significant for this issue, but still...).

GNOME always starts IBus if it's installed (and it's installed by default on Ubuntu).

[Steve Langasek (vorlon)]

> * Which values do the environment variables have?

$ env | grep -E 'XMOD|_IM'
GTK_IM_MODULE=ibus
QT4_IM_MODULE=ibus
XMODIFIERS=@im=ibus
CLUTTER_IM_MODULE=ibus
QT_IM_MODULE=ibus
$

I don't know where these are being set, because Xsession is hell.

> * Are you possibly on Wayland?

No.

> * Can you reproduce it on a fresh 19.10 (or with a newly created user)?

No.

> Assuming that you are on standard Ubuntu, using ibus-setup for
> adding/removing input sources is not the 'right' way

That's not really a satisfactory position, given that the ibus-setup command is shipped in the ibus package, which is seeded as part of the Ubuntu Desktop. If the command should not be used, it should not be present.

> it's supposed to be done in Settings -> Region & Language (probably not
> significant for this issue, but still...).

I can't find anything under the settings screens that resembles the selection of input methods presented by ibus-setup. ibus-setup presents 'Chinese SunPinyin', Settings only presents 'Chinese'; and removing the Chinese input method from ibus-setup's config doesn't change what's displayed in Settings (Chinese is still listed here).

[Gunnar Hjalmarsson (gunnarhj)]

On 2019-10-23 08:09, Steve Langasek wrote:
>> * Which values do the environment variables have?
>
> $ env | grep -E 'XMOD|_IM'
> GTK_IM_MODULE=ibus
> QT4_IM_MODULE=ibus
> XMODIFIERS=@im=ibus
> CLUTTER_IM_MODULE=ibus
> QT_IM_MODULE=ibus
> $

Looks good.

> I don't know where these are being set, because Xsession is hell.

They are set by im-config on X-sessions on Debian/Ubuntu. (Without im-config, the most important ones would have been set by GNOME instead.)

>> * Can you reproduce it on a fresh 19.10 (or with a newly created
>> user)?
>
> No.

Then we know that the issue is caused by something in your $HOME. The question is what and how it ended up there...

There is a ~/.cache/ibus/compose folder with a suspicious cache file. Can you please rename that folder, relogin, and check if the issue is still present.

>> Assuming that you are on standard Ubuntu, using ibus-setup for
>> adding/removing input sources is not the 'right' way
>
> That's not really a satisfactory position, given that the ibus-setup
> command is shipped in the ibus package, which is seeded as part of
> the Ubuntu Desktop. If the command should not be used, it should not
> be present.

Fair point. But on non-GNOME desktops ibus-setup is needed, and made available to the users via an IBus icon. I wouldn't dare to say that you *never* need to use ibus-setup on a GNOME desktop, but GNOME provides an integration where XKB layouts and IBus IMs are presented together in Settings, so normally the users don't need to bother with ibus-setup.

>> it's supposed to be done in Settings -> Region & Language (probably
>> not significant for this issue, but still...).
>
> I can't find anything under the settings screens that resembles the
> selection of input methods presented by ibus-setup.

The "Input Sources" section in Region & Language is what I'm talking about. That interface controls the available input sources and saves the settings as

gsettings get org.gnome.desktop.input-sources sources

The input sources set via ibus-setup are saved as

gsettings get org.freedesktop.ibus.general preload-engines

but on GNOME the latter dconf setting is ignored AFAIK.

> ibus-setup presents 'Chinese SunPinyin', Settings only presents
> 'Chinese'; and removing the Chinese input method from ibus-setup's
> config doesn't change what's displayed in Settings (Chinese is still
> listed here).

For an IBus IM to be presented in Settings or ibus-setup, the package - in your case ibus-sunpinyin - needs to be installed. But when you say that only Chinese is listed in Settings, and assuming that ibus-sunpinyin is still installed on your machine, can it possibly be that if you click the "Chinese" option in Settings, you end up in a sub menu which includes Chinese SunPinyin?

[Steve Langasek (vorlon)]

> There is a ~/.cache/ibus/compose folder with a suspicious cache file.
> Can you please rename that folder, relogin, and check if the issue is
> still present.

What is suspicious about the file?

I moved the directory aside, started ibus-daemon up again, and the problem persists. The ~/.cache/ibus/compose/e6817ed7.cache file has been recreated with identical contents as previously.

Here are my gsettings values currently:

$ gsettings get org.gnome.desktop.input-sources sources
[('xkb', 'us+dvorak-alt-intl'), ('xkb', 'gr+polytonic'), ('xkb', 'cn'), ('xkb', 'ru')]
$ gsettings get org.freedesktop.ibus.general preload-engines
['xkb:us::eng']
$

The problem persists with these settings, as long as ibus-daemon is running.

> For an IBus IM to be presented in Settings or ibus-setup, the package -
> in your case ibus-sunpinyin - needs to be installed. But when you say
> that only Chinese is listed in Settings, and assuming that ibus-
> sunpinyin is still installed on your machine, can it possibly be that if
> you click the "Chinese" option in Settings, you end up in a sub menu
> which includes Chinese SunPinyin?

Settings -> Region & Language -> Input Sources presents Chinese as an option. It can be reordered in the list, but it is not clickable. It has 'view' (eyeball) and 'delete' (trashcan) icons. The 'view' icon pulls up a useless (for Chinese) keymap.

The ibus-sunpinyin package is installed.

[Steve Langasek (vorlon)]

Regardless, it's simply a hypothesis on my part that the fact that I have configured Chinese input methods in the past may be the cause of this problem. The Chinese keyboard setting is not active when this problem manifests.

[Gunnar Hjalmarsson (gunnarhj)]

Starting with the side topic (the easy part).

On 2019-10-23 18:50, Steve Langasek wrote:
> Settings -> Region & Language -> Input Sources presents Chinese as
> an option. It can be reordered in the list, but it is not clickable.
> It has 'view' (eyeball) and 'delete' (trashcan) icons. The 'view'
> icon pulls up a useless (for Chinese) keymap.

Ok, so you have previously added the Chinese keyboard layout. It's actually just an alias to the basic English (US) layout.

Please note that the list you see first only shows the options you have previously made available by adding them. To look for further sources to add you need to click the + button.

I installed ibus-sunpinyin, relogged in, and found the Chinese (SunPinyin) option in Settings among the "Other" input sources. Then I generated the zh_CN.UTF-8 locale, and found that the Chinese item had been converted to a sub menu, which includes Chinese (SunPinyin) together with the Chinese XKB layout.

That's how the GNOME design currently works. Not saying I'm too fond of it. In Unity (with code from GNOME 3.4 or something) the input sources were simply presented in one long list, and when you started to type the source you were looking for, it safely showed up.

I have filed an upstream issue about the most apparent shortcoming with the current design:

https://gitlab.gnome.org/GNOME/gnome-control-center/issues/82

It doesn't directly address the issue you stumbled upon, but it's closely related.

> What is suspicious about the file?
>
> I moved the directory aside, started ibus-daemon up again, and the
> problem persists. The ~/.cache/ibus/compose/e6817ed7.cache file has
> been recreated with identical contents as previously.

It was merely the name of the directory, and the string "IBusComposeTable" in the contents of the cache file which made me say that. After all your Delete key seems to do some kind of composing instead of what it's expected to do.

> The problem persists with these settings, as long as ibus-daemon is
> running.

It's ugly behavior, and all we know so far is that it seems to be caused by something in $HOME. But it may be due to some other cache file, some dconf setting, or whatever. I'm out of ideas for now as regards how to nail it.

Well, one thing you may want to try is to move away the ~/.config/dconf/user file and relogin so it gets recreated with only default values.

Also, do you possibly have an ~/.XCompose file which you have played with previously?

*Now* I'm out of ideas.

[Steve Langasek (vorlon)]

On Wed, Oct 23, 2019 at 07:24:03PM -0000, Gunnar Hjalmarsson wrote:
> Also, do you possibly have an ~/.XCompose file which you have played
> with previously?

Why yes, yes I do.

$ ls -l ~/.XCompose
-rw-r--r-- 1 vorlon vorlon 93 Mar 27 2009 /home/vorlon/.XCompose
$

Moving this file aside and restarting ibus-daemon, the Delete key works as
expected.

Contents of the file are:

include "/usr/share/X11/locale/en_US.UTF-8/Compose"
<\> <)> : "☭" # HAMMER AND SICKLE

So why is ibus interpreting this file differently in 19.10 than in previous releases?

[Steve Langasek (vorlon)]

downgrading the bug to 'medium' since most people don't have ~/.XCompose ;)

[Gunnar Hjalmarsson (gunnarhj)]

On 2019-10-24 00:56, Steve Langasek wrote:
> Contents of the file are:
>
> include "/usr/share/X11/locale/en_US.UTF-8/Compose"
> <\> <)> : "☭" # HAMMER AND SICKLE

Hmm.. Looking at <https://help.ubuntu.com/community/ComposeKey> and wondering if that line shouldn't rather read something like this:

<Multi_key> <backslash> <parenright> : "☭" # HAMMER AND SICKLE

(which works with IBus provided that there is a defined compose key)

Even if there was a regression due to the 1.5.19 -> 1.5.21 upgrade of ibus

* the changed behavior is upstream in nature

* a possible upstream issue would basically say: "Why does IBus no
  longer interpret my syntactically incorrect ~/.XCompose file in
  accordance with my intention?"

So I tend to think that this bug should be closed as invalid, unless you have some convincing arguments for keeping it open. Your call. :)

[Steve Langasek (vorlon)]

On Thu, Oct 24, 2019 at 12:06:38AM -0000, Gunnar Hjalmarsson wrote:
> On 2019-10-24 00:56, Steve Langasek wrote:
> > Contents of the file are:

> > include "/usr/share/X11/locale/en_US.UTF-8/Compose"
> > <\> <)> : "☭" # HAMMER AND SICKLE

> Hmm.. Looking at <https://help.ubuntu.com/community/ComposeKey> and
> wondering if that line shouldn't rather read something like this:

> <Multi_key> <backslash> <parenright> : "☭" # HAMMER AND SICKLE

> (which works with IBus provided that there is a defined compose key)

Sure, this is a valid alternative, and just replacing the <\> with
<backslash> is enough to unbreak my Delete key. (I don't care about whether
it causes the compose sequence to work, I clearly forgot I had ever even
configured this!)

> Even if there was a regression due to the 1.5.19 -> 1.5.21 upgrade of
> ibus

> * the changed behavior is upstream in nature

> * a possible upstream issue would basically say: "Why does IBus no
> longer interpret my syntactically incorrect ~/.XCompose file in
> accordance with my intention?"

IBus does not define the syntax of .XCompose, and this file is an interface.
It is not for ibus to retroactively declare the file to be "syntactically
incorrect"; and regardless, if it has decided that the file is syntactically
incorrect, this should not result in changes to the behavior of an unrelated
key.

> So I tend to think that this bug should be closed as invalid, unless you
> have some convincing arguments for keeping it open. Your call. :)

It is absolutely not invalid. I accept that it's not a high priority, but
it remains the case that ibus is interpreting ~/.XCompose in a way that is
inconsistent with how X itself does so.

[Gunnar Hjalmarsson (gunnarhj)]

On 2019-10-24 07:50, Steve Langasek wrote:
> On Thu, Oct 24, 2019 at 12:06:38AM -0000, Gunnar Hjalmarsson wrote:
>> wondering if that line shouldn't rather read something like this:
>>
>> <Multi_key> <backslash> <parenright> : "☭" # HAMMER AND SICKLE
>>
>> (which works with IBus provided that there is a defined compose key)
>
> Sure, this is a valid alternative, and just replacing the <\> with
> <backslash> is enough to unbreak my Delete key. (I don't care about
> whether it causes the compose sequence to work,

Please.. Whether it works in some case is reasonably relevant.

If I open gedit from terminal, I get these complaints:

(org.gnome.gedit:3821): Gtk-WARNING **: 14:11:14.241: Could not get code point of keysym \

(org.gnome.gedit:3821): Gtk-WARNING **: 14:11:14.242: Could not get code point of keysym )

(org.gnome.gedit:3821): Gtk-WARNING **: 14:11:14.242: compose file /home/gunnar/.XCompose does not include any keys besides keys in en-us compose file

So Gtk does not understand it, and - needless to say - it doesn't work.

If I replace <\> with <backslash> and <)> with <parenright>, Gtk stops complaining and it works, kind of. I.e. pressing \ followed by ) gives me ☭. But then I can't use my \ key for anything else, so this kind of composing is pretty useless without including <Multi_key> too.

My observations are true both with and without IBus. IBus provides the extra 'feature' with involving the Delete key, though.

> IBus does not define the syntax of .XCompose,

Right. Probably Gtk or libx11 does.

> It is not for ibus to retroactively declare the file to be
> "syntactically incorrect";

I suppose you don't know if it worked in 19.04. I guess it did not, and then the file has been incorrect for a long time.

> and regardless, if it has decided that the file is syntactically
> incorrect, this should not result in changes to the behavior of an
> unrelated key.

Ok.. Do you plan to submit an upstream issue?

[Gunnar Hjalmarsson (gunnarhj)]

Just stumbled upon this issue:

https://github.com/ibus/ibus/issues/2130

Common denominators are:

* Error in ~/.XCompose

* Upgrade to ibus 1.5.21

[Gunnar Hjalmarsson (gunnarhj)]

Plus, of course: Delete key affected.

[nshepperd (nshepperd)]

I submitted a patch to the upstream repository that fixes the invalid parsing in ibus: https://github.com/ibus/ibus/pull/2227

[Launchpad Janitor (janitor)]

This bug was fixed in the package ibus - 1.5.23-0ubuntu1

---------------
ibus (1.5.23-0ubuntu1) groovy; urgency=medium

  * New upstream release (LP: #1835541, LP: #1849399)
  * d/p/0003-dconf-Use-dbus-run-session-to-set-up-dconf-overrides.patch:
    - Refreshed
  * Dropped patches, applied upstream:
    - d/p/bashism-installed-tests.patch
    - d/p/remove-glib-check-version.patch
    - d/p/use-wayland-display-for-socket-name.patch

 -- Gunnar Hjalmarsson <email address hidden> Wed, 30 Sep 2020 03:21:00 +0200