Keyboard layout doesn't switch to japanese when using a Japanese (109-key) keyboard and the system language is english

[Marcel Montes]

Disclaimer:
This is my first ever ubuntu bug report... and this isn't a mozc bug per-se, but more like an overall integration glitch. But this is the closest place I could think of.

Change the bug report package accordingly, please.

Steps to reproduce:
1) Install Ubuntu 20.04.
  1.1) Select English as the system language.
  1.2) Select "Japanese 109-key" as the keyboard.
  1.3) Select "English (US, intl, dead keys)" as the keyboard layout.
2) On the newly installed system, run "sudo apt install ibus-mozc"
3) Add Japanese in "Region & Language" with "Japanese (Mozc)" as input source.

Problem:
super+space changes the systray icon, but the keyboard layout isn't switched.

Another noticeable error is that when using "Show Keyboard Layout", the layout name is displayed as "?" in the window title. (see attached img)

fix:
Edit /usr/share/ibus/component/mozc.xml and change "<layout>default</layout>" to "<layout>jp</layout>" as stated in https://help.ubuntu.com/community/JapaneseInput

Comments:
I don't know what mozc "default" layout means. does it get the layout from /etc/default/keyboard?
If it does, it's set to XKBLAYOUT=us and XKBVARIANT=intl (as per installation instructions) so no wonder mozc gets confused.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: ibus-mozc 2.23.2815.102+dfsg-8ubuntu1 [modified: usr/share/ibus/component/mozc.xml]
ProcVersionSignature: Ubuntu 5.13.0-27.29~20.04.1-generic 5.13.19
Uname: Linux 5.13.0-27-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu27.21
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
Date: Thu Jan 20 14:50:45 2022
InstallationDate: Installed on 2022-01-19 (0 days ago)
InstallationMedia: Ubuntu 20.04.3 LTS "Focal Fossa" - Release amd64 (20210819)
SourcePackage: mozc
UpgradeStatus: No upgrade log present (probably fresh install)

[Gunnar Hjalmarsson (gunnarhj)]

Thanks for your report. First a question:

How do you set "Japanese 109-key"? I don't see such a keyboard model option.

But leaving that aside, I suspect that you also want to install the mozc-utils-gui package. Because only switching to Mozc isn't sufficient. You also need to select some other "Input Mode" but "Direct input", for instance Hiragana, and for that I think that the just mentioned package provides the necessary tools.

Mozc works together with many layouts which provide the latin a-z letters, including us+intl, so I don't think Mozc was confused by that layout. ;)

[Marcel Montes (spiceman)]

My bad, it's just "Japanese" (I didn't run the installation again just to check the exact hardcoded text for the bug report).

My point was that the physical actual keyboard is a japanese one with 109 keys (the usual in japanese keyboards).
The japanese layout has some keys that just plainly don't exist in other layouts (全角/半角, 変換, 無変換,カタカナ・ひらがな, a \ to the left of the small right shift, and a yen sign to the left of its small backspace).

Sorry for the confusion.

What I'm saying is that the layout in the OS doesn't change at all.
Note how I never said "I can't type in Japanese" in the bug report. I could if I -like you said- changed input mode to Hiragana in mozc. But that's not how you do it with a japanese keyboard.

With a japanese keyboard (and correct layout) you just press the 全角/半角 (zenkaku/hankaku) key which changes input mode between direct input and the currently/last used kana mode (usually hiragana, but might be katakana), no need to use menues.

Even if Input Mode was in Direct Input the layout should be in Japanese regardless of being unable to type in japanese characters.

eg: If I press the key to the right of the P it should type @ like in the Japanese keyboard, not a [, if I press shift-7 it should be a single quote not an ampersand, etc.

Like I said, I don't know if it's even a mozc problem, and I fixed it by explicitly setting the layout to "jp" in ibus/component/mozc.xml which maybe points to ibus, rather than mozc itself.

One of the first things I said is that mozc probably wasn't the right place for this bug report.

[Marcel Montes (spiceman)]

Oh, and also please check the image I attached.

The keyboard layout program showing "?" instead of the layout name is a clear hint that something is not right.

[Marcel Montes (spiceman)]

Also, for what is worth, I ran "ubuntu-bug ibus-mozc", I don't know why the bug report got filed into this package instead.

[Gunnar Hjalmarsson (gunnarhj)]

On 2022-01-21 14:14, Marcel Montes wrote:
> My bad, it's just "Japanese"

Ok.. In that case, can you please run this command:

gsettings get org.gnome.desktop.input-sources sources

and let us know what it outputs. It may make it easier for me to comment further with that info.

[Marcel Montes (spiceman)]

$ gsettings get org.gnome.desktop.input-sources sources
[('ibus', 'mozc-jp'), ('xkb', 'us+intl')]

I should add that the listed input sources are the same regardless of what I have in <layout> at /usr/share/ibus/component/mozc.xml.

With "jp" in layout it works properly but with "default" it doesn't.

[Marcel Montes (spiceman)]

I uploaded a video demonstrating what happens

https://www.youtube.com/watch?v=W_r8nclOKCI

[Gunnar Hjalmarsson (gunnarhj)]

Thanks! I think I understand now.

"default" means that it keeps the active layout you used right before switching to "Japanese (Mozc)". So since the only XKB layout you have added is "English (US, intl., with dead keys)", that's the layout used by Mozc.

I'm pretty sure that the Mozc developers don't consider it a bug. They have deliberately made that choice instead of always enforcing the "jp" layout. Maybe because not everyone who needs to input Japanese has a Japanese keyboard.

Replacing "default" with "jp" in mozc.xml seems to be the most proper setup for your case.

On 2022-01-21 14:20, Marcel Montes wrote:
> The keyboard layout program showing "?" instead of the layout name is
> a clear hint that something is not right.

That's indeed a bug. But it's a separate thing, and not related to the other observations you made. Despite the "?" instead of the correct label, it still shows the us+intl layout in your case.

I see the same in 20.04, but not in 21.10, so that bug is not present in the latest Ubuntu version. (And it's not important enough to investigate and backport a fix to 20.04.)

On 2022-01-21 17:22, Marcel Montes wrote:
> Also, for what is worth, I ran "ubuntu-bug ibus-mozc", I don't know
> why the bug report got filed into this package instead.

You did the right thing. ibus-mozc is the applicable binary package, but bugs are sorted by source package, and mozc is the source package which builds ibus-mozc (and a bunch of other binary packages).

So, to summarize: I'd say that the principal behavior you reported is not a bug. The "?" thing, OTOH, is a bug, but it has been fixed in latest Ubuntu.

Please let me know whether my explanation is sufficient. If you don't object, I'm inclined to close this bug report.

[Marcel Montes (spiceman)]

In my opinion, this is still a bug.

For arguments' sake, let's say I also have another Japanese input source with Japanese keyboard layout (no mozc).

ie:
$ gsettings get org.gnome.desktop.input-sources sources
[('ibus', 'mozc-jp'), ('xkb', 'us'), ('xkb', 'jp')]

Then, changing the input source to xkb jp, changes the layout. (ie: shift-2 is properly a double quote, not like an at mark like in the us layout, etc).

---> This is the expected behaviour of switching keyboard layouts. <---

To summarise:
If I switch to "Japanese", it switches to a japanese keyboard layout.
If I switch to "Japanese (Mozc)" it just activates mozc with the us layout intl variant intact. Where did the "japanese" part go?

It's effectively a -non existant- "US-intl (Mozc)" keyboard layout, not "Japanese (Mozc)" as advertised.

I don't understand the rationale behind the "Japanese" (no mozc) keyboard layout switching to jp (ie: actually doing setxkbmap jp) but "Japanese (Mozc)" not doing it.

[Marcel Montes (spiceman)]

To test, I also added a spanish keyboard layout.

$ gsettings get org.gnome.desktop.input-sources sources
[('ibus', 'mozc-jp'), ('xkb', 'us+intl'), ('xkb', 'jp'), ('xkb', 'es')]

If I'm using the spanish layout and switch to "Japanese (Mozc)" I end up with a -non existent- "es (mozc)" setup.
If I'm using the us international and switch to "Japanese (Mozc)" I end up with a -non existent- "us+intl (mozc)" setup.
If I'm using the japanese (no mozc) layout and switch to "Japanese (Mozc)" I end up with the proper "Japanese (Mozc)" (by chance).

This is totally inconsistent and is not how switching keyboard layouts should behave, in my opinion.

[Marcel Montes (spiceman)]

Tutorials, Videos in youtube, blogs, everyone mentions to explictly change input mode to "Hiragana" through the menues.

Because their layout is broken!!!

In a japanese setup the key to the left of "1" (tilde "~" in the US keyboard layout) IS the input mode switcher.
Since ibus-mozc is *not* properly configured to ensure that that works and behaves randomly depending on the currently used keyboard layout, everyone has to work around it with the menues.

This is broken behaviour.

[Marcel Montes (spiceman)]

Let me rephrase this. This is not a bug.

This is a misconfiguration in the ubuntu distribution that leads to inconsistencies in switching keyboard layouts when mozc is involved, not a mozc bug.

Mozc just does what it's been told to.

The main point is:
"I don't understand the rationale behind the "Japanese" (no mozc) keyboard layout switching to jp (ie: actually doing setxkbmap jp) but "Japanese (Mozc)" not doing it."

[Gunnar Hjalmarsson (gunnarhj)]

The "default" choice is an upstream one.

https://github.com/google/mozc/blob/master/src/unix/ibus/gen_mozc_xml.py#L227

The packaging for Debian/Ubuntu happens at Debian, and Ubuntu builds the thing using the same source as Debian.

So if you want to suggest a change, you should probably file an upstream issue:

https://github.com/google/mozc/issues

Or possibly a Debian bug:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=ibus-mozc

But before you do, please understand that there is no perfect default configuration which is right for all users. As regards Ubuntu the current configuration works fine in some typical cases:

* If you do a Japanese install of Ubuntu, the jp layout is added together with Japanese (Mozc). My impression is that Japanese users often use the jp layout for typing English.

* If you do an English install of Ubuntu (and don't have a Japanese keyboard), you probably want the layout you are used to for inputting Japanese.

In the latest ibus-mozc version, you should not edit mozc.xml to change it, but there is a user specific configuration file:

$ cat ~/.config/mozc/ibus_config.textproto
engines {
  name : "mozc-jp"
  longname : "Mozc"
  layout : "default"
}

So.. If you bring your suggestion to upstream or Debian, please post the URL here for tracking purposes. The change you suggest is of such a nature that we won't do it in Ubuntu only.

[Marcel Montes (spiceman)]

Thanks, I will take it to debian.

Yes, I understand that no configuration accommodates all users but switching to a japanese keyboard layout not doing setxkbmap jp is inherently broken in my opinion.

There should be ways to accommodate running mozc along other keyboard layouts explicitly should the user want to do so, rather than this inconsistent setup that exists right now, in my opinion.

Thanks for pointing out how to configure ibus for the user only.

[Marcel Montes (spiceman)]

Reverting /usr/share/ibus/component/mozc.xml to default and adding the following configuration didn't work in Ubuntu 20.04.

$ cat ~/.config/mozc/ibus_config.textproto
engines {
  name : "mozc-jp"
  longname : "Mozc"
  layout : "jp"
}

$ apt show ibus-mozc
Package: ibus-mozc
Version: 2.23.2815.102+dfsg-8ubuntu1

[Marcel Montes (spiceman)]

The debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1004329

[Gunnar Hjalmarsson (gunnarhj)]

Right, that's in the *latest* ibus-mozc version. Works in 21.10, but not in 20.04.

Sorry if I wasn't clear enough.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.