18.04.14.7 regression: no alt_shift_toggle in XKBOPTIONS

[Alkis Georgopoulos]

Up to ubuntu-mate-18.04.1.iso, everything was fine. Starting with 18.04.2, XKB_OPTIONS does not contain "alt_shift_toggle" anymore and we cannot switch the keyboard layout to e.g. Greek using Alt+Shift.

Reading the changelog, I see:

$ ~/source/ubiquity$ git show 786a5325ef
+console-setup (1.178ubuntu6) cosmic; urgency=medium
+
+ * keyboard-configuration.{config,templates}: There is no good default for
+ layout toggling, stop pretending there is. Console users can set one
+ with dpkg-reconfigure or editing /etc/defaults/keyboard (LP: #1762952)

I'm guessing that ubiquity duplicates some code from console-setup, and LP: #1762952 caused this regression.

To reproduce:
1) Start ubuntu-mate-18.04.1-desktop-amd64.iso
2) Select Ελληνικά (Greek) and start installing Ubuntu using the default options
3) Right after the keyboard layout step, run:
$ grep XKBOPTIONS /etc/default/keyboard
XKBOPTIONS="grp:alt_shift_toggle,grp_led:scroll"
4) Verify that you can switch to Greek with Alt+Shift

Starting from ubuntu-mate-18.04.2-desktop-amd64.iso (.1=OK, .2=BAD), we cannot switch to Greek using Alt+Shift anymore:
$ grep XKBOPTIONS /etc/default/keyboard
XKBOPTIONS="grp_led:scroll"

Does ubiquity really expect the users to run `dpkg-reconfigure console-setup`?

Note that selecting Greek in the syslinux menu produces the correct XKBOPTIONS, yet ubiquity overwrites them later on with the wrong ones.

[Gunnar Hjalmarsson (gunnarhj)]

I was undoubtedly involved in the fix of bug #1762952.

Why is there a need to switch keyboard layout during the installation?

As regards post-install, i.e. at first boot/login, the desktop environment ought to provide some default shortcut for switching input sources.

[Alkis Georgopoulos (alkisg)]

Hi Gunnar, thank you for your input,

> Why is there a need to switch keyboard layout during the installation?

One example is to set the user name, e.g. Διαχειριστής (Administrator).
Another is to be able to surf etc while ubiquity is running in the background.

> As regards post-install, i.e. at first boot/login, the desktop environment ought to provide some default shortcut for switching input sources.

By with package? Xorg by itself supports Alt+Shift, and this worked fine up to 18.04.1, while it broke in 18.04.2, is "plain xkb" now unsupported? By default, MATE doesn't ship ibus or fcitx, isn't that OK? I haven't tested in other flavors yet..

`setxkbmap -query` shows the wrong settings now; if one runs `dpkg-reconfigure keyboard-configuration` after installation, then it works fine again.

[Gunnar Hjalmarsson (gunnarhj)]

On 2020-08-18 15:35, Alkis Georgopoulos wrote:
>> Why is there a need to switch keyboard layout during the
>> installation?
>
> One example is to set the user name, e.g. Διαχειριστής
> (Administrator).

That makes me think of bug #1875062.

> Another is to be able to surf etc while ubiquity is running in the
> background.

Ok..

>> As regards post-install, i.e. at first boot/login, the desktop
>> environment ought to provide some default shortcut for switching
>> input sources.
>
> By with package? Xorg by itself supports Alt+Shift, and this worked
> fine up to 18.04.1, while it broke in 18.04.2, is "plain xkb" now
> unsupported?

No, XKB is not unsupported.

This topic is a can of worms. If you read the story at bug #1762952, you'll see that the previous behavior, where console-setup aggressively set Alt+Shift in /etc/default/keyboard for all users, caused confusion and issues for users with the GNOME desktop. My proposal, to stop it from doing so only at package upgrades, was first accepted but then reversed due to some ubiquity regression.

Adam Conrad's conclusion was to completely stop console-setup from automatically setting Alt+Shift.

I can't tell which solution would be best to fit all the desktop environments.

> By default, MATE doesn't ship ibus or fcitx, isn't that OK? I haven't
> tested in other flavors yet..

I can't imagine that IBus or Fcitx would be needed to handle this. If MATE depends on XKB for input source switching, there must be other ways to do it besides letting console-setup set Alt+Shift by default for all users.

On Ubuntu there is a dconf setting:

gsettings get org.gnome.desktop.input-sources xkb-options

Probably there is some equivalent dconf key on MATE.

> `setxkbmap -query` shows the wrong settings now; if one runs `dpkg-
> reconfigure keyboard-configuration` after installation, then it
> works fine again.

How do you define "right" and "wrong" in this context?

[Alkis Georgopoulos (alkisg)]

> How do you define "right" and "wrong" in this context?

After selecting "Greece/Greek" in an operating system installer, users should be able to switch between English and Greek in the console, and IF they run a GUI, in the display manager (to type the password), AND in the desktop environment after login.

As far as I can remember, this was solved centrally in console-setup, up to 18.04.2, when layout switching stopped working.

These 3 use cases (console, login screen, display environment) are the essential ones, the "system defaults".
Now additionally, there's another need: in multi-user and multi-cultural environments, per-user settings might be needed.

IF there's need for that, then sysadmins should select a display manager and a desktop environment that supports it. So "most" of the users will still be using the defaults, while "some" of the users will have to go to the desktop environment settings and choose a layout different to the default.
But having the desktop environment support different layouts per user, should in no way cancel the defaults; it should not prohibit the console users or the all-users-have-the-same-language environments from typing Greek, like it does now.

What I'm proposing:
If I run `dpkg-reconfigure keyboard-configuration`, I'm presented with some questions/options.
Most installers like ubiquity or debian-installer ask the same questions/options.

So, /etc/default/keyboard ***SHOULD BE THE SAME IN BOTH CASES***.

I believe that's they key part, if we can agree on that we can the propose solutions. After 18.04.2, ubiquity generates a different /etc/default/keyboard than `dpkg-reconfigure keyboard-configuration`, and I think this is a bug in ubiquity.

[Alkis Georgopoulos (alkisg)]

Regarding the "which shortcut should be used by default for toggling layouts".

Ubuntu or Gnome should not decide that shortcut worldwide. It depends on the locale. In Greece, we've been using Alt+Shift since at least the '80s. Windows recently started offering Win+Space as well, but in parallel, without cancelling Alt+Shift, and very few people use Win+Space here.

AFAIK console-setup already has code to provide the correct defaults for most locales. If, for some locales, there's no sensible default, then ubiquity should ASK the user, not default to "none". None does not make sense; of course a Greek user will need to type Greek. Console-setup already has such a dialog, if ubiquity doesn't want to use a default shortcut after 18.04.2, it should at least present the layout toggling switch dialog, NOT default to "none".

[Gunnar Hjalmarsson (gunnarhj)]

I can't recall that Ubiquity asks about shortcut for switching input sources. Yes, it would make sense to me that such a question is asked for those desktop environments which need it.

Please take into consideration that standard Ubuntu does not need it. The previous default Alt+Shift caused confusion and issues. GNOME (and Unity) rely on a non-XKB mechanism by default.

[Alkis Georgopoulos (alkisg)]

> I can't recall that Ubiquity asks about shortcut for switching input sources.

I believe it relied on console-setup for the correct default.
So, console-setup shouldn't stop providing a default until AFTER ubiquity had a question about "how to toggle the layout".

> Please take into consideration that standard Ubuntu does not need it. The previous default Alt+Shift caused confusion and issues.

I just tested on Ubuntu GNOME and I can't switch the layout with Alt+Shift. I can switch it with Win+Space, which isn't appropriate for Greece. Also, I can't type Greek in neither the console nor in GDM.

Gunnar, what are you proposing?

I believe that LP: #1762952 should be reverted, and if GNOME chokes while reading /etc/default/keyboard, then we can work on fixing GNOME.

Are you proposing that we should file bugs against console-setup, all display managers, and all other desktop environments, to assume a default toggling switch, because we can't fix GNOME?

In which direction should I be trying to resolve this?

[Gunnar Hjalmarsson (gunnarhj)]

If you really read the comments at bug #1762952, you see that the initially proposed change was limited in scope. However, it proved to come with various regressions, so it landed in a solution where the default shortcut for switching keyboard layout was changed in console-setup from "Alt+Shift" to "No toggling".

But it did address incompatibility issues on Ubuntu/GNOME, so reverting it to the old behavior would be bad.

Possibly console-setup could be modified again, so the default shortcut depends on XDG_CURRENT_DESKTOP ("No toggling" for GNOME and "Alt+Shift" for others). That way there wouldn't be a need to involve ubiquity or other desktops.

@Adam: Do you think that would make sense?

On 2020-08-20 11:10, Alkis Georgopoulos wrote:
> I just tested on Ubuntu GNOME and I can't switch the layout with
> Alt+Shift.

Right. Super+Space is default on GNOME.

> I can switch it with Win+Space, which isn't appropriate for
> Greece.

That's reasonably about personal preferences rather than geographical. Many users are used to something else but Super+Space, and on GNOME they can either change it to something else or add some XKB based shortcut via Tweaks.

> Also, I can't type Greek in neither the console nor in GDM.

Super+Space works for me in GDM. A prerequisite is that /etc/default/keyboard includes more than one XKB layouts.

As regards the console, is there a use case which would be worth to take into consideration where there is a need to switch to a non-latin keyboard layout?

[Alkis Georgopoulos (alkisg)]

Hi Gunnar, thank you for the feedback,

> Possibly console-setup could be modified again, so the default shortcut depends on XDG_CURRENT_DESKTOP ("No toggling" for GNOME and "Alt+Shift" for others). That way there wouldn't be a need to involve ubiquity or other desktops.

When launching a console, XDG_CURRENT_DESKTOP isn't guaranteed to be set. E.g. one can just switch to vt2 and login there, or one can run `sudo -i` which doesn't preserve the XDG_* environment variables. Losing the ability to type Greek because I used `sudo -i` wouldn't be appropriate.

> Right. Super+Space is default on GNOME.

Up to now, /etc/default/keyboard allowed for defining a shortcut GLOBALLY, and the console, the display managers, and all the desktop environments respected that. Is supporting "a global shortcut" no longer a Debian/Ubuntu goal?

> That's reasonably about personal preferences rather than geographical.

Well, the official Greek school books teach Alt+Shift for layout toggling, so I'm not sure it can be called a "personal preference" and not a geographical one. That said, I wouldn't mind if Linux decided to globally endorse Win+Space, but AFAIK console doesn't support Win+Space as an option yet.

> As regards the console, is there a use case which would be worth to take into consideration where there is a need to switch to a non-latin keyboard layout?

Normal PC usage is impossible without it. Some examples:

# Change directory to Desktop
cd "Επιφάνεια εργασίας"

# Edit a file
nano Αρχείο.txt

# Set a user's name
usermod -c Άλκης alkisg

[Gunnar Hjalmarsson (gunnarhj)]

Hi again,

On 2020-08-27 08:25, Alkis Georgopoulos wrote:
>> Possibly console-setup could be modified again, so the default
>> shortcut depends on XDG_CURRENT_DESKTOP ("No toggling" for GNOME
>> and "Alt+Shift" for others). That way there wouldn't be a need to
>> involve ubiquity or other desktops.
>
> When launching a console, XDG_CURRENT_DESKTOP isn't guaranteed to be
> set. E.g. one can just switch to vt2 and login there, or one can run
> `sudo -i` which doesn't preserve the XDG_* environment variables.
> Losing the ability to type Greek because I used `sudo -i` wouldn't
> be appropriate.

The configuration typically does not happen when you launch a console; with that approach /etc/default/keyboard would be already changed on non-GNOME machines. So I don't think the absence of XDG_CURRENT_DESKTOP in those situations matters.

>> Right. Super+Space is default on GNOME.
>
> Up to now, /etc/default/keyboard allowed for defining a shortcut
> GLOBALLY, and the console, the display managers, and all the desktop
> environments respected that. Is supporting "a global shortcut" no
> longer a Debian/Ubuntu goal?

It's not unsupported. "dpkg-reconfigure keyboard-configuration" can be run whenever you like.

What we know is that a global shortcut isn't compatible with the way gnome-control-center currently works, since it may easily be overwritten as soon as the user makes changes via Settings.

We also know that setting Alt+Shift prevents a user (at least on GNOME) from using Alt+Shift+<some letter> shortcuts. So the previous default came with that issue (which itself is a very old bug; see bug #36812).

And it's not obvious that just Alt+Shift is the best choice for a global shortcut; many users would prefer Ctrl+Shift, CapsLock, or something else.

Those three reasons make up the background for the changes made in response to bug #1762952.

>> As regards the console, is there a use case which would be worth
>> to take into consideration where there is a need to switch to a
>> non-latin keyboard layout?
>
> Normal PC usage is impossible without it. Some examples:
>
> # Change directory to Desktop cd "Επιφάνεια εργασίας"
>
> # Edit a file nano Αρχείο.txt
>
> # Set a user's name usermod -c Άλκης alkisg

Those are tasks which I would accomplish with a terminal emulator such as gnome-terminal, not a TTY console.

But I think the approach I propose (i.e. special case GNOME) may be a bit complex code wise, and I'm disinclined to propose the actual code changes. It didn't fall out well last time I tried to change the console-setup package. :/

So we would need some involvement from Adam or someone else in the ~canonical-foundations team to evaluate whether the idea is sensible and doable. Pinging some of them on IRC about this bug report might help moving it forward.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alkis Georgopoulos (alkisg)]

The issue is still there in today's Ubuntu MATE 22.04 daily built.
Greeks cannot even type their names in the installer (ubiquity)...

[Gunnar Hjalmarsson (gunnarhj)]

Ubiquity? That sounds more like bug #1875062.