No way to invoke OSK when a hardware keyboard is connected

[Jouni Helminen]

There are mice on the market that claim to be keyboards as well. This inhibits OSK. We can try and find out how many keys they have and be smart about it.

Regardless, we should have a way of forcing the OSK, especially on touch-enabled devices. Whether it's triggered by a touch (as opposed to mouse click) on the text area, or with a button in the text field (replaced with the clear icon when you start typing). Another option is the copy/paste dropdown.

UX solution:
Interim solution until the ideal solution can be implemented : https://docs.google.com/document/d/1YbA0LEwbsNVf2O5C0ObNFRv3Tsuqn2HyYCsjEcY02OA/edit#

Ideal UX solution
<https://goo.gl/CJ1YLB>:
------------
The on-screen keyboard should appear if:
* no external keyboard is connected, and a text field becomes focused; or
* an external keyboard is connected, but the most recent text typed anywhere was with the OSK, and a text field becomes focused; or
* an external keyboard is connected, and the most recent text typed was with an external keyboard, but you tap in the already-focused text field. (App developers are responsible for emulating this behavior in typable areas that are not text fields.)
------------

== original description ==

1. Boot nexus 7
2. In the login screen, test that OSK works by tapping the password textfield
3. swipe keyboard back down after testing it works
4. Turn on Bluetooth mouse to get a mouse cursor
5. Click on password textfield

OSK doesn't pop up, touch tapping on the field doesn't bring up the OSK anymore either.

ie. The login screen is not usable with mouse connected - cannot type in password via OSK since it doesn't come up when clicking the password textfield with mouse.

See attached screenshot. Focus on textfield. Would expect OSK to show up.

[Michael Zanetti (mzanetti)]

Need more information about your device. Please follow this guide to get some logs:
https://docs.google.com/document/d/19Tr1a42bK7jG6mOc7axpSySwyajYpuHaFB4fSgdfn28/edit#

[Evan Wang (wsy324)]

I also reproduced this issue on N4, OSK does not come up once BT mouse is connected, no matter BT keyboard is connected or not, the OSK will come up if disconnect the BT mouse, got the below output, fyi.

$ adb shell ./lsinput
/dev/input/event0
   bustype : (null)
   vendor : 0x0
   product : 0x0
   version : 0
   name : "pmic8xxx_pwrkey"
   phys : "pmic8xxx_pwrkey/input0"
   bits ev : EV_SYN EV_KEY

/dev/input/event1
   bustype : BUS_I2C
   vendor : 0x1
   product : 0x1
   version : 1
   name : "keypad_8064"
   phys : "keypad_8064/input0"
   bits ev : EV_SYN EV_KEY EV_MSC

/dev/input/event2
   bustype : (null)
   vendor : 0x0
   product : 0x0
   version : 0
   name : "touch_dev"
   bits ev : EV_SYN EV_ABS

/dev/input/event3
   bustype : (null)
   vendor : 0x0
   product : 0x0
   version : 0
   name : "apq8064-tabla-snd-card Button Ja"
   phys : "ALSA"
   bits ev : EV_SYN EV_KEY

/dev/input/event4
   bustype : (null)
   vendor : 0x0
   product : 0x0
   version : 0
   name : "apq8064-tabla-snd-card Headset J"
   phys : "ALSA"
   bits ev : EV_SYN EV_SW

/dev/input/event5
   bustype : (null)
   vendor : 0x1
   product : 0x1
   version : 1
   name : "hs_detect"
   bits ev : EV_SYN EV_KEY EV_SW

/dev/input/event6
   bustype : BUS_BLUETOOTH
   vendor : 0x46d
   product : 0xb014
   version : 4608
   name : "Bluetooth Mouse M336/M337/M535"
   phys : "e1:d1:70:f7:d6:98"
   uniq : "ae:5b:98:20:1f:00"
   bits ev : EV_SYN EV_KEY EV_REL EV_ABS EV_MSC EV_LED EV_REP

[Jouni Helminen (jounihelminen)]

On windows 10 there is an icon in the bottom bar that invokes the OSK. I can think of situations you want to invoke it that are not textfield related. Perhaps we could add an icon in the indicators?

And I agree, there should probably be a system setting for auto-invoking it which you can turn on/off. Creating a new bug and assigning a UX designer on this

[Bill Filler (bfiller)]

@design
please triage as we need a proposed solution for this quickly I think

[Matthew Paul Thomas (mpt)]

Taking this since I'm responsible for text fields/editing in the toolkit spec.

[Matthew Paul Thomas (mpt)]

Specification updated. <https://goo.gl/CJ1YLB> However, that is unfortunately a Canonical-only link, toolkit spec changes in general require approval from someone who is not available today, and they aren't visible to others until they are approved.

So for convenience, here's what I wrote:
------------
The on-screen keyboard should appear if:
* no external keyboard is connected, and a text field becomes focused; or
* an external keyboard is connected, but the most recent text typed anywhere was with the OSK, and a text field becomes focused; or
* an external keyboard is connected, and the most recent text typed was with an external keyboard, but you tap in the already-focused text field. (App developers are responsible for emulating this behavior in typable areas that are not text fields.)
------------

This avoids having to design a visual element for, valuable screen space being taken up by, and users having to learn, a separate control for revealing the OSK. If it doesn't appear when they want it to, they can just do what they'd probably try doing anyway, which is tapping again where they want to type.

Meanwhile, this bug report seems to have morphed from the original problem of Ubuntu believing a mouse when it claims to be a keyboard. Is there a separate bug report for that now?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Pat McGowan (pat-mcgowan)]

I am unclear if we can implement it as described since the shell needs to know the text field was entered and that it was a touch event, and it needs to know how the last entry was made which afaik it doesnt

[Michał Sawicz (saviq)]

Agreed, we'll need to discuss it. Today the apps talk to the OSK (maliit) directly, we often talked that this is not ideal, and that the communication should go through the display server, in which case we would likely have easier access to the information.

We do have a comm channel open between maliit and the shell, but it's never been our long-term plan to use it. Also SDK might need adapting here.

[Michał Sawicz (saviq)]

As a power-user option we'll add a device blacklist that the user will be able to populate to ignore certain devices when deciding whether to show the OSK and/or switch to windowed mode.

[Albert Astals Cid (aacid)]

I know i am one of the few people around that likes the BlackBerry 10 OS but they have a solution for this problem.

If you do a two finger swipe from the bottom, this always brings up the OSK no matter where you are.

[Lorn Potter (lorn-potter)]

There is iio-sensor-proxy and an iio-sensor-proxy plugin in QtSensors that works for device orientation on a variety of laptops.

[Matthew Paul Thomas (mpt)]

W.r.t. “SDK might need adapting”, yes, this is largely a toolkit bug, because a toolkit is the most practical component to keep track of when “a text field becomes focused”. You can’t expect every app author to do this themselves, and you can’t expect users to learn+remember any alternative way of invoking the OSK.

An example flow would be:
a. You tap in a text field.
b. The toolkit says to the shell, “hey, I just focused a text field”.
c. The shell checks whether an external keyboard is connected; it is.
d. The shell checks whether the last character typed anywhere was with the OSK; it was.
e. The shell therefore opens the OSK, and tells the toolkit that it’s done so…

(…The shell also tells the toolkit what part of the window, if any, the OSK is obscuring. Then the toolkit says, “oh dear, that’s over top of the text field”, and scrolls the view to keep the field visible. But that’s another story.)

[Dominique Ramaekers (dominique-ramaekers)]

with a Lenovo Yoga 500, same problem. if the screen is rotated 180°, it deactivates the hardware keyboard. I can't bring up the software keyboard.

[Matthew Paul Thomas (mpt)]

(The spec change I proposed a year ago has since been approved.)

[Bill Filler (bfiller)]

Short term solution will be to add a setting to system-settings and launguage indicator to show the OSK.
@saviq, @elleo
please agree on the setting so we can get started on the u-s-s piece

[Launchpad Janitor (janitor)]

This bug was fixed in the package indicator-keyboard - 0.0.0+17.04.20170301-0ubuntu1

---------------
indicator-keyboard (0.0.0+17.04.20170301-0ubuntu1) zesty; urgency=medium

  * Add an OSK switch to the keyboard indicator (LP: #1521518)

 -- Lukáš Tinkl <email address hidden> Wed, 01 Mar 2017 13:42:36 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package unity8 - 8.15+17.04.20170308-0ubuntu1

---------------
unity8 (8.15+17.04.20170308-0ubuntu1) zesty; urgency=medium

  [ Albert Astals Cid ]
  * Let's follow the keyboard order
  * Enable the license check (LP: #1590044)
  * Show a barebones error message if loading the QML doesn't work
  * Make sure cursor is back to normal when closing the window (LP:
    #1666158)
  * Fix small warning when sensitive is undefined
  * Remove unused member variable
  * Initialize m_msecsSinceReference
  * Add warning if registerService fails
  * Move menus Component {} out of the repeater
  * Add tool to browse menus of a running app

  [ Daniel d'Andrada ]
  * Drag child windows with Alt+LeftMouseButton (LP: #1664947)

  [ Gerry Boland ]
  * TopLevelWindowModel: connect to the OSK surface to know when it goes
    away so we can clean up after it. (LP: #1670681)

  [ Josh Arenson ]
  * Initialize sessionMode to single when mock is reset (LP: #1667463)

  [ Lukáš Tinkl ]
  * Make the menu string "Back" translatable
  * Let the OSK be driven by a switch in indicator-keyboard (LP:
    #1521518)
  * Disable the tutorial when there's no touchscreen (LP: #1661557)
  * Shell chrome fixes (LP: #1658117, #1665723, #1665724, #1669010)
  * Cancel the drag as soon as we're entering the spread (LP: #1668642)
  * Fix being unable to use window control buttons or menus with touch
    (LP: #1667604, #1668387)

  [ Michael Terry ]
  * Set QT_IM_MODULE in the greeter so that the OSK can come up. (LP:
    #1670383)
  * Simplify our startup handling of Mir environment variables and set
    the right socket path in snappy.

  [ Michael Zanetti ]
  * Cancel the home key activation if something else is pressed along
    with Meta (LP: #1670678)
  * close the drawer when something is launched from the launcher (LP:
    #1660367)
  * use a svg instead of a png for the Launcher's BFB (LP: #1668057)
  * change the default setting for the launcher autohiding (LP:
    #1670655)

  [ Nick Dedekind ]
  * Split up Shell::test_spreadDisabled to ensure initial state for each
    test.

 -- Michał Sawicz <email address hidden> Wed, 08 Mar 2017 09:56:46 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-system-settings - 0.4+17.04.20170301-0ubuntu1

---------------
ubuntu-system-settings (0.4+17.04.20170301-0ubuntu1) zesty; urgency=medium

  [ Michael Sheldon ]
  * Add option to enable the on-screen keyboard when an external
    keyboard is present (LP: #1521518)

 -- Lukáš Tinkl <email address hidden> Wed, 01 Mar 2017 13:42:41 +0000