input events not being flushed in browser until pressing hardware keys

[Bill Filler]

using webbrowser-app from Silo 9 and oxide in universe

Log in to Facebook on the phone
Click "comment" in someone's post
Start typing

Expected results:
The comment button should become enabled so you can submit comment

Actual results:
The comment button always stays disabled.

Tested this on webbrowser-app on desktop with oxide (explicilty loading m.facebook.com) and becomes enabled as it should after typing. Seems specific to device. Guessing it's a UA String issue.

Is there a way to see the JS console to see if a JS error is occuring?

[Bill Filler (bfiller)]

Looking at the specifc Facebook code in the page. It's basically a <textarea id="composerInput"> and a <button> in a <form> as follows:

<form method="post" class="composer_form" data-store="&#123;&quot;expand_composer&quot;:true&#125;" id="comment_form_1068640102_10202428868777549" action="/a/comment.php?actionsource=2&amp;ft_ent_identifier=10202428868777549&amp;gfid=AQB-5WrCooTqARii&amp;__av=726399492&amp;refid=52" onsubmit="return window.MAjaxify&amp;&amp;MAjaxify.form(event,this,&quot;async_composer&quot;,&quot;cache&quot;,null,false);">
<input type="hidden" name="fb_dtsg" value="AQAEgIey" autocomplete="off" />
<input type="hidden" name="charset_test" value="€,´,€,´,水,Д,Є" />
<table cellspacing="0" cellpadding="0" class="comboInput"><tr><td class="inputCell"><div class="mentions" id="u_0_6">

<textarea class="composerInput mentions-input" id="composerInput" name="comment_text" rows="1" placeholder="Write a comment..." cols="15" data-sigil="textarea"></textarea></div></td>
<td class="btnCell"><button type="submit" value="Comment" class="touchable _56bs _5of- _56bt" disabled="1" name="post" data-sigil="touchable composer-submit">
<span class="_55sr">Comment</span></button>
</td>
</tr>
</table>
</form>

There must be some code that listens for changes to the textarea and enables the button appropriately and this is not working.
Can we verify that our input is correctly getting propagated to the underlying textarea on the device? Seems as if it's not somehow, even though it is correctly displayed. Perhaps the QtInputMethod is not getting commited correctly on multi-line text areas?

[Bill Filler (bfiller)]

Pat found out some quite interesting information. If you type in the text field and then press the volume down button (or power button), the "comment" button becomes enabled!

Seems the same behavior is happening on other sites that display search-as-you-type results. Like ebay.com or m.imdb.com.
The behavior is this:
- go to ebay.com on the device
- start typing in a search (like "cars")
- you see no results
- press the down volume button
- All of a sudden the input events seem to get flushed and the results are shown

Somehow input events are not being flushed until pressing one of the hardware buttons. What can explain this behavior?

[Bill Filler (bfiller)]

note this happens with both oxide and qtwebkit based browsers, so not related to the browser backend.

[Olivier Tilloy (osomon)]

If a JS error was happening, we’d see it in ~/.cache/upstart/application-legacy-webbrowser-app-.log.
But I’m not seeing anything there when browsing to ebay.com and following the steps in comment #2 to reproduce the issue (which I can reliably reproduce on my N7).

[Olivier Tilloy (osomon)]

I’m attaching a simple HTML page with which I cannot reproduce the issue. Whatever facebook, ebay and imdb do, they apparently don’t listen to the 'input' event of the text box.

[Olivier Tilloy (osomon)]

If, in the above HTML example, I change "input" to "keyup" as the name of the event, I can reproduce the issue on my N7 (and pressing a hardware button such as volume up or volume down "flushes the state").

So it seems that those websites, instead of listening to the "input" event, only listen to key events. However they work fine on e.g. chrome on android, so there must be something chrome is doing that triggers a key event when the OSK is used to input text, or something similar.

[Olivier Tilloy (osomon)]

I tested webbrowser-app on my desktop with the sample HTML page, and using onboard (an OSK) to input text, and the "keyup" events are received, which seem to indicate that the problem could be in maliit not sending the corresponding keyboard event on devices.

[Olivier Tilloy (osomon)]

I have instrumented oxide::qt::RenderWidgetHostView::HandleKeyEvent(QKeyEvent*) in oxide and when running on desktop with onboard the method is being called (twice, keydown and keyup events) for each key pressed on the virtual keyboard. On device, the method is never called when pressing keys on the OSK, which explains why no "keyup" event is received on the javascript side.
Pressing the volume up/down hw keys does call HandleKeyEvent(…) (but the previous events have been lost, they are not being flushed as initially thought).

The next step is to instrument maliit itself to try and understand whether it emits the key events or not, and if it does, what happens to them.

[Bill Filler (bfiller)]

I applied all outstanding patches from upstream and rebuilt maliit-framework, but still experiencing the same behavior on device. I decided to try running maliit on the desktop to see what the behavior is there. I was able to get it to work (barely) but enough to test the behavior and it appears the key events are not making it to the webbrowser-app.

Here is how I tested on the desktop:
- install maliit-framework, ubuntu-keyboard-english
- export QT_IM_MODULE=maliitphablet
- maliit-server
- from another window, also export QT_IM_MODULE=maliitphablet
- run webbrowser-app http://m.imdb.com (don't move the window at all)
- click in the search field
- you'll see a white rectangle at the bottom of the screen where the OSK should be showing, but it's not displaying correctly
- start clicking in the white rectangle and characters will start getting displayed in the search field,
- you don't see the drop down of suggestions being displayed, which means the site is not recieving key press events
- if you do anything else in the browser that causes the keyboard to change it crashes the app somewhere in maliit

@olivier would be good for you to do this same test with your instrumented code to make sure we're not getting the events on the desktop as well

[Bill Filler (bfiller)]

I found this very simple Qt based virtual keyboard here: git://anongit.kde.org/scratch/tolszak/MockupVirtualKeyboard.git with an explanation here: http://tolszak-dev.blogspot.com/2013/04/qplatforminputcontext-and-virtual.html

It's a simple virtual keyboard implemented in Qt/QML just like maliit. Tested it on the desktop. Same behavior as maliit. Could be a Qt bug that is causing this issue.

To test this keyboard do the following:
- git clone <url from above>
- cd to directory
- qmake
- make all
- sudo make install
- then export QT_IM_MODULE=mockup
- run webbrowser-app and you'll get the OSK appearing when you click in input field

[Olivier Tilloy (osomon)]

I tested with my instrumented version of oxide on desktop:

 - I was able to run maliit on desktop with your instructions, and the keyboard displayed correctly at the bottom of the screen, however clicking the keys didn’t send any events (the text field in the webpage didn’t even get updated)

 - the mockup virtual keyboard works (input field gets updated), but I can confirm that oxide doesn’t receive the key events

This seems to discard maliit as a possible culprit, and points to Qt (I also tested with webbrowser-app with QtWebKit as a backend, and the same issue is there, so I don’t think oxide is doing anything wrong).

[Olivier Tilloy (osomon)]

Actually, grepping the code of MockupVirtualKeyboard, there’s no trace of QKeyEvent, so it doesn’t seem this OSK sends any key event at all. Which explains why we’re not getting any in oxide. So this doesn’t discard maliit as a possible culprit.

[Olivier Tilloy (osomon)]

I had a closer look at ubuntu-keyboard’s code, and it seems to send key events only for "special" keys (such as backspace). Normal character keys don’t trigger a system key event, and that seems intentional (I tested sending a key event for all keys, but in that case characters get input twice in the text field). So it might be that this issue can be dealt with in oxide itself, by faking a key event when text is being input.

[Olivier Tilloy (osomon)]

Also note that onboard is build for X11, it uses XTestFakeKeyEvent to generate a key event when clicking a key, so it can’t be compared to Qt’s architecture, which is display server independent.

[Olivier Tilloy (osomon)]

There are relevant comments in https://code.google.com/p/chromium/codesearch#chromium/src/content/browser/renderer_host/gtk_im_context_wrapper.cc (in particular in GtkIMContextWrapper::ProcessInputMethodResult(…)) which suggest that oxide should probably have a similar logic to send native keyboard events when receiving input method events.

[Olivier Tilloy (osomon)]

I changed the affected projects to oxide and ubuntu-keyboard, as fixing this bug will require fixes in both code bases:

 - send fake key events in oxide when composing with an IM
 - patch ubuntu-keyboard to send KeyRelease events when backspace is pressed (it only sends KeyPress events atm)

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-keyboard - 0.99.trunk.phablet2+14.04.20140402-0ubuntu1

---------------
ubuntu-keyboard (0.99.trunk.phablet2+14.04.20140402-0ubuntu1) trusty; urgency=low

  [ Olivier Tilloy ]
  * When sending a key event, send KeyPress AND the corresponding
    KeyRelease event. (LP: #1300382)
 -- Ubuntu daily release <email address hidden> Wed, 02 Apr 2014 19:32:28 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package oxide-qt - 1.0.0~bzr475-0ubuntu1

---------------
oxide-qt (1.0.0~bzr475-0ubuntu1) trusty; urgency=medium

  * Update to r475
    - Update to Chromium 35.0.1916.6
    - If WebView.newViewRequested isn't implemented, navigations should happen
      in the originating view
    - Refine WebView.newViewRequested and WebView.navigationRequested API's
      (LP: #1300891)
    - Improve the form factor detection (LP: #1301678)
    - Add WebView.javaScriptConsoleMessage signal to allow the application
      to intercept console messages from content (LP: #1291389)
    - Improve the LoadEvent API
    - Fix an abort when setting WebContext.sessionCookieMode on a context
      that has no path (LP: #1301650)
    - Expose the default WebContext via Oxide.defaultWebContext() (LP: #1297552)
    - Send fake keydown and keyup events when composing text with an
      input method (LP: #1300382)
    - Bump the API version to 1.0
 -- Chris Coulson <email address hidden> Fri, 04 Apr 2014 19:28:27 +0100