Unity8/Mir is unable to open autopilot uinput devices

[Thomi Richards]

When running unity8 on top of mir, the input events generated by autopilot have no effect on the application under test. I've spent the best part of a day debugging this, so what follows is a brain dump of everything we've tried, so others can continue the effort while I sleep.

Basic Notes:
===========

To run the autopilot tests, there are a few things you need to do - these steps all need to be run as the phablet user (from the host, run: "adb shell" then "sudo -i -u phablet"):

0) If it hasn't already made it into the image, install the patched python-ubuntu-platform-api package from here: http://people.canonical.com/~thomir/python-ubuntu-platform-api_1.1daily13.06.13-0ubuntu1_armhf.deb

1) Get the test cases. From the host, run 'phablet-click-test-setup'.

2) Switch to running mir: "touch ~phablet/.display-mir && sudo reboot"

3) Install the test dependencies. In this example I'll use the ubuntu ui toolkit test suite:

sudo apt-get install ubuntu-ui-toolkit-autopilot

4) Unlike surface flinger, you *must* have the unity8 shell running when trying to run a test suite. This is becasue without unity8, there's no display server, so you can't launch any applications. This caught me out for a while, because the error message you get isn't particularly intuitive.

5) Now you can run the test. Unlock the unity shell (i.e.- make sure it's not on the greeter) and then I recommend just running a single test:

cd ~/autopilot
autopilot run -v ubuntuuitoolkit.tests.gallery.test_gallery.GenericTests.test_navigation

The test will fail because the input events never make it through to the application.

Debugging mir:
=============

If you export some environment variables...

initctl set-env MIR_SERVER_INPUT_REPORT=log
initctl set-env MIR_SERVER_LEGACY_INPUT_REPORT=log

and then restart unity8 (so those environment variables are picked up), then re-run the autopilot test, you'll get additional log output from mir in ~/.cache/upstart/unity8.log

If you run those tests again, you'll see two messages like this:

[EE, android-input] [EventHub]could not open /dev/input/event6, Permission denied
[EE, android-input] [EventHub]could not open /dev/input/event7, Permission denied

This is mir finding the new device nodes that autopilot creates, but failing to open them.

I ran unity8 under strace, and the resulting log file is here:

http://people.canonical.com/~thomir/strace_out.gz

It's a large log file, but the relevant sections are:

6214 open("/dev/input/event7", O_RDWR|O_LARGEFILE|O_CLOEXEC <unfinished ...>
6214 <... open resumed> ) = -1 EACCES (Permission denied)

Note that, at this point, you probably want an easier way to reproduce this than running the entire autopilot test. You can simply run this in a shell:

python -c "from time import sleep; from autopilot.input import Touch; t = Touch.create(); sleep(60)"

Debugging Permissions:
=====================

So it looks like a file permissions problem. We've looked at the following:

i) The file permissions in the ubuntu-side:

$ ls -l /dev/input/event*
crw-rw---- 1 root android_input 13, 64 Oct 1 23:37 /dev/input/event0
crw-rw---- 1 root android_input 13, 65 Oct 1 23:37 /dev/input/event1
crw-rw---- 1 root android_input 13, 66 Oct 1 23:37 /dev/input/event2
crw-rw---- 1 root android_input 13, 67 Oct 1 23:37 /dev/input/event3
crw-rw---- 1 root android_input 13, 68 Oct 1 23:37 /dev/input/event4
crw-rw---- 1 root android_input 13, 69 Oct 1 23:37 /dev/input/event5
crw-rw---- 1 root android_input 13, 70 Oct 2 02:36 /dev/input/event6
crw-rw---- 1 root android_input 13, 71 Oct 2 02:36 /dev/input/event7

ii) The permissions android-side:

$ sudo android-chroot
root@android:/ # ls -l /dev/input/event*
crw-rw---- root input 13, 64 2013-10-01 23:37 event0
crw-rw---- root input 13, 65 2013-10-01 23:37 event1
crw-rw---- root input 13, 66 2013-10-01 23:37 event2
crw-rw---- root input 13, 67 2013-10-01 23:37 event3
crw-rw---- root input 13, 68 2013-10-01 23:37 event4
crw-rw---- root input 13, 69 2013-10-01 23:37 event5
crw-rw---- root input 13, 70 2013-10-02 02:36 event6
crw-rw---- root input 13, 71 2013-10-02 02:36 event7

iii) We've verified that this is not apparmor doing something to us:

$ grep DENIED /var/log/syslog

Other Interesting Things:
=====================

If we create the autopilot touch device before unity8 starts, then mir is able to open the device nodes just fine. This suggests to me that:

 - it's not something to do with these devices being uinput, rather than "real" devices.

The code that opens this device is (in lp:mir HEAD)

3rd_party/android-input/android/frameworks/base/services/input/EventHub.cpp line 959 and looks like this:

~~~
int fd = open(devicePath, O_RDWR | O_CLOEXEC);
if(fd < 0) {
    ALOGE("could not open %s, %s\n", devicePath, strerror(errno));
    return -1;
}
~~~

I wondered if the open mode flags were causing a problem, but a simple program that opens the same device in the same way works perfectly.

Current Suspicions:
=================

The big clue here seems to be that mir can open all devices when it starts, but fails to open devices that are created later. Perhaps:

 - mir has fewer priviledges / permissions later, than it does when it starts up?

 - perhaps there's a race condition, and both mir *and* something else are trying to access the same device node at the same time, and this is somehow causing problems.

[Thomi Richards (thomir-deactivatedaccount)]

Should have mentioned - this is using the devel-proposed ubuntu-system image.

[kevin gunn (kgunn72)]

could it be related to this https://bugs.launchpad.net/mir/+bug/1233245 (volume up/down key not working)

[Daniel van Vugt (vanvugt)]

Yeah bug 1233245 smells similar. Like in both cases Mir is holding/locking input devices.

[kevin gunn (kgunn72)]

robert_ancell tried replicating the problem on mir on the desktop to no avail.

wonder about the 3rd_party/android-input/android/frameworks/base/services/input/EventHub.cpp
~~~
int fd = open(devicePath, O_RDWR | O_CLOEXEC);
if(fd < 0) {
    ALOGE("could not open %s, %s\n", devicePath, strerror(errno));
    return -1;
}
~~~

this code is effectively duplicated from android upstream. historically this has been in libui.so or libinput.so. is there something that normally happens with surfaceflinger that isn't happening with mir - something primordial in terms of android startup ?

[Daniel d'Andrada (dandrader)]

So, as a workaround for now (just to get things going as investigation on this bug continues), can't autopilot just restart unity8 after creating those fake input devices?

[Daniel d'Andrada (dandrader)]

I've added a bit more debug to EventHub and here's what I got:

[EventHub]owner permissions: rw
[EventHub]group permissions: --
[EventHub]owner: root
[EventHub]group: root
[EventHub]could not open /dev/input/event6, Permission denied

[EventHub]owner permissions: rw
[EventHub]group permissions: --
[EventHub]owner: root
[EventHub]group: root
[EventHub]could not open /dev/input/event7, Permission denied

So those files permissions are changed only later in time, after EventHub has already handled the inotify telling about their creation.

[Daniel d'Andrada (dandrader)]

So my theory is that those input files are first created (with default root permissions) and shortly after have their group permissions updated to android_input rw (from root --).

If that holds, one solution would be to make EventHub a bit smarter and retry opening an input file if its permissions have changed...

[Daniel d'Andrada (dandrader)]

or update uinput or whatever uses it to create those files already with their final permissions

[Martin Pitt (pitti)]

How did you enable Mir on the current builds? I just flashed --channel devel-proposed on mako (Nexus G4) and it's still using SF.

How do the permissions of "real" input devices compare with the uevent synthesized ones? I expect they are the same (root:android_input 664), then it seems your hunch about Mir dropping "android_input" group privileges at some point is right. In a world of hotplug input devices it must not do that, as input devices can come and go at any time. Even on a phone one might attach an USB or a bluetooth keyboard. It might be interesting to strace the Mir process for calls like setresuid(), setresgid(), setegid() and the like.

[Olli Ries (ories)]

re comment #9:

that's from the initial description:
2) Switch to running mir: "touch ~phablet/.display-mir && sudo reboot"

[Martin Pitt (pitti)]

Pardon my ignorance. "touch ~/.display-mir" enables Mir, and there is no such thing as a "Mir server process". It's all going through libraries only, so on the Ubuntu side unity8 and other user process directly access input devices in /dev/input/events/* (through its bundled copy of the Android input stack). This only works because the phablet user is in the android_input group (yuck big security hole!)

According to Daniel d'Andrada, Mir does not listen to uevents to pick up new input devices, but uses an inotify watch on /dev/input/ (!). So what likely happens is the following:

(1) a new input device gets created by uinput, or an USB/bluetooth hotplug event
(2) /dev/input/eventX gets created, with the usual initial root:root 600 permissions
(3) inotify event gets sent to Mir, which tries to open it and can't
(4) udev applies its rules and changes mode to root:android_input 0600
(5) Mir does not try to re-open the device after an initial failure, and thus can't see the device

So as an immediate hotfix I suggest that Mir needs to re-try opening the device again after it sees a *change* of an existing device.

But I strongly recommend dropping the inotify approach (which really isn't meant for this kind of thing) and use the much more efficient and race free uevent handling like everything else that handles hardware. With libudev you get the add/change events after rules are processed (thus the devices have the correct permissions and properties when you receive the event), and you avoid wakeups on every change in /dev/input, as you can tell libudev to only call back on events for a particular subsystem ("input" in that case).

[Martin Pitt (pitti)]

For the record, http://www.freedesktop.org/software/systemd/libudev/ is the reference documentation (also in libudev-dev); you want the udev_monitor_* bits for receiving events, the enumerate bits for detecting devices at startup (coldplug), and udev_device_* for querying for attributes and properties (grep "udevadm info --export-db" for "input" about available properties, such as ID_INPUT_KEYBOARD=1).

http://www.signal11.us/oss/udev/ has an useful intro and example how to use this.

[Daniel d'Andrada (dandrader)]

A fix for the permissions issue is in:
https://code.launchpad.net/~dandrader/mir/hack_lp1233944/+merge/188889

Although, as its description says, that might be further issues to solve.

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:~mir-team/mir/development-branch at revision None, scheduled for release in mir, milestone Unknown

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:mir at revision None, scheduled for release in mir, milestone phone-v1-freeze

[Launchpad Janitor (janitor)]

This bug was fixed in the package mir - 0.0.13+13.10.20131003-0ubuntu1

---------------
mir (0.0.13+13.10.20131003-0ubuntu1) saucy; urgency=low

  [ kg ]
  * bump version for ABI break (LP: #1229212)

  [ Robert Ancell ]
  * Bump version to 0.0.12

  [ Kevin Gunn ]
  * bump version to 0.0.13

  [ Daniel d'Andrada ]
  * Fix for LP#1233944 Fixes the Mir-side of bug
    https://bugs.launchpad.net/mir/+bug/1233944 Event files are first
    created with root:root permissions and only later udev rules are
    applied to it, changing its permissions to root:android-input and
    therefore making it readable by unity8-mir in short: Retry opening a
    file when its permissions change as it might be readable now. (LP:
    #1233944)

  [ Robert Carr ]
  * Fix for LP#1233944 Fixes the Mir-side of bug
    https://bugs.launchpad.net/mir/+bug/1233944 Event files are first
    created with root:root permissions and only later udev rules are
    applied to it, changing its permissions to root:android-input and
    therefore making it readable by unity8-mir in short: Retry opening a
    file when its permissions change as it might be readable now. (LP:
    #1233944)

  [ Ubuntu daily release ]
  * Automatic snapshot from revision 1089
 -- Ubuntu daily release <email address hidden> Thu, 03 Oct 2013 06:34:41 +0000

[Daniel van Vugt (vanvugt)]

Merged at 1089 means it went in 0.0.13.