xutils module using PyQt and Xlib

[Henning]

Hello,
I created a branch at https://code.launchpad.net/~henning-schroeder/xpresser/pyqt
Are you interested in integrating PyQt support?
Greets
Henning

[Gustavo Niemeyer (niemeyer)]

Thanks for the changeset Henning.

Please note that, even though it depends on gtk to *implement* its features, at this point there's absolutely nothing specific to gtk on the features and interface that Xpresser offers.

With this in mind, I don't see a big reason to be pushing two different implementations in parallel.

If we manage to find a way to avoid gtk *and* qt entirely, then that might be an interesting move. It should certainly be possible, given what we're doing with Xpresser.

As a side note, running tests with your qt interface breaks the test suite, which so far has almost 100% coverage, and ideally the screenshot taking routine should avoid touching the disk, for speed purposes.

[Henning (henning-schroeder)]

Sorry I had not looked into the tests yet. I fixed take_screenshot to grab only a part of the screen and now all tests succeed.
The temporary file was a quick hack because I could not find documentation how the numpy array of gtk.gdk.Pixbuf.get_pixels_array can be build.
But I have to agree it would be even better to not depend on any heavy toolkit. My branch currently only needs PyQt for taking the screenshot.
I found some code to do it with xlib but again I don't know how to create the numpy array. Do you have any idea?

[Quasar Jarosz (quasar-ja)]

Do we know if using xlib will work on windows clients? I know windows support is probably not really a consideration, but I'm interested in it, and keeping everything pure python seems like a good idea too. I will try out the PyQt version under windows today. I could not come across a working version of PyGTK with numpy support for windows to get the original working...

[Henning (henning-schroeder)]

Quasar, the PyQt branch won't fully work on Windows. It depends on xlib to simulate mouse clicks and key presses.
For windows you have to add new code, something like this:

def type(string):
  import win32com.client as comclt
  wsh= comclt.Dispatch("WScript.Shell")
  #wsh.AppActivate("Notepad")
  wsh.SendKeys(string)

[Quasar Jarosz (quasar-ja)]

Thanks Henning. I noticed xlib was not going to work on Windows. There is no cross-platform way to handle the mouse clicks right?

I have been using something similar to your example, or using another tool via com (like AutoItX) to handle clicking, typing, etc.

[Henning (henning-schroeder)]

I completed the xlib-only version :-)
For testing call the example or the test suite with XPRESSER_TOOLKIT=xlib
The PyQt version got improved, too. It now creates a numpy array instead of a temporary file.

[Gustavo Niemeyer (niemeyer)]

Oh, nice! Can you please attach the xlib-only branch to this bug?

[Henning (henning-schroeder)]

Ok I have linked my branch with this bug now. Currently the branch still contains code for PyGtk and PyQt. Do you think this support should be removed?

[Gustavo Niemeyer (niemeyer)]

Yeah, I don't see a reason to maintain multiple backends in this case. What we want to offer is a first class GUI automation library, independent of backend.

The only question I have is about speed. I suppose the generic code is doing it all in Python. Have you made some back-of-the-envelope performance checks to see how the interpreted backend behaves?

[Henning (henning-schroeder)]

I have done some benchmarking. At first I thought the bottle neck would be xlib but my numpy code actually took more time than grabbing the screen. This is fixed now in r14. The optimized numpy code is now three times faster. Here is a benchmark with timeit of take_screenshot() of my laptop (1280x900)

xutils_gtk: 0.15272808075
xutils_qt: 0.153820991516
xutils_xlib: 0.329011201859

This does not look too bad IMHO.

[Henning (henning-schroeder)]

In r15 I have removed PyGtk and PyQt support.

Ran 48 tests in 41.908s
PASSED (successes=48)

By the way: With kwin a test may fail because the window manager shows a tooltip when the mouse cursor is over a window icon.
Because the tooltip overlaps with the test image it cannot be recognized correctly.

[Gustavo Niemeyer (niemeyer)]

Thanks a lot Henning.

The speed is a little concerning to me, since it's more than twice as slow in a procedure which is already very slow nowadays. Since the visible benefits of the switch are pretty low, it still makes me wonder about the switch over.

Nevertheless, I'll look at your implementation and see if there's anything I could to do speed it up, since it'd be nice to be pure Python there.

Oh, and can you please sign the Canonical contributor agreement before that, so that I can actually integrate the changes if there's a way to? The details are at:

https://www.canonical.com/contributors

Thanks a again, Henning.

[Henning (henning-schroeder)]

I just signed the Canonical contributor agreement but haven't got any response so far.

I played with http://code.google.com/p/pyxlib-ctypes/ which looked interesting to get more speed. Actually grabbing a screenshot and converting to a numpy array only takes 0.109460830688 seconds. But it lacks support for the xtest extension (fake_input). So both python-xlib and pyxlib-ctypes would be needed.

[Gustavo Niemeyer (niemeyer)]

Wow, it's *faster* than gtk? That's awesome!

Maybe we can convince the author to add the xtest extensions, and get someone to package it for Ubuntu.

[Gustavo Niemeyer (niemeyer)]

Actually, being ctypes, we might simply bundle it into a subdirectory of xpresser itself, I believe.

I've just filed a bug into pyxlib-ctypes with the suggestion:

http://code.google.com/p/pyxlib-ctypes/issues/detail?id=1

Let's see what they say.

PS. Thanks for signing and submitting the contributor agreement!

[Henning (henning-schroeder)]

I integrated pyxlib_ctypes in r16

Ran 48 tests in 37.299s
PASSED (successes=48)

[Gustavo Niemeyer (niemeyer)]

After reading the full patch and thinking about this for a while, I've come to a conclusion on this.

This is a refactoring made purely to avoid the dependency on gdk, which is a library to interact with X.

To avoid the gdk library, though, we're adding a dependency on three new modules: python-xlib, autox.py, and pyxlib_ctypes, and are embedding 8 thousand lines of code to avoid the issue that some of these dependencies are not pre-packaged in most distributions.

With this rationale, i don't think it makes much sense to integrate these changes from a maintenance and usage perspective. There's no actual benefit for the library user, and replacing three calls to a single X library by several calls to multiple libraries and embedding some of them in the code will increase maintenance cost which I don't want to have.