[research] Sikuli over WebDriver API / JSONWireProtocol?

[RaiMan]

complete info in related question
------------------------------------------------

Has anyone thought about or be interested in using/working with Sikuli via a WebDriver API (e.g. the JSONWireProtocol)?

I don't think such a project exists yet, and I had plans to work on a proof of concept, but never got around to it yet.

Basically, we would wrap the Sikuli (Java) API around the WebDriver API such that any WebDriver language binding can invoke Sikuli.

In general we can map the click and sendKeys() methods and whatever else is applicable, and we would only support a subset of the WebDriver API.

Find elements would be based on captured saved (PNG) images for Sikuli to use and be based on the following WebDriver-based location strategies:

By name - name of PNG image to find, stored in some default location we define globally

By ID - optional to decide how to implement, could also be alias to by name

By XPath - absolute (or relative) path to image in filesystem (probably no validation of the "XPath", we just check that the filepath exists. In essence, we override XPath to mean path to file system rather than really XPath

By CSS selector (or any other unused strategy) - override it as a way to send base64 encoded representation of a PNG image. For use in remote WebDriver / Grid type test deployment where you can send a file locally (read bytes into memory and encode as base64) to the server (which then decodes the base64, saves to temp location, and uses it with Sikuli to locate the "element"). In essence, this would be similar to WebDriver's way of handling file uploads (via sendKeys) and FirefoxProfiles for remote WebDriver/Grid deployments where the file/profile need not reside on the actual node but can be from the machine sending the WebDriver commands.

I think this type of solution offers a good way to integrate Sikuli with Selenium/WebDriver/Appium/ios-driver tools for a system integration type test or where those tools are lacking that Sikuli complements. Something more powerful and flexible than (Firefox) SikuliWebDriver (https://code.google.com/p/sikuli-api/wiki/SikuliWebDriver) and which would work over local/remote/grid deployments.

This would be in essence, building a WebDriver server interface to Sikuli so that WebDriver clients can call Sikuli as if it was a WebDriver with minimal code changes on the client side to support.

Let me know what you think of this.

[RaiMan (raimund-hocke)]

just to have it on my list.

please add future comments here.