[request] Transparent pixels should be ignored in matching process

[Vladimir Petrov]

OpenCV 3+ now seems to have a matchTemplate() with a masking feature.
To be tested as candidate for an implementation in version 2.

-----------------------------------------------------------------------

I'm using Sikuli 0.10.2 on a 32-bit Windows XP system. I just noticed that if you are using a png file, containing transparent pixels as a pattern for image recognition, these pixels are not skipped during the matching process, thus resulting in a wrong match if you have some kind of background under the transparent part of the image. I think this behavior is incorrect - and Sikuli could be even more powerful if able to skip some pixels from the pattern.

[Human (bob-igo)]

I would like to make the opposite request, that the current behavior remains. Editing the captured PNGs to contain transparent regions is the easiest way to interact with GUI elements that change according to time or date.

For example, a calendar popup is launched by clicking on the current date. It's not always feasible to match nearby landmarks and use offsets to get to where the current date is displayed. Removing the date section of the image and making it transparent will cause it to match _any_ date.

[RaiMan (raimund-hocke)]

Both requests are the same:
transparent parts of an image should be ignored during the image search (in the first case the outer parts in the second some inner parts of the reference image).

This is a special case for an option to define some masking (in this case use transparency), to exclude parts of an image from the matching process.

In the second case transparency would not be needed, if it was possible, to have an option for the Pattern object like
Pattern(image).exclude(offset, region)
where offset is the upper left offset of region inside the image, that should be ignored.
This could even be supported in the Preview window like TargetOffset.
I am not sure, wether it makes sense, to have more than one of these exclude-regions per image.

additional:
Pattern.getExclude()
Pattern.setExclude()

to dynamically adjust the exclude regions.

[Denis K (dnsk)]

This is a really useful feature and this request is one year old!! Do we have a chance to get this feature?

[Laurent Vaucher (laurent-vaucher)]

If I was interested in trying to implement that feature, where should I start?
I've seen that the matching process relies on OpenCV function 'matchTemplate', but from its documentation I'm not even able to tell if there is a way to exclude some pixels.

[RaiMan (raimund-hocke)]

Hi Laurent,

I just made some tests and found out, that transparency works on backgrounds that are not too dark.

I created a png image of the Sikuli logo from the launchpad page and made the surrounding white pixels transparent (i have it attached) (done with Pixelmator on my Mac ;-).

On the white background it is found without any problems. Even background with some colored structure is not a problem, as long as it is not too dark.
Might be, that this is because (I am not sure so) internally the images are turned to grayscale under normal conditions (finder.cpp -> TemplateFinder::find()). I will later look into the code again.
Furthermore, I have no idea, how the transparent pixels are treated in The OpenCV.matchTemplate() with the used method CV_TM_CCOEFF_NORMED.

Nevertheless I think it is worth some testing.

BTW: I am not one of the developers ;-)

Raimund

Am 06.02.2012 um 16:39 schrieb Laurent Vaucher:

> If I was interested in trying to implement that feature, where should I start?
> I've seen that the matching process relies on OpenCV function 'matchTemplate', but from its documentation I'm not even able to tell if there is a way to exclude some pixels.
>
> --
> You received this bug notification because you are subscribed to Sikuli.
> https://bugs.launchpad.net/bugs/673995
>
> Title:
> [request] Transparent pixels should be ignored in matching process
>
> Status in Sikuli:
> New
>
> Bug description:
> I'm using Sikuli 0.10.2 on a 32-bit Windows XP system. I just noticed
> that if you are using a png file, containing transparent pixels as a
> pattern for image recognition, these pixels are not skipped during the
> matching process, thus resulting in a wrong match if you have some
> kind of background under the transparent part of the image. I think
> this behavior is incorrect - and Sikuli could be even more powerful if
> able to skip some pixels from the pattern.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/sikuli/+bug/673995/+subscriptions

[RaiMan (raimund-hocke)]

The above test (https://bugs.launchpad.net/sikuli/+bug/673995/comments/5) was done on Mac OS X 10.7.3 with Sikuli X 1.0rc3 - r930.

[Laurent Vaucher (laurent-vaucher)]

Like you, I don't know yet how matchTemplate operates on transparent pixels. I still have to dive deeper into the code. But it's possible that it ignores the alpha layer entirely. It would explain that transparent white could be a good fit for light background. If you try with transparent black, it will probably be better on dark backgrounds.

Anyway, I'll still investigate the OpenCV code, which is interesting in its own right.

Thanks.

[user308 (user308)]

Hi RaiMan,

BenedicteLC has developed a alpha-dependent template matching function for OpenCV (https://github.com/BenedicteLC/CrossCorrelation/blob/master/crosscorrelation.cpp). It is able to ignore transparent pixels during template matching process. I think it can solve this problem. Can you add this function to Sikuli? Thanks.

[Josh Graham (hoi-hoi87)]

This feature is critical for my project. Everything I need to verify is in a frame with an image inside.

[RaiMan (raimund-hocke)]

@Josh
I do not understand, what this has to do with transparency.
Could you give more details.

[Josh Graham (hoi-hoi87)]

Hi RaiMan,
The image I am trying to capture has a thin circular frame around it. Each image in the frame is different. If I want to count the number of occurrences of the frames, it is difficult because the image inside the frame will be in the image capture. If I could make the entire middle of the frame and outside of the frame transparent or ignored in matching, it would be much easier to get the frames.

[Brian Cullinan (bjcullinan)]

This is critical for my project as well. Our use is visual verification with a Swift project in XCode running alongside the Simulator. We have a light translucent image on a dark background, and resolution changes within Simulator using the window scale and using different devices. It would be great if Sikuli could help automate iOS Apps. Is there a way we can elect an algorithm to be used on screen for matching images with different properties? (transparency, resolution, etc)

[Brian Cullinan (bjcullinan)]

I meant to add, after some additional research, I can also confirm that transparent pngs don't match when applied to dark backgrounds. Using the "Matching Preview" window in SikuliX IDE.

[Paul Simpson (wierdbeard65)]

Hi, Any news on this?

My use case actually requires masking out areas inside the image...

One of the tasks I need to perform is to verify that the screenshots taken of a product for training do not need updating when the version changes. So, assume I have a dialog box which has some (pre-filled) information in it. The information is irrelevant to the training guide, but the overall look of the dialog is what we need to show. If I have a png created of the box, but with all the fields replaced by transparent areas, then I should be able to use Sikuli to search for it and have it find the box, even if the data is different.

In this way, I'm able to verify that the dialog box hasn't had extra fields added, or controls moved around etc.

For my purposes, speed is not a major issue (I know you mentioned elsewhere that ignoring transparencies would be slow.)

Thank you!

[RaiMan (raimund-hocke)]

will be a feature in version 2.

[akazen GmbH (akazengmbh)]

Its not working correctly in 2.0.5.
I already posted some more examples on Bitbucket but the threat got stuck. The thing is, that Sikulix reacts completly different if there is only one pixel either transparent or a true color. If you are interesed in futher explanation I may send you an email with screenshots to explain in clear.