[request] Transparent pixels should be ignored in matching process

Reported by Vladimir Petrov on 2010-11-11
62
This bug affects 11 people
Affects Status Importance Assigned to Milestone
Sikuli
Wishlist
RaiMan

Bug Description

I'm using Sikuli 0.10.2 on a 32-bit Windows XP system. I just noticed that if you are using a png file, containing transparent pixels as a pattern for image recognition, these pixels are not skipped during the matching process, thus resulting in a wrong match if you have some kind of background under the transparent part of the image. I think this behavior is incorrect - and Sikuli could be even more powerful if able to skip some pixels from the pattern.

Human (bob-igo) wrote :

I would like to make the opposite request, that the current behavior remains. Editing the captured PNGs to contain transparent regions is the easiest way to interact with GUI elements that change according to time or date.

For example, a calendar popup is launched by clicking on the current date. It's not always feasible to match nearby landmarks and use offsets to get to where the current date is displayed. Removing the date section of the image and making it transparent will cause it to match _any_ date.

RaiMan (raimund-hocke) on 2010-11-20
summary: - Transparent pixels should be skipped in image recognition
+ [Request] Transparent pixels should be ignored in matching process
RaiMan (raimund-hocke) wrote :

Both requests are the same:
transparent parts of an image should be ignored during the image search (in the first case the outer parts in the second some inner parts of the reference image).

This is a special case for an option to define some masking (in this case use transparency), to exclude parts of an image from the matching process.

In the second case transparency would not be needed, if it was possible, to have an option for the Pattern object like
Pattern(image).exclude(offset, region)
where offset is the upper left offset of region inside the image, that should be ignored.
This could even be supported in the Preview window like TargetOffset.
I am not sure, wether it makes sense, to have more than one of these exclude-regions per image.

additional:
Pattern.getExclude()
Pattern.setExclude()

to dynamically adjust the exclude regions.

Changed in sikuli:
importance: Undecided → Wishlist
RaiMan (raimund-hocke) on 2011-01-12
summary: - [Request] Transparent pixels should be ignored in matching process
+ [request] Transparent pixels should be ignored in matching process
Denis K (dnsk) wrote :

This is a really useful feature and this request is one year old!! Do we have a chance to get this feature?

If I was interested in trying to implement that feature, where should I start?
I've seen that the matching process relies on OpenCV function 'matchTemplate', but from its documentation I'm not even able to tell if there is a way to exclude some pixels.

  • test1.png Edit (7.5 KiB, image/png; x-unix-mode=0644; name=test1.png)

Hi Laurent,

I just made some tests and found out, that transparency works on backgrounds that are not too dark.

I created a png image of the Sikuli logo from the launchpad page and made the surrounding white pixels transparent (i have it attached) (done with Pixelmator on my Mac ;-).

On the white background it is found without any problems. Even background with some colored structure is not a problem, as long as it is not too dark.
Might be, that this is because (I am not sure so) internally the images are turned to grayscale under normal conditions (finder.cpp -> TemplateFinder::find()). I will later look into the code again.
Furthermore, I have no idea, how the transparent pixels are treated in The OpenCV.matchTemplate() with the used method CV_TM_CCOEFF_NORMED.

Nevertheless I think it is worth some testing.

BTW: I am not one of the developers ;-)

Raimund

Am 06.02.2012 um 16:39 schrieb Laurent Vaucher:

> If I was interested in trying to implement that feature, where should I start?
> I've seen that the matching process relies on OpenCV function 'matchTemplate', but from its documentation I'm not even able to tell if there is a way to exclude some pixels.
>
> --
> You received this bug notification because you are subscribed to Sikuli.
> https://bugs.launchpad.net/bugs/673995
>
> Title:
> [request] Transparent pixels should be ignored in matching process
>
> Status in Sikuli:
> New
>
> Bug description:
> I'm using Sikuli 0.10.2 on a 32-bit Windows XP system. I just noticed
> that if you are using a png file, containing transparent pixels as a
> pattern for image recognition, these pixels are not skipped during the
> matching process, thus resulting in a wrong match if you have some
> kind of background under the transparent part of the image. I think
> this behavior is incorrect - and Sikuli could be even more powerful if
> able to skip some pixels from the pattern.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/sikuli/+bug/673995/+subscriptions

RaiMan (raimund-hocke) wrote :

The above test (https://bugs.launchpad.net/sikuli/+bug/673995/comments/5) was done on Mac OS X 10.7.3 with Sikuli X 1.0rc3 - r930.

Like you, I don't know yet how matchTemplate operates on transparent pixels. I still have to dive deeper into the code. But it's possible that it ignores the alpha layer entirely. It would explain that transparent white could be a good fit for light background. If you try with transparent black, it will probably be better on dark backgrounds.

Anyway, I'll still investigate the OpenCV code, which is interesting in its own right.

Thanks.

RaiMan (raimund-hocke) on 2013-02-22
Changed in sikuli:
status: New → In Progress
assignee: nobody → RaiMan (raimund-hocke)
tags: added: fkt-region
removed: background transparency transparent
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments