Linux: VNC: IDE: Part of a dialog gets black when trying to take a screenshot

[Caio Araújo Teixeira Campos]

When I try to take a screenshot, run sikuli script or use Matching Preview in a specific dialog, part of this dialog gets black.

If I move the dialog this won't happen anymore, so I can still run my script just dragging and dropping my dialog before performing my actions on this dialog.

It is the first time I see it and it only happens with this dialog.

I'm running Sikuli X-1.0rc2 x86_64, on a Red Hat Enterprise Linux Server release 6.1 (Santiago) through a VNC session. (Sikuli installation and my VNC server are on this machine)

I'm not sure how to reproduce this bug since it is the only situation that I can see this happening.

I've already tried in a new VNC session but the same problem happened.

[Caio Araújo Teixeira Campos (caioatc)]

Any news on this?

The problem is proliferating, now two context menus that weren't affect by this before are completely black when I try to capture them or run a Sikuli script.

[RaiMan (raimund-hocke)]

multi monitor environment?

[Caio Araújo Teixeira Campos (caioatc)]

No. (In fact I'm using two monitors but I'm running a Windows 7 64 bit with two monitors and my Sikuli is running inside a VNC session that is a Red Hat Enterprise Linux Server release 6.1 (Santiago) with only one monitor. Anyway I've tried it without the external monitor and nothing changed.)

Since yesterday I'm trying to find the root cause of this bug.

Here are some information that may help:
- The dialog gets only partially black and it starts exactly where some check boxes and spinners start (but in another dialog that have spinners and check boxes this doesn't happens). If I move this dialog it works
- The context menus and pull-down menus are all black, but I've just found that if they have some part of it outside the window, they will be fine
- I'm using Gnome but I've also tried Fluxbox and nothing changed
- The application I'm trying to test with Sikuli is based on Qt 4.6
- If I run Sikuli over a VNC session, i.e. if Sikuli is in a different display that my application there is nothing black

Attached is the yum.log of the machine that I'm running Sikuli, since I noticed this problem after theses updates, maybe there is something that you already know that can be cause of this problem, .

Thanks in advance.

[RaiMan (raimund-hocke)]

If you are currently running the script from within the IDE, you should give this a try:

java -jar path-to-sikuli-script.jar path-to-your-script.sikuli

This does not use the IDE at all.

Furthermore it would be interesting wether the output or behavior of the following makes sense in the faulty situation:

print SCREEN.getBounds()
hover(SCREEN.getBottomRight())
print Env.getMouseLocation()
print find(capture(0,0,300,300))

[Caio Araújo Teixeira Campos (caioatc)]

It still doesn't work.

If I include the lines you've mentioned before the screenshot that is usually black, I get these messages:

[info] Sikuli vision engine loaded.
[info] VDictProxy loaded.
[log] CLICK on (664,84)
[log] RIGHT CLICK on (645,162)
java.awt.Rectangle[x=0,y=0,width=1024,height=768]
(1023,767)
Match[0,0 300x300] score=1.00, target=center

Any idea?

Running this way I don't know if the menu is black or if it is not black and Sikuli can't find it.

[RaiMan (raimund-hocke)]

--- you say: It still doesn't work.

does this mean using
java -jar path-to-sikuli-script.jar path-to-your-script.sikuli ???

--- the test
The screen size is 1024x768, but the mouse does not reach the lower right corner: (1023,767).
The captured upper right 300x300 is found with no problem.

--- you say:
Running this way I don't know if the menu is black or if it is not black and Sikuli can't find it.

pls. clarify what you mean, I do not understand.

--- real life
it would be helpful, to have a screenshot of the scene and the relevant part of the script.
put all together in a zip and attach.

[Caio Araújo Teixeira Campos (caioatc)]

Thanks for your quick responses.

Answers preceded by A:

--- you say: It still doesn't work.

does this mean using
java -jar path-to-sikuli-script.jar path-to-your-script.sikuli ???

A: yes

--- the test
The screen size is 1024x768, but the mouse does not reach the lower right corner: (1023,767).
The captured upper right 300x300 is found with no problem.

A: Doesn't the screen start on 0,0? I thought it was right the mouse reaching 1023,767

--- you say:
Running this way I don't know if the menu is black or if it is not black and Sikuli can't find it.

pls. clarify what you mean, I do not understand.

A: When using Sikuli to screen capture or matching preview I'm certain that Sikuli sees black regions, but using the jar, instead of the IDE, there is no way to know if Sikuli is seeing things black.

--- real life
it would be helpful, to have a screenshot of the scene and the relevant part of the script.
put all together in a zip and attach.

A: To see the problem happening you don't even need a script, just try to take a screenshot and you will see the black thing. I will try to get a screenshot of the scene but I don't have it now since the application I'm testing is confidential and I couldn't find a similar bug yet.

[Caio Araújo Teixeira Campos (caioatc)]

Attaching a screenshot of the problem.

The screenshot was taken when I was trying to capture the target of my click and there is nothing on the script yet.

The green rectangles were added by me and are not present in the real screen.

If I drag this dialog there will be nothing black on the screen.

[RaiMan (raimund-hocke)]

save the following script to somewhere.sikuli

import os
import shutil
dir = getBundlePath()
img = capture()
shutil.mv(img, os.path.join(dir, "captured.png")

and run it from commandline using
java -jar path-to-sikuli-script.jar path-to-your-somewhere.sikuli

after you have arranged the critical screen situation.

You should be asked to capture. Capture the dialog no matter what it looks like. After the script has terminated, you should find the captured image as captured.png in the folder somewhere.sikuli.

Talk about your experiences ;-)

[Caio Araújo Teixeira Campos (caioatc)]

The captured image still having a black part.

Ps. shutil.mv(img, os.path.join(dir, "captured.png") should be shutil.move(img, os.path.join(dir, "captured.png")), right?

[RaiMan (raimund-hocke)]

--- Ps. shutil.mv(img, os.path.join(dir, "captured.png") should be shutil.move(img, os.path.join(dir, "captured.png")), right?
I could say: ok, you passed the the test ;-)
.... but I say simply: Sorry, i mixed it up :-(

--- the problem
ok, it seems to be some problem in the internal handling of the interactive capture feature together with the VNC-situation.
I doubt, that anyone else will come up with a solution, if not yourself.

I cannot test it in any way, since I do not have the systems available and lacking some knowledge in the Linux/VNC area.

The next thing I would try, is to use some other capture tool, to produce images somewhere and load them into the IDE (2nd button from top left) and check, wether the find operations are running correctly.

[Caio Araújo Teixeira Campos (caioatc)]

I wish the capture were the problem, but when I saw this problem for the first time the first thing I tried was capture the image myself and use sikuli just to run the script, but it doesn't work also.

Yesterday I was looking my e-mails and I found that this problem wasn't happening when I was using Sikuli 0.9 on a CentOs 5.5 machine, today I will try Sikuli 0.9 in this Red Hat 6 machine to see if the problem is related to Sikuli or the environment.

Maybe the problem is related to some change that happened between these two versions 0.9 and X1.0rc2.

Any ideia? I never know when I will get bored and want to look some Sikuli code.

[Caio Araújo Teixeira Campos (caioatc)]

Some updates:

- Sikuli 0.10.2 is not affected by this bug, at least not in a CentOS 5.5 machine
- Sikuli X-1.0rc2 x86_64 (running on virtual box and without VNC this time) has a similar bug, the difference is that the dialogs and context menus get transparent instead of black

I'm wondering if this bug is related to Qt, since the application I'm testing is based on it.

If my guess is right, would be a good idea to start looking the changes on code related to screen capture between 0.10.2 and X-1.0rc2 x86_64 that could be sensible to Qt.

[LE GOAS YANN (ylegoas)]

Hello,
Arrived soon here...does this problem gets kind of fixing solution?
THX

[RaiMan (raimund-hocke)]

@ylegoas
Using latest version of SikuliX?
1.1.1 or even better 1.1.2 nightly?

[LE GOAS YANN (ylegoas)]

Hi RaiMan,
1.1.1 used on Ubuntu 16.04LTS (on Vm or real machine)
Post .zip attachement showing open folder before capture and same during capture (blacked area)
THX

[LE GOAS YANN (ylegoas)]

Attachement linked to post #16

[RaiMan (raimund-hocke)]

please try with 1.1.2 nightly

[LE GOAS YANN (ylegoas)]

No change with 1.1.2 on WM
:-(

[RaiMan (raimund-hocke)]

LOL, should have looked at your screenshots :-(

Your "problem" has nothing to do with this question thread: that is about SikuliX using VNC.

You are simply running SikuliX on an Ubuntu machine as an interactive Jython session.
At the Jython level
capture()

starts an interactive capture.
So you have to paint the capture region.

... and the "black" area is part of the feature (the drawing pane).

That the screen image is offset by the left bar is a known glitch caused by the Ubuntu GUI manger.