[1.1.4] Windows 10: Move mouse for click not working since new JRE and Windows update with Java 8 and HiDPI --- workaround Java 9 and scaling 100% OR try with Java 11

Bug #1730645 reported by RaiMan on 2017-11-07
42
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Sikuli
Medium
RaiMan

Bug Description

********************* Java problem
seems to be related to a problem with HiDPI and scaled monitor setups
https://bugs.openjdk.java.net/browse/JDK-8196030

The problem seems to exist with Java 8 not supporting HiDPI.

Using Java 9 or 10 on HiDPI systems with a scaling of 100% seems to work

Using Java 11 these experiences are reported:
early build of jdk 11 and sikuli 1.1.4 is working good for hidpi, but still we observed few issues mentioned below:

1) Sikuli scripts written using 4k (3840*2160) with recommended scaling (300%) is working only for the same recommended settings. If we change the resolution to 2K or 1080P or below , and also if we change scaling, scripts are failing

2) Sikuli scripts written on 2k (2560*1600) with recomended scaling (200%) is working for for resolutions below 2k like 800*600 to 1920*1080 (Downscaling is working with this) and for 4k it is not working (Upscaling is not working)
-------------------------------------------------

wondering if anyone else is seeing this.

I install the latest big update for Windows this morning and also accepted an update to the latest JRE.

Now my when Sikuli moves the mouse to click on a match it doesn't land on the coordinates. It even recognizes that it missed and has an error message saying it tried to move the mouse to (x, y), but landed on (p, q) instead.

Any idea what is happening and how to fix it?

Thanks!

RaiMan (raimund-hocke) on 2017-11-07
Changed in sikuli:
status: New → Opinion
assignee: nobody → RaiMan (raimund-hocke)
milestone: none → 1.1.2
RaiMan (raimund-hocke) wrote :

attached this is a test jar

usage:
java -jar MoveMouseAround.jar loops delay

loops and delay are optional integers (positional parameters):

loops: how often should the mouse be moved randomly inside the area (margin of 200 all sides inside screen), default: 5

delay: how many milliseconds should java.awt.Robot wait after a move, default 20 (max 60)

the output is something like this:
[MoveMouseAround] Using: loops = 5; Robot.autoDelay = 20
[MoveMouseAround] GraphicsEnvironment: 1 ScreenDevices
[MoveMouseAround] Screen: (0,0 1920x1080)
[MoveMouseAround] using area: (200,200 1520x680)
[MoveMouseAround] *** visitCorners of (200,200 1520x680)
[MoveMouseAround] Moved to: (200,200)
[MoveMouseAround] Moved to: (1720,200)
[MoveMouseAround] Moved to: (1720,880)
[MoveMouseAround] Moved to: (200,880)
[MoveMouseAround] Moved to: (200,200)
[MoveMouseAround] *** movingAround in (200,200 1520x680) for 5 times
[MoveMouseAround] Moved to: (1225,347)
[MoveMouseAround] Moved to: (741,717)
[MoveMouseAround] Moved to: (1714,546)
[MoveMouseAround] Moved to: (1670,272)
[MoveMouseAround] Moved to: (1656,424)

if a move is not correct:
[MoveMouseAround] Moved to: (...,...) but is: (..., ...)

RaiMan (raimund-hocke) wrote :
RaiMan (raimund-hocke) wrote :

... and here the source if someone is interested.

RaiMan (raimund-hocke) on 2017-11-10
Changed in sikuli:
status: Opinion → In Progress
importance: Undecided → Medium
RaiMan (raimund-hocke) on 2018-02-05
description: updated
RaiMan (raimund-hocke) on 2018-02-05
summary: [1.1.1] Windows 10: Move mouse for click not working since new JRE and
- Windows update
+ Windows update with Java 8 and HiDPI
summary: [1.1.1] Windows 10: Move mouse for click not working since new JRE and
- Windows update with Java 8 and HiDPI
+ Windows update with Java 8 and HiDPI --- workaround Java 9 and scaling
+ 100%

Thanks RaiMan. I'll give this a try. I suspected there might be something in the scaling. I fiddle with it a bit, but no success.

RaiMan (raimund-hocke) on 2018-02-05
description: updated
RaiMan (raimund-hocke) on 2018-03-27
Changed in sikuli:
milestone: 1.1.2 → 1.1.3
sdraganov (sdraganov) wrote :

Is there workaround for this bug?
Can't find Java 9 at www.oracle.com - should I download it from third party website?

RaiMan (raimund-hocke) wrote :

@sdraganov: Try with Java 10

sdraganov (sdraganov) wrote :

I tried that already but it gives me the following error when running SikuliX 1.1.2
https://pastebin.com/dWerkpTa

Will try with the nightly 1.1.3 version.

sdraganov (sdraganov) wrote :

Seems to work fine with 1.1.3, at least with very simple script, just a single click command.
Will do more tests and will keep you posted in case something goes wrong.

RaiMan (raimund-hocke) on 2018-09-03
summary: - [1.1.1] Windows 10: Move mouse for click not working since new JRE and
+ [1.1.4] Windows 10: Move mouse for click not working since new JRE and
Windows update with Java 8 and HiDPI --- workaround Java 9 and scaling
100%
Changed in sikuli:
milestone: 1.1.3 → 1.1.4
RaiMan (raimund-hocke) on 2018-09-10
summary: [1.1.4] Windows 10: Move mouse for click not working since new JRE and
Windows update with Java 8 and HiDPI --- workaround Java 9 and scaling
- 100%
+ 100% OR try with Java 10

Hi @Raiman,

we are using sikuli 1.1.3 with java 9.0.4 and jython 2.7.1...Sikuli is unable to detect images in 4k resolution even when scripted on same 4k panel..what else i need to try?

RaiMan (raimund-hocke) wrote :

as mentioned in the bug-title:
try with Java 10.

I am sorry, I cannot help you:
- I do not have a 4K display (Mac Retina works)
- this is definitely a Java problem, as you can see in the net looking for "Java hiDPI"

RaiMan (raimund-hocke) on 2018-09-25
summary: [1.1.4] Windows 10: Move mouse for click not working since new JRE and
Windows update with Java 8 and HiDPI --- workaround Java 9 and scaling
- 100% OR try with Java 10
+ 100% OR try with Java 11
description: updated
Michelle (mnovabox) wrote :

Hi @Raiman,

So I have the same error:

[error] RobotDesktop: checkMousePosition: should be L(3520,540)@S(1)[2560,0 1920x1080]
but after move is L(3504,557)@S(1)[2560,0 1920x1080]
Possible cause in case you did not touch the mouse while script was running:
 Mouse actions are blocked generally or by the frontmost application.
You might try to run the SikuliX stuff as admin.

Now I switched to java 11.
Now I get the following error:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.python.core.PySystemState (file:/C:/jython2.7.0/jython.jar) to method java.io.Console.encoding()
WARNING: Please consider reporting this to the maintainers of org.python.core.PySystemState
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

This is a known problem for Java 9/10/11 and the developers of jython don't know how to fix it:
https://stackoverflow.com/questions/46230413/jdk9-an-illegal-reflective-access-operation-has-occurred-org-python-core-pysys

How did you bypass this?

RaiMan (raimund-hocke) wrote :

@Michelle
your question is not related to this bug topic !

see bug 1749262

Michelle (mnovabox) wrote :

ok thanks

omearam (mike-omeara) wrote :

Ok using W10, Java 11 & IDE 1.1.3 and it has fixed the issue but seeing this IDE output on startup, does seem to affect IDE behaviour in general.

C:\WINDOWS\system32>c:\temp\silkuli\runsikulix.cmd
+++ running this Java
java version "11.0.1" 2018-10-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.1+13-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.1+13-LTS, mixed mode)
+++ trying to run SikuliX
+++ using: -Xms64M -Xmx512M -Dfile.encoding=UTF-8 -Dsikuli.FromCommandLine -jar c:\temp\silkuli\sikulix.jar
[error] RunTimeINIT: *** BE AWARE: Running on Java 8+ *** Please report problems ***
[info] HotkeyManager: add Capture Hotkey: NUM LOCK (144, 0)
[info] HotkeyManager: add Abort Hotkey: ALT+SHIFT C (67, 9)
Exception in thread "Thread-2" java.lang.UnsatisfiedLinkError: C:\Users\omearam\AppData\Local\Temp\jna--1294165632\jna12381214207587878606.dll: Can't find dependent libraries
        at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
        at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2430)
        at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2487)
        at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2684)
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2617)
        at java.base/java.lang.Runtime.load0(Runtime.java:767)
        at java.base/java.lang.System.load(System.java:1831)
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851)
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826)
        at com.sun.jna.Native.<clinit>(Native.java:140)
        at com.sun.jna.Pointer.<clinit>(Pointer.java:41)
        at com.sun.jna.Structure.<clinit>(Structure.java:2078)
        at com.tulskiy.keymaster.windows.WindowsProvider$1.run(WindowsProvider.java:55)
        at java.base/java.lang.Thread.run(Thread.java:834)
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.python.core.PySystemState (file:/C:/temp/silkuli/sikulix.jar) to method java.io.Console.encoding()
WARNING: Please consider reporting this to the maintainers of org.python.core.PySystemState
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

To post a comment you must log in.