[1.0.1] Mac 10.7: Java dragDrop not working with game app

[Fabrizio Borgia]

Sikuli version: 1.0.1 (released 2013-08-12)
Operative System: Mac OS X 10.7.5 (64bit)

Hi, i have developed an automatic farming tool for a game named "Battle Nations". I have been using the Sikuli IDE, but recently, i have switched to Java, because I am a Java developer, and I wanted to add more complexity to my project.

The problem is that the dragDrop function is not working, within the BattleNations window and MORE IMPORTANTLY within any Finder window.
I tried two different approaches to achieve the drag n'drop:

1) Using dragDrop (the click is used to ensure the focus on the window).

     Screen theScreen = new Screen();
     theScreen.wait(3.5);
     theScreen.click(theScreen.getCenter());
     theScreen.dragDrop(theScreen.getCenter(), theScreen.getCenter().offset(0, -100));

2) Using mouse events

     theScreen.wait(3.5);
     theScreen.click(theScreen.getCenter());
     theScreen.mouseDown(Button.LEFT);
     theScreen.wait(0.5);
     theScreen.hover(theScreen.getCenter().offset(0, -100));
     theScreen.wait(0.5);
     theScreen.mouseUp(Button.LEFT);

Both approaches work on any window (browsers, NetBeans itself) without any problem. But when it comes to my BattleNations window, or the a Finder window, the drag is not performed: the mose moves correctly from a location to the other. But nothing changes in the user interface (as if the mouse is not pressed anymore).

The funny thing is that the dragDrop works perfectly using the Sikuli IDE, even with BattleNations!
Can someone help?

I also read the bug report at
https://bugs.launchpad.net/sikuli/+bug/886632
but it did not solve my problem.

[Fabrizio Borgia (fabrizio-borgia)]

A few more details:

1) Is it normal to have the following message on the FIRST time that i run my project from command line (so not on Netbeans)?

[error] ResourceLoaderBasic: check: No valid libs path available until now!
[action] ResourceLoaderBasic: check: Please wait! Trying to extract libs to jar parent folder: /Users/fabrizioborgia/NetBeansProjects/SikuliTest/dist/lib/

Immediately after this, the logging of my appliocation starts. Could it be of any help? Other Sikuli functionalities are working, so I guess everyithing is ok, right?

2) Ok, this is REALLY strange, i tried using Region instead of Screen:

        App.focus("BattleNations");
        Region theWindow = App.focusedWindow();
        theWindow.click(theWindow.getCenter());

        theWindow.mouseDown(Button.LEFT);
        theWindow.wait(4.5);
        theWindow.hover(theWindow.getCenter().offset(0, -100));
        theWindow.mouseUp(Button.LEFT);

and it's not working, AND, if i try to manually (physically) move my mouse during the 4.5 seconds follwing the mouseDown, i see no scroll in the interface. I should see it if the mouse would be really down, right?

3) Please find attached my project configuration and the project directory structure (i hope i put everything in the right place)

[RaiMan (raimund-hocke)]

--- Usage in Netbeans not as recommended (nothing to do with the problem)
It is recommended, that you do not move around copies of the Sikuli stuff. In your Netbeans project, you should use references to the sikuli jar, that resides in the setup folder.
This would eliminate the error message at the beginning, since the libs folder will be permanent after the first run.
In your case the dist folder is rebuilt from time to time and hence the libs folder has to be created newly and the libs exported from the jar.

-- Region or Screen does not matter
all mouse, keyboard, search, ... functions are Region functions and inherited by Screen objects (Screen extends Region).
furthermore all preliminaries with a click, hover or other high level mouse actions finally result in a point on a physical screen, that is used by the respective series of mouse functions processed by the Java AWT Robot deep down internally.

-- does not work on Finder windows
I made this test on my OSX 10.7.5 and had no problem moving the window 500 to the right:
  App.focus("Finder");
  s.wait(1f);
  s.click(new Location(200,50));
  s.dragDrop(s.atMouse(), s.atMouse().right(500));

--- you might try:
    Screen s = new Screen();
     App.focus("BattleNations");
     App.focusedWindow().click(); // should click the centre of the window
     s.wait(0.,5)
     s.mouseDown(Button.LEFT);
     s.wait(0.5);
     s.hover(s.atMouse().above(100));
     s.mouseUp();

might be that your GUI interprets the mouseDown() immediately after the click as part of a doubleClick.

[RaiMan (raimund-hocke)]

BTW: I removed your bug link to the question from 2011, since this is no longer relevant since RC3.

[Fabrizio Borgia (fabrizio-borgia)]

------- Libray issue: SOLVED.
I solved the console message, as you suggested, I installed Sikuli on a single directory.
Now all my Sikuli Projects aro pointing to it. Thanks!

------- Dragdrop issue: UNSOLVED.
Ok, something is wrong, definitely.
I tried both you code chunks, with no effects. The windows are correctly focused, the movement is correctly performed, but no effect is produced on the GUI (the Finder window, the BN window).

------- Further details - JAVA VERSION?
I remember that when I switched from Sikuli IDE to Netbeans, and I installed the JDK (because I didn't have it on my machine) all the drag n' drop functions stopped working, even in the Sikuli IDE.
I am sure of this, because i removed my java installation, and the drag n' drop returned just fine.
I was wondering, which java versione are you using? Here is mine.
       java version "1.7.0_45"
      Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
      Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

------- Further details - REGION VS SCREEN?
Here comes the strangest part: if use this code, the drag n'drop works on the Finder, but not on BattleNations.

        App.focus("Finder");
        Region theWindow = App.focusedWindow();

        theWindow.click(theWindow.getCenter());
        theWindow.wait(1.0);

        theWindow.mouseDown(Button.LEFT);
        theWindow.wait(2.0);
        theWindow.hover(theWindow.getCenter().offset(0, -100));
        theWindow.mouseUp();

[Fabrizio Borgia (fabrizio-borgia)]

Even the appearence of Sikuli IDE is slightly changed (fint size, java icon, etc.) after the instllation of the JDK.

[RaiMan (raimund-hocke)]

------- Libray issue: SOLVED.
one more thing: you should repeat the setup and select options 1 and 3.
3 gives you a much smaller sikuli-java.jar, that is intended to be used in pure Java programming or Jython programming with a standalone Jython or in other IDEs.

------- Further details - JAVA VERSION?
I am running with jdk1.7.0_45.jdk too.
If you want to have a look at your Java's you have, in a Terminal:
/usr/libexec/java_home -V
lists all available Java installations on your machine

Depending on how you installed Netbeans (latest version 7.4 ?), you might run different Java inside Netbeans.

Another thing: you should set your source version to 1.6 (I am developing with Java 7, but found out, that this setting makes it more compatible)

-- mouse not working
I made some more tests and found out, that the only necessary wait (0.5 was sufficient) is between the click and mouseDown. dragDrop does not need such a wait, because this is done internally before mouseDown.
All mouse actions should work in IDE and in Java on every normal Mac app.

Since I cannot check your app here, I guess it is a special problem with it up to the possibility, that it blocks mouse actions coming from a JVM (which in the past some people experienced especially with games).

[Fabrizio Borgia (fabrizio-borgia)]

------- Libray issue: SOLVED.
Thanks for the tip, i did as you suggested. I also checked out for the drag 'n drop issue, but it'still there.
Now, in my Sikuli setup folder, i have also the Sikuli IDE app, i don't think I'll use it for now, should i leave it there/remove it/put in applications?

------- Further details - JAVA VERSION.
Here are all the available JAVA installations on my system.

Matching Java Virtual Machines (3):
    1.7.0_45, x86_64: "Java SE 7" /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home
    1.6.0_65-b14-462, x86_64: "Java SE 6" /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
    1.6.0_65-b14-462, i386: "Java SE 6" /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home

/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home

The Netbeans version is the 7.4, installed a week ago.
I'll change the source versione and let you know!

I don't thnk it's a game issue, since the drag drop worked bedore (and still works on another machine with only the sikuli IDE installed).

[Fabrizio Borgia (fabrizio-borgia)]

SOLVED!!

I added a new platform on Netbeans, targeting the JDK 1.6.
Any drag n' drop operation works now, even on BattleNations.

There must be some incomplatibility there, in JDK 1.7, which prevents the drag 'n drop operations from working correctly with some applications.

Thanks RaiMan for your patience and competence!!!
If you require any other info about this issue, I'll be glad to provide it to you!

[RaiMan (raimund-hocke)]

Fine! and thanks for the feedback.

The Mac app Sikuli-IDE is only for convenience and users who are not used to the usage of a command line.
It does no harm, if you leave it where it is ;-) but if you want to use it, you must copy/move it to /Applications.

That your problem is solved with a Java 1.6 in Netbeans is rather astonishing: I have exactly the same stuff on my 10.7 machine, using source/target version 1.6 and as platform 1.7.
But I admit, that I might not have tested with such a critical application.
Thanks. I do not need more information for that case.

I set this bug to confirmed, to leave it visible, but out of my way ;-)