[1.1.4] Windows: Android: Screenshot problem - lower part of screen image is black (varying) --- fixed 2018-12-12 build #145

[Lin U Min]

Sikuli versions: 1.1.4
OS: Windows 10 X64
IDE: Jython IDE

I'm using adb to connect my emulator.
Using menuToolAndroid to test, there're things failed.
1.Clicking "Check" Button:
The first and second check was success.
In the third check:
a.The capture icon feature sometimes didn't pop out and cause the IDE didn't return.
b.Although the capture icon feature pop out, Sikuli didn't click the icon I've captured and the IDE didn't return.

2.clicking "Default" Button:
After clicking default,clicking the "Take a screenshot" Button on IDE doesn't pop out the screenshot feature and the IDE disappeared.Only I can do is to close java with task manager and restart the IDE.

3.In /API/src/main/java/org/sikuli/android/ADBScreen.java line 106,method

  public List<ADBDevice> getDevices() {
    List<ADBDevice> devices = new ArrayList<>();
    if (device != null) {
     //<= no code written
    }
    return devices;
  }

which always return empty list.

Also, my test code is:

adbscr = adbStart()
reg = adbscr.newRegion(0, 0, 1600, 900)
img = adbscr.capture(reg)
img.save(path_to_save)

it returns ADBDevice: captureDeviceScreenMat: image type not RGBA

[RaiMan (raimund-hocke)]

--- IDE: Jython IDE
what is that?

--- emulator
what exactly is that?
possible for me to get it on my Windows 10?

--- it returns ADBDevice: captureDeviceScreenMat: image type not RGBA
adbscr.newRegion(0, 0, 1600, 900)
does it mean the whole screen?
if yes try:
adbscr = adbStart()
img = adbscr.capture()

if still error:
adb::executeShell("screencap")
returns different image type than the only expected (RGBA)

You have to check, what your emulator should return in response to this request and wether there are any options for screencap to ensure RGBA

--- getDevices()
currently only one device is supported anyways (though more than one can be attached)
... but since after having started adb successfully, you can select another device using getScreenWithDevice(int id), there should be a function like ADBScreen.getScreens() that lists the attached devices.

--- menuToolAndroid
Should also be aware of more than one device attached.

Generally my environment (only a Nexus7) and lack of Android/adb knowledge limits my possibilities to really help with this "experimental" implementation.

If it helps:
https://github.com/Genymobile/scrcpy

[RaiMan (raimund-hocke)]

BTW: I guess the IDE hangs caused by this non-RGBA problem

[Lin U Min (s1k2y37st)]

--- IDE:what is that?

Sorry for my description It's just the SikulixIDE with Jython language. :)

--- emulator:what exactly is that? possible for me to get it on my Windows 10?

It's an emulator called "ldplayer".(http://www.ldplayer.net/) (Android 5.1)
It works success on Windows 10.

--- it returns ADBDevice: captureDeviceScreenMat: image type not RGBA
adbscr.newRegion(0, 0, 1600, 900)
does it mean the whole screen?
if yes try:
adbscr = adbStart()
img = adbscr.capture()

=> still error

if still error:
adb::executeShell("screencap")
returns different image type than the only expected (RGBA)

You have to check, what your emulator should return in response to this request and wether there are any options for screencap to ensure RGBA

=> Not really sure what to do.But I tried these adb command in my command line:

adb shell screencap -p /sdcard/screen.png
adb pull /sdcard/screen.png

And I checked the screen.png.It's a 32bits-depth image.I think it's RGBA format.

Maybe the "-p" arg is needed.

[Lin U Min (s1k2y37st)]

I also try the screencap command without -p

It's still return a 32bits-depth image.

[RaiMan (raimund-hocke)]

ok, thanks for feedback and tests.

I have my stuff with my Nexus 7 running on my Windows 10 ...
... and everything works as expected.

My next step: get the emulator working.

[RaiMan (raimund-hocke)]

BTW: might be a problem with Android 5.x --- but as mentioned: Android experience very, very bad ;-)

[RaiMan (raimund-hocke)]

Ok, a step further ;-)

I have the emulator working.

At least I have found a problem and fixed it, so the capture() principally works now.

But it does not give access to the whole screen content with varying amount of black pixels towards the bottom.

The test now in the IDE works, if you capture something from the "filled area" towards the top.

I currently do not have any idea and have to test further.

The new build will be available later today.

[RaiMan (raimund-hocke)]

On Windows I have the "black-lower-part" effect with the emulator and my real device.

On Mac it works as it should with my real device.
... but I do not have an emulator for Mac yet

[Lin U Min (s1k2y37st)]

You can try bignox(https://en.bignox.com/) which supports mac.
But I do not have a mac system so I have no idea if the adb works on emulator in mac system.

[RaiMan (raimund-hocke)]

No problem.

I decided to run AndroidStudio on my Win10 to get the official emulator working.
The Android eco-system is much more similar to a "normal" u*ix system like macOS, so I do not have panic to step into the basics in a short time.

I will come back with any insights.

[Lin U Min (s1k2y37st)]

It's weird that I build a same code of captureDeviceScreen(0, 0, 1600, 900) and output the image file.
The image file is perfect. No black part in the lower part.

But not testing with captureDeviceScreen(0, 0, -1, -1).

[RaiMan (raimund-hocke)]

--- same code of captureDeviceScreen(0, 0, 1600, 900) and output the image file.
could you paste the code?

[Lin U Min (s1k2y37st)]

I traced some code from the source for my test.

And I tried to safe the Mat and BufferedImage.

Both of them are success.

[RaiMan (raimund-hocke)]

The capture section is completely revised and should work now (including the IDE menu check).

added goody: now a search on the whole screen (in my case 1200 x 1920) takes less than 3 secs (before more than 6 secs).

[Lin U Min (s1k2y37st)]

Thanks!!!
I saw that and tested it immediately.

A issue:

In the document said "a click gets a tap and type/paste result in an input text".
My Test Code:

Debug.on(3)
adbScr = adbStart()
use(adbScr)

testImg = "aImg.png"

click(testImg) ##not worked
adbScr.click(testImg) ##not worked
aTap(testImg) ##worked
adbScr.aTap(testImg) ##worked

testTypeImg = "aType.png"
type(testTypeImg) ##worked
adbScr.type(testTypeImg) ##worked

Seems like the click() doesn't work?
-------

I will keep on testing!

[RaiMan (raimund-hocke)]

Ok, thanks for immediate feedback.

I will check the click, and fix asap.