unity8.indicators.tests.test_indicators.IndicatorExistsTestCase.test_indicator_exists hangs on mako #245 #246

[Andreas Pokorny]

Steps to reproduce - take

* flash ubuntu-touch/devel-proposed reviosion 245 or later to mako
* configure pin, developer mode, writable-image, network, ..
* sudo apt-get install unity8-autopilot
* sudo powerd-cli display on bright
* autopilot run -v unity8-autopilot unity8

tests seem to run fine up to this point:

07:15:01.082 DEBUG __init__:305 - Unity started, waiting for it to be ready.
07:15:01.088 DEBUG dbus:433 - Selecting objects of type QQuickView with attributes: {}
07:15:01.185 DEBUG dbus:433 - Selecting objects of any type with attributes: {'objectName': 'greeterContentLoader'}
07:15:02.658 DEBUG __init__:307 - Unity loaded and ready.
07:15:02.659 DEBUG dbus:433 - Selecting objects of type QQuickView with attributes: {}
07:15:02.730 DEBUG dbus:433 - Selecting objects of type Greeter with attributes: {}
07:15:04.068 INFO process_helpers:95 - Greeter unlocked, continuing.
07:15:04.069 DEBUG dbus:433 - Selecting objects of type QQuickView with attributes: {}
07:15:04.166 DEBUG dbus:433 - Selecting objects of type DefaultIndicatorWidget with attributes: {'objectName': 'indicator-power-widget'}
07:15:05.095 INFO __init__:330 - Stopping unity
07:15:06.262 INFO __init__:246 - Resetting upstart env QT_LOAD_TESTABILITY to None
07:15:06.432 INFO testresult:46 - OK: unity8.indicators.tests.test_indicators.IndicatorExistsTestCase.test_indicator_exists(Power,Native Device)
07:15:06.436 INFO globals:58 - ************************************************************
07:15:06.436 INFO globals:59 - Starting test unity8.indicators.tests.test_indicators.IndicatorExistsTestCase.test_indicator_exists (Sound,Native Device)
07:15:06.438 WARNING testcase:116 - No tracing available - install the python-autopilot-trace package!
07:15:06.443 WARNING testcase:157 - Process manager backend unavailable, application snapshot support disabled.
07:15:06.452 WARNING utilities:265 - WARNING: in file "/usr/lib/python2.7/dist-packages/unity8/shell/tests/__init__.py", line 164 in setUp
This function is deprecated. Please use 'the Touch class to instantiate a device object' instead.

07:15:06.727 INFO __init__:268 - Lib path is '/usr/lib/arm-linux-gnueabihf/unity8', binary path is '/usr/bin/unity8'
07:15:06.729 INFO __init__:346 - Setting up LightDM mock type 'single'
07:15:06.837 INFO __init__:355 - New library path: /usr/lib/arm-linux-gnueabihf/unity8/qml/mocks/libusermetrics:/usr/lib/arm-linux-gnueabihf/unity8/qml/mocks/LightDM/single
07:15:06.884 INFO __init__:108 - New QML2 import path: /usr/lib/arm-linux-gnueabihf/unity8/qml/mocks
07:15:06.951 INFO __init__:317 - Starting unity
07:15:07.047 INFO process_helpers:151 - Starting job unity8 with arguments ('BINARY=/usr/bin/unity8', 'ARGS=', 'XDG_DATA_DIRS=/usr/share/unity8/mocks/data:/usr/share/ubuntu-touch:/usr/local/share:/usr/share', 'LD_LIBRARY_PATH=/usr/lib/arm-linux-gnueabihf/unity8/qml/mocks/libusermetrics:/usr/lib/arm-linux-gnueabihf/unity8/qml/mocks/LightDM/single', 'QML2_IMPORT_PATH=/usr/lib/arm-linux-gnueabihf/unity8/qml/mocks', 'QT_LOAD_TESTABILITY=1').
07:15:07.679 INFO process_helpers:159 - unity8 start/running, process 7450

07:15:33.540 ERROR proxies:410 - Introspect error on :1.391:/com/canonical/Autopilot/Introspection: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
07:15:33.547 DEBUG proxies:413 - Executing introspect queue due to error
07:16:25.189 ERROR proxies:410 - Introspect error on :1.413:/com/canonical/Autopilot/Introspection: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
07:16:25.195 DEBUG proxies:413 - Executing introspect queue due to error
07:17:15.309 ERROR proxies:410 - Introspect error on :1.391:/com/canonical/Autopilot/Introspection: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
07:17:15.311 DEBUG proxies:413 - Executing introspect queue due to error
07:18:05.418 ERROR proxies:410 - Introspect error on :1.391:/com/canonical/Autopilot/Introspection: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
07:18:05.419 DEBUG proxies:413 - Executing introspect queue due to error
07:18:55.558 ERROR proxies:410 - Introspect error on :1.391:/com/canonical/Autopilot/Introspection: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
07:18:55.561 DEBUG proxies:413 - Executing introspect queue due to error
07:19:45.859 ERROR proxies:410 - Introspect error on :1.391:/com/canonical/Autopilot/Introspection: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
...

[kevin gunn (kgunn72)]

so greyback had investigated too, understanding we chatted with jhodapp and thinks it's in media-hub or lower.
marking this as critical b/c this is blocking people from running unity8-ap tests locally to ok landings.

[Gerry Boland (gerboland)]

Introspect error on :1.391:/com/canonical/Autopilot/Introspection: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply.

This nearly always means the application under test has either crashed or has hung.

Digging, I found that indeed unity8 is handing. Backtrace points to Qt's GUI thread being blocked on the android side. I made a guess and commented out the Audio{} component in qml/Notifications/Notifications.qml and found the hang disappeared.

The final lines of output from unity8 also lend weight to that guess:

  Initializing AalServicePlugin
  Creating a new static Service instance
<hang>

I tried to grab a backtrace, but this is all I managed and I suspect the bionic bits are just wrong:
http://pastebin.ubuntu.com/8380144/
Thread1 is Qt's GUI thread which is blocked. Frame3 of that backtrace not in the address space. Also why is MD5 being calculated? So I'm confused! Probable my build of bionic libc not matching that on device.

[kevin gunn (kgunn72)]

worth noting, this was working last week

[Alexander Sack (asac)]

can you nail down a regression window by image and maybe even package? This would help finding the cause I guess...

[Alexander Sack (asac)]

if there was a new media-hub landing, then guess that might be a good candidate to try backing out locally and see if that helps.

[Oliver Grawert (ogra)]

we see a similar error in utopic smoke tests since a while, these do not show up in rtm image tests at all. paul nailed down the first image where it started to show up in smoke testing to http://people.canonical.com/~ogra/touch-image-stats/233.changes ...

the adbd and system-settings packages on that list are also in rtm which makes them very unlikely to be the cause.

this leaves pcap, gettext and gstreamer ... i wonder if downgrading one of them to their former version might help.

[Jim Hodapp (jhodapp)]

GStreamer is obviously very heavily used by media-hub, so I'd recommend trying to downgrade gstreamer first.

[kevin gunn (kgunn72)]

ok, so for devel-proposed. I've at least determined this happened between image #235 (where AP would run to completion) & image #237 (where is hangs). Worth noting, #136 impossible to work with, can't get adb to recognize the device no matter what.

[kevin gunn (kgunn72)]

ok this is definitely media-hub
I started with #235
running unity8 AP each time....
upgraded/installed... one at a time

1) libmedia-hub-client2 & libmedia-hub-common2 - passed
2) libdbus-cpp4 - passed
3) media-hub - failed....got stuck

as the packages are not maintained for the intermediate builds this was using the latest packages in archive but the failure was the exact same

[Oliver Grawert (ogra)]

did anyone try to downgrade to the former gstreamer1.0-plugins-base version yet ?

[kevin gunn (kgunn72)]

Ok, downloaded and manually installed the deb from here for gstreamer
https://launchpad.net/ubuntu/utopic/armhf/libgstreamer1.0-0/1.4.0-1

Did this on today's devel-proposed image on N4, and unity8 still hung.

[kevin gunn (kgunn72)]

or rather, unity8 AP test still hung eventually using https://launchpad.net/ubuntu/utopic/armhf/libgstreamer1.0-0/1.4.0-1

[kevin gunn (kgunn72)]

so i tried again, only this time i downgraded gstreamer a bit further to
https://launchpad.net/ubuntu/utopic/armhf/libgstreamer1.0-0/1.2.4-1

and this time unity8 AP testing passed

[Jim Hodapp (jhodapp)]

So it seems it is the core GStreamer library upgrade that broke it then. We'll need to look at the diff between the last version we had and this one to try to narrow down what broke things.

[Alberto Aguirre (albaguirre)]

@Gerry, you are right, it's hanging in android. It looks like it's waiting for the binder response to a registerSession call initiated by decoding_service_create_session. I wonder why there's no response....

http://pastebin.ubuntu.com/8421608/

[Alberto Aguirre (albaguirre)]

It looks like media-hub exhausted all the binder threads, they are all stuck in core::Signal<>::~Signal() called from
core::ubuntu::media::PlayerImplementation::~PlayerImplementation()

http://pastebin.ubuntu.com/8421674/

[Alberto Aguirre (albaguirre)]

Ok so the issue is the ~Signal() is being called from withing the Signal executor...so recursively acquiring the same lock.

An async execution of the code that ends up deleting the PlayerImplentation solves that issue and avoids hangs in the unity8 autopilot tests.

There's still an issue where ocassionally ~PlayerStub (during unity8 stop) hangs waiting for the dbus worker thread to finish; however that doesn't cause the autopilot tests to hang as the unity8 process is eventually sigkilled if that's the case.

[Alberto Aguirre (albaguirre)]

A stress test for the issue:
http://pastebin.ubuntu.com/8423451