Mako locks up roughly once a day since R226

[Dave Morley]

Mako screen will randomly lock up roughly once a day since R226 landed. Currently I am on R229

root@ubuntu-phablet:/# dpkg -l |grep mir
ii libmirclient7:armhf 0.1.5+14.04.20140212-0ubuntu1 armhf Display server for Ubuntu - client library
ii libmirplatform:armhf 0.1.5+14.04.20140212-0ubuntu1 armhf Display server for Ubuntu - platform library
ii libmirprotobuf0:armhf 0.1.5+14.04.20140212-0ubuntu1 armhf Display server for Ubuntu - protocol implementation
ii libmirserver15:armhf 0.1.5+14.04.20140212-0ubuntu1 armhf Display server for Ubuntu - server library
ii libubuntu-application-api-mirclient1:armhf 0.20+14.04.20140221-0ubuntu1 armhf Implementation of the Platform API for a Mir client
ii libubuntu-application-api-mirserver1:armhf 0.20+14.04.20140221-0ubuntu1 armhf Implementation of the Platform API for a Mir server
ii libunity-mir1:armhf 0.2+14.04.20140221.1-0ubuntu1 armhf Qt plugins for Unity specific Mir APIs
ii mir-test-tools:armhf 0.1.5+14.04.20140212-0ubuntu1 armhf Display Server for Ubuntu - stress tests and other test tools

root@ubuntu-phablet:/# dpkg -l |grep unity8
ii unity8 7.84+14.04.20140306-0ubuntu1 armhf Unity 8 shell
ii unity8-fake-env:armhf 7.84+14.04.20140306-0ubuntu1 armhf Fake environment for running Unity 8 shell
ii unity8-private:armhf 7.84+14.04.20140306-0ubuntu1 armhf Unity 8 private libs

[Kevin DuBois (kdub)]

Are there steps to reproduce?

This is the only suspicious thing I see in the logs:
W/Adreno-ES20( 2085): <core_glReadPixels:212>: GL_INVALID_OPERATION
W/Adreno-EGLSUB( 2085): <CacheInvalidateHandle:243>: PMEM_INV_CACHES undefined

[Dave Morley (davmor2)]

Kevin

I've not found a way to reproduce it easily hence leving it in the broken state most of today. It just seems to happen, Since Friday it has hapen 3 times, First doning something similar closing an app back to the dash, second swiping from home scope to apps scope and third again open app back to apps scope (long swipe from the left).

[Dave Morley (davmor2)]

davmor2@stryder:~$ adb shell
root@ubuntu-phablet:/# gdb unity8 2085
GNU gdb (Ubuntu 7.7-0ubuntu3) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from unity8...(no debugging symbols found)...done.
Attaching to program: /usr/bin/unity8, process 2085
Reading symbols from /usr/lib/arm-linux-gnueabihf/libQt5Gui.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libQt5Gui.so.5
Reading symbols from /usr/lib/arm-linux-gnueabihf/libQt5Qml.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libQt5Qml.so.5
Reading symbols from /usr/lib/arm-linux-gnueabihf/libQt5Quick.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libQt5Quick.so.5
Reading symbols from /usr/lib/arm-linux-gnueabihf/libQt5Core.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libQt5Core.so.5
Reading symbols from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
Reading symbols from /lib/arm-linux-gnueabihf/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libgcc_s.so.1
Reading symbols from /lib/arm-linux-gnueabihf/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libc.so.6
Reading symbols from /lib/ld-linux-armhf.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux-armhf.so.3
Reading symbols from /lib/arm-linux-gnueabihf/libpng12.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libpng12.so.0
Reading symbols from /lib/arm-linux-gnueabihf/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libz.so.1
Reading symbols from /usr/lib/arm-linux-gnueabihf/libhybris-egl/libGLESv2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libhybris-egl/libGLESv2.so.2
Reading symbols from /lib/arm-linux-gnueabihf/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libm.so.6
Reading symbols from /usr/lib/arm-linux-gnueabihf/libQt5Network.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libQt5Network.so.5
Reading symbols from /usr/lib/arm-linux-gnueabihf/libQt5V8.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libQt5V8.so.5
Reading symbols from...

[Dave Morley (davmor2)]

(gdb) thread apply all bt

Thread 32 (Thread 0xb3c9d450 (LWP 2092)):
#0 0xb6568d12 in epoll_wait () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0xb414b69a in boost::asio::detail::epoll_reactor::run(bool, boost::asio::detail::op_queue<boost::asio::detail::task_io_service_operation>&) ()
   from /usr/lib/arm-linux-gnueabihf/libmirclient.so.7
#2 0xb414cbd6 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /usr/lib/arm-linux-gnueabihf/libmirclient.so.7
#3 0xb414a2d4 in ?? () from /usr/lib/arm-linux-gnueabihf/libmirclient.so.7
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 31 (Thread 0xb25f4450 (LWP 2095)):
#0 0xb609c5c4 in __libc_do_syscall ()
   from /lib/arm-linux-gnueabihf/libpthread.so.0
#1 0xb60981cc in pthread_cond_wait@@GLIBC_2.4 ()
   from /lib/arm-linux-gnueabihf/libpthread.so.0
#2 0xb31ccb62 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 30 (Thread 0xb1df4450 (LWP 2097)):
#0 0xb6568d12 in epoll_wait () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0xb4155490 in android::Looper::pollInner(int) ()
   from /usr/lib/arm-linux-gnueabihf/libmirclient.so.7
---Type <return> to continue, or q <return> to quit---

[Dave Morley (davmor2)]

davmor2@stryder:~$ adb shell
root@ubuntu-phablet:/# gdb unity8 2085
GNU gdb (Ubuntu 7.7-0ubuntu3) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from unity8...(no debugging symbols found)...done.
Attaching to program: /usr/bin/unity8, process 2085
Reading symbols from /usr/lib/arm-linux-gnueabihf/libQt5Gui.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libQt5Gui.so.5
Reading symbols from /usr/lib/arm-linux-gnueabihf/libQt5Qml.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libQt5Qml.so.5
Reading symbols from /usr/lib/arm-linux-gnueabihf/libQt5Quick.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libQt5Quick.so.5
Reading symbols from /usr/lib/arm-linux-gnueabihf/libQt5Core.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libQt5Core.so.5
Reading symbols from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
Reading symbols from /lib/arm-linux-gnueabihf/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libgcc_s.so.1
Reading symbols from /lib/arm-linux-gnueabihf/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libc.so.6
Reading symbols from /lib/ld-linux-armhf.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux-armhf.so.3
Reading symbols from /lib/arm-linux-gnueabihf/libpng12.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libpng12.so.0
Reading symbols from /lib/arm-linux-gnueabihf/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libz.so.1
Reading symbols from /usr/lib/arm-linux-gnueabihf/libhybris-egl/libGLESv2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libhybris-egl/libGLESv2.so.2
Reading symbols from /lib/arm-linux-gnueabihf/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/arm-linux-gnueabihf/libm.so.6
Reading symbols from /usr/lib/arm-linux-gnueabihf/libQt5Network.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libQt5Network.so.5
Reading symbols from /usr/lib/arm-linux-gnueabihf/libQt5V8.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/arm-linux-gnueabihf/libQt5V8.so.5
Reading symbols fro...

[kevin gunn (kgunn72)]

interesting, mir hasn't changed in since ~Feb 20th image

[kevin gunn (kgunn72)]

Does this image incl Qt5.2 ?

[Dave Morley (davmor2)]

NO this is pre Qt5.2 this is still standard image. Qt5.2 still has some quirks that need ironing out.

[Daniel van Vugt (vanvugt)]

The stack trace in comment 9 reveals the problem [Thread 22 (Thread 0xad8ff450 (LWP 2105)):]. An exception has happened (somewhere!?) and Mir is in the process of unwinding and exiting cleanly. However this process is blocked in unity-mir. Obviously it should not block.

So the first problem we can see is that unity-mir's SessionListener::stopping() is hung and failing to return. The second problem seems like it was probably an exception but we can't tell what that was. A stdout/err log from unity8 would help. But what would help more would be for unity-mir to not hang in this situation so we can probably get a crash and core file.

[Daniel van Vugt (vanvugt)]

Actually there are several threads shutting down and deadlocked (stack trace from comment #9). They're all stuck in QBasicMutex::lockInternal() via unity-mir.

[Daniel van Vugt (vanvugt)]

Let's treat the lock-up separately to the exception that triggered it. We'll have to solve the former in first.

[Dave Morley (davmor2)]

This might be fixed in 0.1.6 as it was seen in themwc image. Lets hold off on further work till that version lands.

[Dave Morley (davmor2)]

This is not fixed in 0.1.6 it is still happening on mir 0.1.6 but less frequently.

[Daniel van Vugt (vanvugt)]

As this bug is just about the unity-mir bug causing the hang, and not the exception which preceded that, Mir versions are not relevant to this bug. We need a fix for unity-mir to stop it hanging/deadlocking per above.

[Dave Morley (davmor2)]

Daniel that's fine, kgunn pointed out that it might actually be fixed with the mir update so i said I would check, and was reporting on the findings of that. :)

[Daniel d'Andrada (dandrader)]

From what I can tell, unity8 is in a deadlock because the mutex that protects Qt's event system was locked by Qt itself in some thread at some point in time and never released. Then when other threads wake up because, eg, a mir session is being closed and its wants to inform objects in the Qt side about it (such as some qml items) by posting some event or emitting a signal Qt internally tries to acquire that event system lock before doing so and then get blocked indefinitely.

Besides this basic assessment I'm pretty much clueless at the moment :)

Also wans't able to reproduce the freeze yet (got a unity8 crash+restart though)

[Gerry Boland (gerboland)]

Has anyone checked if pressing the power button a second time unlocks things? I just managed to reproduce a state where the device had locked up while running a barrage of autopilot tests, even though I had locked the display on. But hitting the power button unlocked shell and everything carried on as normal.

[Dave Morley (davmor2)]

Gerry this is a hard lock not the screen blanking.

However since the latest landing of mir I have not seen this issue since so I don't know if it touched something that has stabilised this issue

[Dave Morley (davmor2)]

This seems to of been fixed so I am marking it invalid for now if it starts up again I will reopen