maliit-server becomes deadlocked

[Bill Filler]

on krillin, rc-proposed 175

I'm experiencing keyboard not popping up on any application. Can't remember if this was after a reboot or if it was working for a while and then stopped. I believe it's after a reboot.

Upon inspection, it appears maliit-server process is deadlocked (see thread dump below). Also, there is no log in .cache/upstart/maliit-server.log only old logs maliit-server.log.1.gz etc, which leads me to believe this problem happens after reboot.

Seems that some QNetwork initialization is happening and this is causing an issue (see Threads 3, 2, and 1). We should figure out why we are even doing any network stuff at all, as we probably shouldn't be.

(gdb) t a a bt

Thread 14 (Thread 0xb2cc2440 (LWP 2844)):
#0 0xb66da4e2 in poll () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0xb3d03cdc in ?? () from /usr/lib/arm-linux-gnueabihf/libmircommon.so.5
#2 0xb3d04c96 in ?? () from /usr/lib/arm-linux-gnueabihf/libmircommon.so.5
#3 0xb67ec2a0 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#4 0xb6010490 in start_thread () from /lib/arm-linux-gnueabihf/libpthread.so.0
#5 0xb66e0c4c in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 13 (Thread 0xb2197440 (LWP 3128)):
#0 0xb435a840 in ?? ()
#1 0xb436e29c in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 12 (Thread 0xb240fb40 (LWP 3129)):
#0 0xb5c61836 in ?? () from /lib/arm-linux-gnueabihf/librt.so.1
#1 0xb5c618a4 in ?? () from /lib/arm-linux-gnueabihf/librt.so.1
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 11 (Thread 0xb1976440 (LWP 3130)):
#0 0xb6018d44 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
#1 0xb60146b0 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
#2 0xb225e0a8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 10 (Thread 0xb1176440 (LWP 3131)):
---Type <return> to continue, or q <return> to quit---
#0 0xb6018d44 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
#1 0xb60146b0 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
#2 0xb225e0a8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 9 (Thread 0xb0697440 (LWP 3143)):
#0 0xb66da4e2 in poll () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0xb5cac0d8 in ?? () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 8 (Thread 0xafcff440 (LWP 3144)):
#0 0xb66da4e2 in poll () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0xb5cac0d8 in ?? () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 7 (Thread 0xaeaff440 (LWP 3146)):
#0 0xb66da4e2 in poll () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0xb5cac0d8 in ?? () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 6 (Thread 0xae1ff440 (LWP 3148)):
#0 0xb66da4e2 in poll () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0xb5cac0d8 in ?? () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 5 (Thread 0xacb49440 (LWP 3167)):
#0 0xb66e1132 in epoll_wait () from /lib/arm-linux-gnueabihf/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1 0xb3b62312 in ?? () from /usr/lib/arm-linux-gnueabihf/libdbus-cpp.so.4
#2 0xb3b642ea in ?? () from /usr/lib/arm-linux-gnueabihf/libdbus-cpp.so.4
#3 0xb3b6484a in ?? () from /usr/lib/arm-linux-gnueabihf/libdbus-cpp.so.4
#4 0xb3b4d678 in core::dbus::Bus::run() () from /usr/lib/arm-linux-gnueabihf/libdbus-cpp.so.4
#5 0xb67ec2a0 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#6 0xb6010490 in start_thread () from /lib/arm-linux-gnueabihf/libpthread.so.0
#7 0xb66e0c4c in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 4 (Thread 0xac349440 (LWP 3169)):
#0 0xb66da4e2 in poll () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0xad94c0a6 in ?? () from /usr/lib/arm-linux-gnueabihf/libpulse.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 3 (Thread 0xa7b48440 (LWP 3170)):
#0 0xb6018d44 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
#1 0xb60146b0 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
#2 0xb689a138 in wait (time=4294967295, this=0xae216748) at thread/qwaitcondition_unix.cpp:128
#3 QWaitCondition::wait (this=<optimized out>, mutex=mutex@entry=0xae2151a8, time=time@entry=4294967295)
    at thread/qwaitcondition_unix.cpp:200
#4 0xb6895254 in QSemaphore::acquire (this=this@entry=0xa7b47520, n=n@entry=1) at thread/qsemaphore.cpp:137
#5 0xb6a042da in QMetaMethod::invoke (this=this@entry=0x3a, object=object@entry=0xae2148b0,
    connectionType=connectionType@entry=Qt::AutoConnection, returnValue=..., val0=..., val1=..., val2=...,
    val3=..., val4=..., val5=..., val6=..., val7=..., val8=..., val9=...) at kernel/qmetaobject.cpp:2240
#6 0xb6a06eb6 in QMetaObject::invokeMethod (obj=0xae2148b0, member=<optimized out>, type=Qt::AutoConnection,
    ret=..., val0=..., val1=..., val2=..., val3=..., val4=..., val5=..., val6=..., val7=..., val8=..., val9=...)
    at kernel/qmetaobject.cpp:1478
---Type <return> to continue, or q <return> to quit---
#7 0xb5bda584 in QNetworkConfigurationManagerPrivate::updateConfigurations() ()
   from /usr/lib/arm-linux-gnueabihf/libQt5Network.so.5
#8 0xb5bda910 in QNetworkConfigurationManagerPrivate::initialize() ()
   from /usr/lib/arm-linux-gnueabihf/libQt5Network.so.5
#9 0xb5bd5c20 in qNetworkConfigurationManagerPrivate() () from /usr/lib/arm-linux-gnueabihf/libQt5Network.so.5
#10 0xb5bd5cd8 in QNetworkConfigurationManager::QNetworkConfigurationManager(QObject*) ()
   from /usr/lib/arm-linux-gnueabihf/libQt5Network.so.5
#11 0xb5bac358 in QNetworkAccessManager::QNetworkAccessManager(QObject*) ()
   from /usr/lib/arm-linux-gnueabihf/libQt5Network.so.5
#12 0xacfa50c4 in QSampleCache::networkAccessManager() () from /usr/lib/arm-linux-gnueabihf/libQt5Multimedia.so.5
#13 0xacfa529e in QSample::load() () from /usr/lib/arm-linux-gnueabihf/libQt5Multimedia.so.5
#14 0xb6a1da54 in QObject::event (this=<optimized out>, e=<optimized out>) at kernel/qobject.cpp:1245
#15 0xb69fccb2 in QCoreApplication::notify (this=<optimized out>, receiver=<optimized out>,
    event=<optimized out>) at kernel/qcoreapplication.cpp:997
#16 0xb69fcd78 in QCoreApplication::notifyInternal (this=0xbe9ec454, receiver=receiver@entry=0xb2dd68,
    event=event@entry=0xb2de10) at kernel/qcoreapplication.cpp:935
#17 0xb69fe584 in sendEvent (event=0xb2de10, receiver=0xb2dd68)
    at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:228
#18 QCoreApplicationPrivate::sendPostedEvents (receiver=receiver@entry=0x0, event_type=event_type@entry=0,
    data=0xb2da48) at kernel/qcoreapplication.cpp:1552
#19 0xb69fe97a in QCoreApplication::sendPostedEvents (receiver=receiver@entry=0x0, event_type=event_type@entry=0)
    at kernel/qcoreapplication.cpp:1410
#20 0xb6a37950 in postEventSourceDispatch (s=0xae20dc00) at kernel/qeventdispatcher_glib.cpp:271
#21 0xb5cabf68 in g_main_context_dispatch () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
#22 0xb5cac114 in ?? () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

---Type <return> to continue, or q <return> to quit---
Thread 2 (Thread 0xa7348440 (LWP 3171)):
#0 0xb6018d44 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
#1 0xb601695c in __lll_lock_wait () from /lib/arm-linux-gnueabihf/libpthread.so.0
#2 0xb6012202 in pthread_mutex_lock () from /lib/arm-linux-gnueabihf/libpthread.so.0
#3 0xb5fda272 in dbus_connection_send_with_reply () from /lib/arm-linux-gnueabihf/libdbus-1.so.3
#4 0xb5fda4c4 in dbus_connection_send_with_reply_and_block () from /lib/arm-linux-gnueabihf/libdbus-1.so.3
#5 0xb6bbff3c in q_dbus_connection_send_with_reply_and_block (error=0xa7347844, timeout_milliseconds=-1,
    message=0xace26cc0, connection=0xae211aa0) at qdbus_symbols_p.h:135
#6 QDBusConnectionPrivate::sendWithReply (this=0xae2105d0, message=..., sendMode=<optimized out>,
    timeout=timeout@entry=-1) at qdbusintegrator.cpp:2046
#7 0xb6bb3314 in QDBusConnection::call (this=this@entry=0xace268f0, message=..., mode=mode@entry=QDBus::Block,
    timeout=timeout@entry=-1) at qdbusconnection.cpp:576
#8 0xb6bc768e in QDBusAbstractInterface::callWithArgumentList (this=0xa7347998, mode=QDBus::Block, method=...,
    args=...) at qdbusabstractinterface.cpp:476
#9 0xa6b06502 in ?? () from /usr/lib/arm-linux-gnueabihf/qt5/plugins/bearer/libqnmbearer.so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0xb6fc2000 (LWP 2791)):
#0 0xb66de620 in syscall () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0xb6893e6a in _q_futex (op=0, val=3, timeout=0x0, addr=<optimized out>) at thread/qmutex_linux.cpp:146
#2 lockInternal_helper<false> (timeout=-1, elapsedTimer=0x0, d_ptr=...) at thread/qmutex_linux.cpp:187
#3 QBasicMutex::lockInternal (this=this@entry=0xae210e8c) at thread/qmutex_linux.cpp:203
#4 0xb6893f0e in lock (this=0xae210e8c) at thread/qmutex.h:59
#5 lock (timeout=-1, this=0xae210e80) at thread/qmutex.cpp:620
#6 QMutex::lock (this=this@entry=0xae2105f4) at thread/qmutex.cpp:215
#7 0xb6bbc586 in QDBusMutexLocker (m=0xae2105f4, s=0xae2105d0, a=ToggleWatchAction, this=<synthetic pointer>)
    at qdbusthreaddebug_p.h:183
---Type <return> to continue, or q <return> to quit---
#8 QDBusDispatchLocker (s=0xae2105d0, a=ToggleWatchAction, this=<synthetic pointer>) at qdbusthreaddebug_p.h:198
#9 qDBusRealToggleWatch (d=0xae2105d0, watch=0xae2117a0, fd=40) at qdbusintegrator.cpp:346
#10 0xb5fea8f6 in ?? () from /lib/arm-linux-gnueabihf/libdbus-1.so.3
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
Download as text
Poster:
Bill <email address hidden>

Your name (30 characters max)
Syntax:
Content:
  Paste!

[Pat McGowan (pat-mcgowan)]

Adding qt as qdbus seems suepect
see bug #1421009 and perhaps https://bugreports.qt.io/browse/QTBUG-48410 and https://bugreports.qt.io/browse/QTBUG-44836

[Albert Astals Cid (aacid)]

Agreeing with Pat here dbus_connection_send_with_reply on one thread and qDBusRealToggleWatch in the other are basically the signs of the known qdbus lock we've seen before.

[Albert Astals Cid (aacid)]

As per

> We should figure out why we are even doing any network stuff at all, as we probably shouldn't be.

It comes from the fact that QtMultimedia uses network transparent methods to load urls used for audio playing, so even if in the keyboard we're using a local sound for audio feedback, QtMultimedia ends up bringing up the network backend because it uses those network transparent methods.

[Tony Espy (awe)]

Any Qt code that uses QNetworkAccessManager, QNetworkSession, or related classes can cause the QNetwork Bearer Management to get instantiated, which in turn loads a network-manager specific plugin. Please see bug #1480877 for details. As it's really long, you might want to check out just the following comments:

https://bugs.launchpad.net/ubuntu-rtm/+source/location-service/+bug/1480877/comments/65
https://bugs.launchpad.net/ubuntu-rtm/+source/location-service/+bug/1480877/comments/68

Besides the problems with match rules that are described in the bug, we also discovered that there's quite a bit of blocking DBus logic in this bearer mgmt plugin.

We have an updated version of this plugin available for testing in silo-026, which also contains some location-services DBus fixes as well. If you want, you can just install the qtbase network package:

https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-026/+files/libqt5network5_5.4.1%2Bdfsg-2ubuntu11%7Evivid1%7Etest1_armhf.deb

We basically removed much of the code that pays attention to NM access points and devices, which included a good portion of the blocking DBus calls that were being made. That said, not all were removed, so there still might be the potential for lockup...

[Tony Espy (awe)]

@Bill

Can you have someone re-test with the latest rc-proposed? As mentioned in my previous comment, we removed a large amount of NM related DBus code from the Qt Bearer Management plugin as a fix for bug #1480877. This work all landed in the PPA and is being tested by QA for release as OTA8.5.

[Jean-Baptiste Lallement (jibel)]

Could someone test and confirm if it is still an issue or not as requested by Tony. I didn't experience this problem with latest builds.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.