[TOPBLOCKER] unity8 crash in image krillin rtm 139

[Selene ToyKeeper]

Earlier today, a few QA folks tested image krillin rtm 138 plus silo 13, and found that it fixed the unity8 crash from bug 1382595.

Silo 13 plus silo 10 landed in image 139.

Now, in image 139, unity8 is crashy again. Olli mentioned getting two crashes within a few minutes, and I got a unity8 crash while trying to accept an incoming call. I'm not sure if it's the same crash as before, or if it's a new one.

Attached is the crash dump I got just after hitting 'accept' for an incoming call. I tried to pre-process it in apport-cli, but it failed with "Sorry, the program "unity8" closed unexpectedly // Your computer does not have enough free memory to automatically analyze the problem and send a report to the developers."

[Brendan Donegan (brendan-donegan)]

In 3 hours testing this morning I got one crash: http://people.canonical.com/~brendan-donegan/_usr_bin_unity8.32011.crash

[Brendan Donegan (brendan-donegan)]

When you say it's 'crashy' how many crashes did you actually get - just the one? The description doesn't make that clear.

[Michał Sawicz (saviq)]

Yours, Brendan, is unknown as of yet, it's in errors as https://errors.ubuntu.com/oops/ac6af12e-60fa-11e4-940c-fa163e22e467 and https://errors.ubuntu.com/oops/e8b1c5f4-60e1-11e4-9f52-fa163e525ba7.

[Selene ToyKeeper (toykeeper)]

I think this is still happening in image 140. Here's the crash dump I got on 140 after trying to turn wifi off in the network indicator.

iahmad reported unity8 crashes on this build (several times) when attempting to answer incoming calls, and pmcgowan managed to reproduce that issue once too.

[Víctor R. Ruiz (vrruiz)]

I made two incoming and not accepted calls to krillin, which apparently caused a crash.

[Víctor R. Ruiz (vrruiz)]

As a matter of fact:

current build number: 140
device name: krillin
channel: ubuntu-touch/ubuntu-rtm/14.09-proposed

[I Ahmad (iahmad)]

here is the unity8 crash dump I got

[I Ahmad (iahmad)]

as well as the telephony service approver crash dump

[Olli Ries (ories)]

I have tried a variety of incoming calls, denied/accepted and could not reproduce on krillin #140

[Oliver Grawert (ogra)]

for me it happened out of the blue without any incoming calls yesterday ... (with a few webapps and dekko open)

[Thomas Voß (thomas-voss)]

A full crash report with a core dump would be really helpful.

[Bill Filler (bfiller)]

Here is the stack from the telephony-service crash that is attached. Looks like thread 1 is the culprit, is using media-hub client, I'm guessing to play incoming ringtone on incoming call.

Thread 7 (Thread 0xb19ff3d0 (LWP 4010)):
#0 0xb64a4aa2 in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0xb66f40c0 in ?? () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 6 (Thread 0xb07ff3d0 (LWP 4018)):
#0 0xb64a4aa2 in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0xb66f40c0 in ?? () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 5 (Thread 0xb0fff3d0 (LWP 4014)):
#0 0xb64a4aa2 in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0xb60b91b8 in ?? () from /lib/arm-linux-gnueabihf/libdbus-1.so.3
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 4 (Thread 0xb2bcd3d0 (LWP 4008)):
#0 __libc_do_syscall () at ../ports/sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:43
#1 0xb63fe192 in __pthread_cond_wait (cond=0x14bdfb0, mutex=0x14bdf90) at pthread_cond_wait.c:187
#2 0xb60a7a5e in ?? () from /lib/arm-linux-gnueabihf/libdbus-1.so.3
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 3 (Thread 0xb23b03d0 (LWP 4009)):
#0 0xb64a4aa2 in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0xb66f40c0 in ?? () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 2 (Thread 0xb6f2f220 (LWP 4001)):
#0 0xb64a4aa2 in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0xb66f40c0 in ?? () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0xaf4153d0 (LWP 4025)):
---Type <return> to continue, or q <return> to quit---
#0 __libc_do_syscall () at ../ports/sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:44
#1 0xb643de5e in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#2 0xb643eb4e in __GI_abort () at abort.c:89
#3 0xb64390c4 in __assert_fail_base (fmt=0x1 <error: Cannot access memory at address 0x1>,
    assertion=0xb6402f5c "mutex->__data.__owner == 0", assertion@entry=0x0,
    file=0xb64032d4 "pthread_mutex_lock.c", file@entry=0xaf4153d0 "\001", line=80,
    line@entry=3058708652,
    function=function@entry=0xb6402fd4 <__PRETTY_FUNCTION__.10294> "__pthread_mutex_lock")
    at assert.c:92
#4 0xb643915a in __GI___assert_fail (assertion=0x0, file=0xaf4153d0 "\001", line=3058708652,
    line@entry=80, function=0xb6402fd4 <__PRETTY_FUNCTION__.10294> "__pthread_mutex_lock")
    at assert.c:101
#5 0xb63fc834 in __GI___pthread_mutex_lock (mutex=0xb65030ac <lock>) at pthread_mutex_lock.c:80
#6 0xafd0db94 in ?? () from /usr/lib/arm-linux-gnueabihf/libmedia-hub-client.so.2
#7 0xafd179aa in ?? () from /usr/lib/arm-linux-gnueabihf/libmedia-hub-client.so.2
#8 0xafd1b016 in ?? () from /usr/lib/arm-linux-gnueabihf/libmedia-hub-client.so.2
#9 0xb393f09a in ?? () from /usr/lib/arm-linux-gnueabihf/libdbus-cpp.so.4
#10 0xb3935e0c in ?? () fr...

[Bill Filler (bfiller)]

how come none of the unity8 stack traces have CoreDumps associated with them? Is there a way to configure apport to produce these always?

[Bill Filler (bfiller)]

I just rebooted the phone and attached to unity8 with gdb:
- sudo gdb --pid XXXX /usr/bin/unity8

First strange thing noticed, after typing continue in gdb, I see Threads endlessly being created and destroyed, this never stops:
[New Thread 0xa18d2450 (LWP 11331)]
[Thread 0xa18d2450 (LWP 11331) exited]
repeated continously..

Then I use the phone, call it a few times. Finally it crashes. Gdb shows me:
Program received signal SIGSEGV, Segmentation fault.
0xb61d342a in _int_malloc (av=av@entry=0xb62694e8 <main_arena>, bytes=bytes@entry=8)
    at malloc.c:3351
3351 malloc.c: No such file or directory.
(gdb)
Continuing.
[Thread 0x9a905450 (LWP 11767) exited]
[Thread 0xa18d2450 (LWP 11434) exited]
[Thread 0xb37e0450 (LWP 2488) exited]
[Thread 0xb2fe0450 (LWP 2489) exited]
[Thread 0xb2729450 (LWP 2494) exited]
[Thread 0xb1c02450 (LWP 2497) exited]
[Thread 0xb12feb40 (LWP 2498) exited]
[Thread 0xb12f8450 (LWP 2499) exited]
[Thread 0xb0af8450 (LWP 2500) exited]
[Thread 0xb00b2450 (LWP 2501) exited]
[Thread 0xaf6ff450 (LWP 2502) exited]
[Thread 0xaecff450 (LWP 2503) exited]
[Thread 0xae4ff450 (LWP 2504) exited]
[Thread 0xac522450 (LWP 2540) exited]
[Thread 0xabd22450 (LWP 2547) exited]
[Thread 0xaaa5f450 (LWP 2621) exited]
[Thread 0xa93ec450 (LWP 2674) exited]
[Thread 0xa8425450 (LWP 2788) exited]
[Thread 0xa761c450 (LWP 2797) exited]
[Thread 0xa5e62450 (LWP 2805) exited]
[Thread 0xa46fc450 (LWP 2831) exited]
[Thread 0xa3e92450 (LWP 2832) exited]
[Thread 0xa3692450 (LWP 2833) exited]
[Thread 0xa2e91450 (LWP 2874) exited]
[Thread 0xa10d2450 (LWP 2907) exited]
[Thread 0xa08d2450 (LWP 2908) exited]
[Thread 0xa00d2450 (LWP 2909) exited]
[Thread 0x9f285450 (LWP 3037) exited]
[Thread 0x9c105450 (LWP 3039) exited]
[Thread 0x9b905450 (LWP 3040) exited]
[Thread 0x9b105450 (LWP 3041) exited]
[Thread 0xb6fc8000 (LWP 2454) exited]

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb) break
Breakpoint 1 at 0xb61d342a: file malloc.c, line 3351.

So I believe we are running out of memory because of all these threads being created. I do not know how to determine who is creating the threads or why, but assuming this is the issue. Saw the exact same thing when attaching to system-settings after unity8 crashed.

[Bill Filler (bfiller)]

Per tvoss, here is the requested output:
  (1.) sudo apt-get install libc6-dbg ​libstdc++6-4.9-dbg
  (2.) Attach with gdb to Unity 8 as Bill pointed out
  (3.) In gdb, enable a breakpoint on thread creation with: break pthread_create
  (4.) When the breakpoint is hit, call "t a a bt" and add the output to https://bugs.launchpad.net/ubuntu/+source/unity8/+bug/1387959

output of "t a a bt" after breakpoint hit:
http://pastebin.ubuntu.com/8794120/

output of "bt full"
http://pastebin.ubuntu.com/8794202/

[Olli Ries (ories)]

escalated to [TOPBLOCKER]

[Michał Sawicz (saviq)]

I was only able to reproduce the thread pounding on the phone, neither X11, Mir desktop session or the emulator have shown this same behaviour.

On the device, however, this is happening right from the start. The incomplete backtrace also suggests this is triggered from the android side, IIUC.

[Michał Sawicz (saviq)]

Re: threads, there was bug #1359951 before, it was concluded that the thread is created and destroyed by either the driver or the hwcomposer, not much can be done without vendor support.

[kevin gunn (kgunn72)]

worried this may be a duplicate of this
https://bugs.launchpad.net/unity-system-compositor/+bug/1359951

[kevin gunn (kgunn72)]

scratch that on
https://bugs.launchpad.net/unity-system-compositor/+bug/1359951
seems the driver thread spawing is unrelated to the crash we're seeing. (which makes sense, it's been in there since the beginning)

[Launchpad Janitor (janitor)]

This bug was fixed in the package qtubuntu-media - 0.7.1+15.04.20141105.2-0ubuntu1

---------------
qtubuntu-media (0.7.1+15.04.20141105.2-0ubuntu1) vivid; urgency=low

  [ thomas-voss ]
  * Get rid of custom ref-counting and reuse pre-created object. (LP:
    #1387959)
 -- Ubuntu daily release <email address hidden> Wed, 05 Nov 2014 12:10:39 +0000

[Bill Filler (bfiller)]

Got this stack trace on a crash of unity8-dash after a call ended:
(gdb) t a a bt

Thread 1 (Thread 0xb6f87000 (LWP 6251)):
#0 0xb61558e6 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0xb6163e5e in raise () from /lib/arm-linux-gnueabihf/libc.so.6
#2 0xb6164b4e in abort () from /lib/arm-linux-gnueabihf/libc.so.6
#3 0xb637a7d6 in QMessageLogger::fatal(char const*, ...) const ()
   from /usr/lib/arm-linux-gnueabihf/libQt5Core.so.5
#4 0xb3f7e356 in ?? ()
   from /usr/lib/arm-linux-gnueabihf/qt5/plugins/platforms/libqpa-ubuntumirclient.so
#5 0xb3f80bc4 in ?? ()
   from /usr/lib/arm-linux-gnueabihf/qt5/plugins/platforms/libqpa-ubuntumirclient.so
#6 0xb6d0120a in QPlatformIntegrationFactory::create(QString const&, QStringList const&, int&, char**, QString const&) () from /usr/lib/arm-linux-gnueabihf/libQt5Gui.so.5
#7 0xb6d08c34 in QGuiApplicationPrivate::createPlatformIntegration() ()
   from /usr/lib/arm-linux-gnueabihf/libQt5Gui.so.5
#8 0xb6d09396 in QGuiApplicationPrivate::createEventDispatcher() ()
   from /usr/lib/arm-linux-gnueabihf/libQt5Gui.so.5
#9 0xb64e9f20 in QCoreApplication::init() ()
   from /usr/lib/arm-linux-gnueabihf/libQt5Core.so.5
#10 0xb64e9fa6 in QCoreApplication::QCoreApplication(QCoreApplicationPrivate&) ()
   from /usr/lib/arm-linux-gnueabihf/libQt5Core.so.5
#11 0xb6d09e06 in QGuiApplication::QGuiApplication(int&, char**, int) ()
   from /usr/lib/arm-linux-gnueabihf/libQt5Gui.so.5
#12 0x00013594 in ?? ()
#13 0xb6155630 in __libc_start_main () from /lib/arm-linux-gnueabihf/libc.so.6
#14 0x0001412c in _start ()
(gdb)

[Michał Sawicz (saviq)]

Bill, that's a different bug/crash, can you please file one, ideally with steps to repro? Can you find the OOPS id of this in /var/log/upstart/whoopsie.log, too?

[Launchpad Janitor (janitor)]

This bug was fixed in the package qtubuntu-media - 0.7.1+15.04.20141105.2~rtm-0ubuntu1

---------------
qtubuntu-media (0.7.1+15.04.20141105.2~rtm-0ubuntu1) 14.09; urgency=low

  [ thomas-voss ]
  * Get rid of custom ref-counting and reuse pre-created object. (LP:
    #1387959)
 -- Ubuntu daily release <email address hidden> Wed, 05 Nov 2014 12:10:39 +0000