Unity8 reliably crashes on logout due to heap corruption (libprotobuf.so.9 called from libprotobuf-lite.so.9), causing delays while it dumps core and reports errors.

[Andrea Bernabei]

https://errors.ubuntu.com/problem/72327e33abf9e9da2cd9ed1ac28b9d843c18b5bb

This is happening on a fresh setup, which is basically:
Xenial 17Jan iso + unity8-desktop-session-mir package

When I logout, I get a black screen. Turns out unity8 is crashing with a "double free or linked list corruption"

Please find unity8.log attached

[Albert Astals Cid (aacid)]

Could you get us a gdb backtrace?

[Albert Astals Cid (aacid)]

I wonder if it's the same as https://bugs.launchpad.net/ubuntu/+source/unity8/+bug/1521138 or not

[Andrea Bernabei (faenil)]

there you go

Continuing.
[Thread 0x7f9aa8959700 (LWP 5318) exited]
[Thread 0x7f9a9cef3700 (LWP 4924) exited]
[Thread 0x7f9ae5bb9700 (LWP 4530) exited]
[Thread 0x7f9aab9dc700 (LWP 4858) exited]
[Thread 0x7f9aa995b700 (LWP 5323) exited]
[Thread 0x7f9ab305d700 (LWP 4856) exited]
[Thread 0x7f9ae63ba700 (LWP 4529) exited]
[Thread 0x7f9ae6bbb700 (LWP 4528) exited]
[Thread 0x7f9ae73bc700 (LWP 4527) exited]
[Thread 0x7f9ae7bbd700 (LWP 4526) exited]
[Thread 0x7f9af6402700 (LWP 4524) exited]

Program received signal SIGABRT, Aborted.
0x00007f9b03971227 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

Thread 17 (Thread 0x7f9af7404700 (LWP 4522)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f9afab3887c in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
#2 0x00007f9b024b066a in start_thread (arg=0x7f9af7404700) at pthread_create.c:333
#3 0x00007f9b03a42e4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 16 (Thread 0x7f9af6c03700 (LWP 4523)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f9afab3887c in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
#2 0x00007f9b024b066a in start_thread (arg=0x7f9af6c03700) at pthread_create.c:333
#3 0x00007f9b03a42e4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 14 (Thread 0x7f9af5787700 (LWP 4525)):
#0 0x00007f9b03a3783d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f9afa448ae9 in ?? () from /usr/lib/x86_64-linux-gnu/libmircommon.so.5
#2 0x00007f9afa44acfe in ?? () from /usr/lib/x86_64-linux-gnu/libmircommon.so.5
#3 0x00007f9b03fd5d10 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007f9b024b066a in start_thread (arg=0x7f9af5787700) at pthread_create.c:333
#5 0x00007f9b03a42e4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 8 (Thread 0x7f9ad1e0b700 (LWP 4531)):
#0 0x00007f9b03a3783d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f9b0135530c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2 0x00007f9b0135541c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3 0x00007f9b01355459 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4 0x00007f9b0137bae5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#5 0x00007f9b024b066a in start_thread (arg=0x7f9ad1e0b700) at pthread_create.c:333
#6 0x00007f9b03a42e4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 7 (Thread 0x7f9ad160a700 (LWP 4532)):
#0 0x00007f9b03a3783d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f9b0135530c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2 0x00007f9b01355692 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3 0x00007f9af8ddc016 in ?? () from /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
#4 0x00007f9b0137bae5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#5 0x00007f9b024b066a in start_thread (arg=0x7f9ad160a700) at pthread_create.c:333
#6 0x00007f9b03a42e4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 6 (Thread 0x7f9ac7fff700 (L...

[Albert Astals Cid (aacid)]

Seems like a mir issue?

Thread 1 (Thread 0x7f9afc45c8c0 (LWP 4435)):
#0 0x00007f9b03971227 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007f9b03972e8a in __GI_abort () at abort.c:89
#2 0x00007f9b039b4bb3 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7f9b03acd128 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007f9b039bcbc9 in malloc_printerr (ptr=<optimised out>, str=0x7f9b03acd238 "double free or corruption (out)", action=1) at malloc.c:4965
#4 _int_free (av=<optimised out>, p=<optimised out>, have_lock=0) at malloc.c:3834
#5 0x00007f9b039c07fc in __GI___libc_free (mem=<optimised out>) at malloc.c:2950
#6 0x00007f9ac6838901 in google::protobuf::internal::DestroyDefaultRepeatedFields() () from /usr/lib/x86_64-linux-gnu/libprotobuf.so.9
#7 0x00007f9af99a172b in google::protobuf::ShutdownProtobufLibrary() () from /usr/lib/x86_64-linux-gnu/libprotobuf-lite.so.9
#8 0x00007f9afa1f85e9 in ?? () from /usr/lib/x86_64-linux-gnu/libmirprotobuf.so.3
#9 0x00007f9b05e68bf7 in _dl_fini () at dl-fini.c:252
#10 0x00007f9b03975cf2 in __run_exit_handlers (status=0, listp=0x7f9b03d00698 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#11 0x00007f9b03975d45 in __GI_exit (status=<optimised out>) at exit.c:104
#12 0x00007f9b0395ca07 in __libc_start_main (main=0x406810, argc=1, argv=0x7ffefeb6f5b8, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>,
    stack_end=0x7ffefeb6f5a8) at libc-start.c:323
#13 0x0000000000406939 in _start ()

[Alberto Aguirre (albaguirre)]

It's a protobuf problem really.

The issue is Mir's libmirprotobuf links against libprotobuf-lite; the call to ShutdownProtobufLibrary is expected to be resolved by libprotobuf-lite; in this case presumably libprotobuf was also loaded and the linker is resolving "ShutdownProtobufLibrary" to libmirprotobuf instead hence the crash.

[Alberto Aguirre (albaguirre)]

An strace log would be helpful to see what's loading libprotobuf.

[Andrea Bernabei (faenil)]

using lsof I noticed that libprotobuf was being loaded by
unity8
telephony-service-handler

After inspection with ldd, I found out that
- unity8 doesn't explicitly load libprotobuf
- /usr/bin/telephony-service* (-handler, -indicator, -approver) all link to libprotobuf.so.9

Does that help?

[Andrea Bernabei (faenil)]

I also modified /usr/share/upstart/sessions/unity8.conf to run "strace unity8"

and unity8 doesn't seem to be directly loading libprotobuf (even though lsof refers to unity8)

:~$ cat .cache/upstart/unity8.log | grep protobuf
open("/usr/lib/x86_64-linux-gnu/unity8/libmirprotobuf.so.3", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libmirprotobuf.so.3", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/x86_64-linux-gnu/unity8/libprotobuf-lite.so.9", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libprotobuf-lite.so.9", O_RDONLY|O_CLOEXEC) = 3
read(33, "x-gnu/libmirprotobuf.so.3\n7f821c"..., 1024) = 1024
read(33, "x-gnu/libmirprotobuf.so.3\n7f821c"..., 1024) = 1024

[Alberto Aguirre (albaguirre)]

Right, so somehow unity8 loads libmirprotobuf (a plugin perhaps?)

Not much we can do from mir side.

It's a consequence of loading libprotobuf and libprotobuf-lite in the same process, as the library symbols will have name collisions (a protobuf issue).

[Alberto Aguirre (albaguirre)]

Ooops s/libmirprotobuf/libprotobuf (typo)

[Andrea Bernabei (faenil)]

full strace (sorry, internal as it may contain private info)

https://pastebin.canonical.com/148214/

let me know how to proceed :)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Albert Astals Cid (aacid)]

Mir uses libprotobuf-lite
telephony-service (though libphonenumber) uses libprotobuf

According to Alberto both are incompatible so we need to make Mir use libprotobuf or make libphonenumber use libprotobuf-lite

Anyone knows what would be a better idea/possible?

[Albert Astals Cid (aacid)]

When i say telephony-service i mean libtelephonyservice-qml.so that unity8 uses as a plugin.

[Tiago Salem Herrmann (tiagosh)]

It is possible to link libphonenumber against protobuf-lite with this patch: http://pastebin.ubuntu.com/14680746/ .

[Daniel van Vugt (vanvugt)]

Indeed libprotobuf-lite is more desirable than libprotobuf. It was a significant step forward when Alberto made it possible for Mir to switch to "lite". Although both protobuf versions suffer for coexistence problems, obviously.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andrea Bernabei (faenil)]

was this fixed in libphonenumber already?

I can logout after today's upgrade...

[Christopher Townsend (townsend)]

I still this crash on a fully updated Xenial machine.

[Daniel van Vugt (vanvugt)]

libphonenumber still uses the /unsafe/ library on xenial:
$ ldd /usr/lib/x86_64-linux-gnu/libphonenumber.so.7.0
 libprotobuf.so.9 => /usr/lib/x86_64-linux-gnu/libprotobuf.so.9 (0x00007f4aebb5d000)

However I'm not sure that's the primary problem. I'm getting a very similar crash on Unity8 login now. Although that could just be this same shutdown crash triggered by some other start-up failure.

[Daniel van Vugt (vanvugt)]

Sorry, my unity8.log's are from yesterday as I can see evidence of me running clients. My crashing unity8 on login is doing so without a trace :(

[Daniel van Vugt (vanvugt)]

If we are to solve this crash, we either need Mir to switch from libprotobuf-lite back to libprotobuf, or all of these to switch to libprotobuf-lite:

/usr/lib/x86_64-linux-gnu/history-service/plugins/libsqlitehistoryplugin.so:
 libprotobuf.so.9 => /usr/lib/x86_64-linux-gnu/libprotobuf.so.9 (0x00007fe06ea55000)
/usr/lib/x86_64-linux-gnu/libhistoryservice.so.0.0.0:
 libprotobuf.so.9 => /usr/lib/x86_64-linux-gnu/libprotobuf.so.9 (0x00007f8120b7d000)
/usr/lib/x86_64-linux-gnu/libhistoryservice.so.0:
 libprotobuf.so.9 => /usr/lib/x86_64-linux-gnu/libprotobuf.so.9 (0x00007fdf2a6ce000)
/usr/lib/x86_64-linux-gnu/libphonenumber.so.7.0:
 libprotobuf.so.9 => /usr/lib/x86_64-linux-gnu/libprotobuf.so.9 (0x00007f3dd1822000)
/usr/lib/x86_64-linux-gnu/libphonenumber.so.7:
 libprotobuf.so.9 => /usr/lib/x86_64-linux-gnu/libprotobuf.so.9 (0x00007fa96806a000)
/usr/lib/x86_64-linux-gnu/qt5/qml/Ubuntu/Telephony/libtelephonyservice-qml.so:
 libprotobuf.so.9 => /usr/lib/x86_64-linux-gnu/libprotobuf.so.9 (0x00007fd491586000)

[Daniel van Vugt (vanvugt)]

Or for someone to fix protobuf itself, of course.

[Alan Griffiths (alan-griffiths)]

I suspect the right solution is to fix the approach taken in Mesa to detecting Mir:

All the above modules should be loaded with RTLD_LOCAL. As far as I can tell they are so they shouldn't interact.

However, Mir's mesa.so.X client platform module reloads itself with RTLD_GLOBAL so that Mesa can find a Mir entry point. (Yes, this is an ugly hack.)

That is likely what causes symbols to be resolved from the wrong protobuf library and the problems seen here.

[Daniel van Vugt (vanvugt)]

I was wondering why we don't get more heat from users on this crash since it happens every time you log out of Unity8. Turns out we do get the crash reports and nobody noticed...

https://errors.ubuntu.com/problem/72327e33abf9e9da2cd9ed1ac28b9d843c18b5bb

[Andrea Azzarone (azzar1)]

This prevents to cleanly restart unity8 too.

[Daniel van Vugt (vanvugt)]

Not in progress any more; that was just an experiment in the branch attached.

Although I have not verified this bug still happens. The last known definite record of it was dupe bug 1654308 from 5 January.