Open3D segfaults when compiled with LTO

[Jochen Sprickerhof]

$ sudo apt install open3d-gui
$ Open3D
...
Segfault

We had the same problem in Debian and disabled LTO because of that.

https://github.com/isl-org/Open3D/issues/4747

[Michael Hudson-Doyle (mwhudson)]

Turns out open3d is already in lto-disabled-list so something else is going on.

[Michael Hudson-Doyle (mwhudson)]

This is the traceback of how it's crashing:

(gdb) bt
#0 0x00007ffff6f5779f in unlink_chunk (p=p@entry=0x555555d7b380, av=0x7ffff70cac80 <main_arena>) at ./malloc/malloc.c:1628
#1 0x00007ffff6f5a6ab in _int_malloc (av=av@entry=0x7ffff70cac80 <main_arena>, bytes=bytes@entry=4584) at ./malloc/malloc.c:4307
#2 0x00007ffff6f5b1b9 in __GI___libc_malloc (bytes=bytes@entry=4584) at ./malloc/malloc.c:3329
#3 0x00007ffff6efa880 in __GI___qsort_r (b=0x555555d7a790, n=191, s=24, cmp=0x7ffff5faf320, arg=0x0) at ./stdlib/msort.c:221
#4 0x00007ffff5fb76f1 in stbrp_pack_rects () from /lib/x86_64-linux-gnu/libstb.so.0
#5 0x00007ffff7ced2da in ?? () from /lib/x86_64-linux-gnu/libOpen3D.so.0.14
#6 0x00007ffff7ceb0ed in ImFontAtlas::GetTexDataAsAlpha8(unsigned char**, int*, int*, int*) () from /lib/x86_64-linux-gnu/libOpen3D.so.0.14
#7 0x00007ffff7ca88bf in open3d::visualization::gui::Window::CreateRenderer() () from /lib/x86_64-linux-gnu/libOpen3D.so.0.14
#8 0x00007ffff7ca95fa in open3d::visualization::gui::Window::Window(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int, int, int) () from /lib/x86_64-linux-gnu/libOpen3D.so.0.14
#9 0x00007ffff7ca968f in open3d::visualization::gui::Window::Window(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int) ()
   from /lib/x86_64-linux-gnu/libOpen3D.so.0.14
#10 0x00007ffff7c34aed in open3d::visualization::GuiVisualizer::GuiVisualizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)
    () from /lib/x86_64-linux-gnu/libOpen3D.so.0.14
#11 0x00005555555567e0 in ?? ()
#12 0x00007ffff6edfd90 in __libc_start_call_main (main=main@entry=0x555555556440, argc=argc@entry=1, argv=argv@entry=0x7fffffffe6a8) at ../sysdeps/nptl/libc_start_call_main.h:58
#13 0x00007ffff6edfe40 in __libc_start_main_impl (main=0x555555556440, argc=1, argv=0x7fffffffe6a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=0x7fffffffe698) at ../csu/libc-start.c:392
#14 0x0000555555556655 in ?? ()

I don't know all that much about glibc's malloc implementation but maybe it's heap corruption or something?

[Michael Hudson-Doyle (mwhudson)]

I spent a bit too long on this and I think this valgrind error is the core of the issue:

==62437== Invalid write of size 4
==62437== at 0x685A6C0: stbrp_pack_rects (stb_rect_pack.h:548)
==62437== by 0x522A2D9: ImFontAtlasBuildWithStbTruetype(ImFontAtlas*) (in /usr/lib/x86_64-linux-gnu/libOpen3D.so.0.14.1)
==62437== by 0x52280EC: ImFontAtlas::GetTexDataAsAlpha8(unsigned char**, int*, int*, int*) (in /usr/lib/x86_64-linux-gnu/libOpen3D.so.0.14.1)
==62437== by 0x51E58BE: CreateFonts (Window.cpp:117)
==62437== by 0x51E58BE: open3d::visualization::gui::Window::CreateRenderer() (Window.cpp:413)
==62437== by 0x51E65F9: open3d::visualization::gui::Window::Window(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int, int, int) (Window.cpp:396)
==62437== by 0x51E668E: open3d::visualization::gui::Window::Window(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int) (Window.cpp:262)
==62437== by 0x5171AEC: open3d::visualization::GuiVisualizer::GuiVisualizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int) (GuiVisualizer.cpp:627)
==62437== by 0x10A7DF: construct<open3d::visualization::GuiVisualizer, char const (&)[7], int, int> (new_allocator.h:162)
==62437== by 0x10A7DF: construct<open3d::visualization::GuiVisualizer, char const (&)[7], int, int> (alloc_traits.h:516)
==62437== by 0x10A7DF: _Sp_counted_ptr_inplace<char const (&)[7], int, int> (shared_ptr_base.h:519)
==62437== by 0x10A7DF: __shared_count<open3d::visualization::GuiVisualizer, std::allocator<open3d::visualization::GuiVisualizer>, char const (&)[7], int, int> (shared_ptr_base.h:650)
==62437== by 0x10A7DF: __shared_ptr<std::allocator<open3d::visualization::GuiVisualizer>, char const (&)[7], int, int> (shared_ptr_base.h:1342)
==62437== by 0x10A7DF: shared_ptr<std::allocator<open3d::visualization::GuiVisualizer>, char const (&)[7], int, int> (shared_ptr.h:409)
==62437== by 0x10A7DF: allocate_shared<open3d::visualization::GuiVisualizer, std::allocator<open3d::visualization::GuiVisualizer>, char const (&)[7], int, int> (shared_ptr.h:863)
==62437== by 0x10A7DF: make_shared<open3d::visualization::GuiVisualizer, char const (&)[7], int, int> (shared_ptr.h:879)
==62437== by 0x10A7DF: Run(int, char const**) (Open3DViewer.cpp:55)
==62437== by 0x5769D8F: (below main) (libc_start_call_main.h:58)
==62437== Address 0x1b4f33ac is 12 bytes after a block of size 3,056 alloc'd
==62437== at 0x4847899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==62437== by 0x522AA87: ImFontAtlasBuildWithStbTruetype(ImFontAtlas*) (in /usr/lib/x86_64-linux-gnu/libOpen3D.so.0.14.1)
==62437== by 0x52280EC: ImFontAtlas::GetTexDataAsAlpha8(unsigned char**, int*, int*, int*) (in /usr/lib/x86_64-linux-gnu/libOpen3D.so.0.14.1)
==62437== by 0x51E58BE: CreateFonts (Window.cpp:117)
==62437== by 0x51E58BE: open3d::visualization::gui::Window::CreateRenderer() (Window.cpp:413)
==62437== by 0x51E65F9: open3d::visualization::gui::Window::Window(std::__cxx11::basic_string<char, std::char_traits<char>, std::a...

[Jochen Sprickerhof (v-launchpad-jochen-sprickerhof-de)]

Hi Michael,

> tl;dr: probably an imgui bug causing an out of bounds write trashing
> malloc's internal data structures

Thanks for looking into it, sounds right.

I found that imgui in jammy is compiled with LTO and then statically linked
into open3d. Recompiling open3d (in jammy) with the imgui from Debian fixes
the problem. So I would propose to add imgui to the lto-disabled-list and
recompile open3d afterwards. What do you think?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Michael Hudson-Doyle (mwhudson)]

> So I would propose to add imgui to the lto-disabled-list and
> recompile open3d afterwards. What do you think?

Sounds plausible. I'll try it in my PPA first though!

[Launchpad Janitor (janitor)]

This bug was fixed in the package lto-disabled-list - 23

---------------
lto-disabled-list (23) jammy; urgency=medium

  * Add imgui to list as linking the static library produced with LTO enabled
    into open3d crashes. (LP: #1963556)

 -- Michael Hudson-Doyle <email address hidden> Tue, 08 Mar 2022 16:44:21 +1300