I spent a bit too long on this and I think this valgrind error is the core of the issue:
==62437== Invalid write of size 4 ==62437== at 0x685A6C0: stbrp_pack_rects (stb_rect_pack.h:548) ==62437== by 0x522A2D9: ImFontAtlasBuildWithStbTruetype(ImFontAtlas*) (in /usr/lib/x86_64-linux-gnu/libOpen3D.so.0.14.1) ==62437== by 0x52280EC: ImFontAtlas::GetTexDataAsAlpha8(unsigned char**, int*, int*, int*) (in /usr/lib/x86_64-linux-gnu/libOpen3D.so.0.14.1) ==62437== by 0x51E58BE: CreateFonts (Window.cpp:117) ==62437== by 0x51E58BE: open3d::visualization::gui::Window::CreateRenderer() (Window.cpp:413) ==62437== by 0x51E65F9: open3d::visualization::gui::Window::Window(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int, int, int) (Window.cpp:396) ==62437== by 0x51E668E: open3d::visualization::gui::Window::Window(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int) (Window.cpp:262) ==62437== by 0x5171AEC: open3d::visualization::GuiVisualizer::GuiVisualizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int) (GuiVisualizer.cpp:627) ==62437== by 0x10A7DF: construct<open3d::visualization::GuiVisualizer, char const (&)[7], int, int> (new_allocator.h:162) ==62437== by 0x10A7DF: construct<open3d::visualization::GuiVisualizer, char const (&)[7], int, int> (alloc_traits.h:516) ==62437== by 0x10A7DF: _Sp_counted_ptr_inplace<char const (&)[7], int, int> (shared_ptr_base.h:519) ==62437== by 0x10A7DF: __shared_count<open3d::visualization::GuiVisualizer, std::allocator<open3d::visualization::GuiVisualizer>, char const (&)[7], int, int> (shared_ptr_base.h:650) ==62437== by 0x10A7DF: __shared_ptr<std::allocator<open3d::visualization::GuiVisualizer>, char const (&)[7], int, int> (shared_ptr_base.h:1342) ==62437== by 0x10A7DF: shared_ptr<std::allocator<open3d::visualization::GuiVisualizer>, char const (&)[7], int, int> (shared_ptr.h:409) ==62437== by 0x10A7DF: allocate_shared<open3d::visualization::GuiVisualizer, std::allocator<open3d::visualization::GuiVisualizer>, char const (&)[7], int, int> (shared_ptr.h:863) ==62437== by 0x10A7DF: make_shared<open3d::visualization::GuiVisualizer, char const (&)[7], int, int> (shared_ptr.h:879) ==62437== by 0x10A7DF: Run(int, char const**) (Open3DViewer.cpp:55) ==62437== by 0x5769D8F: (below main) (libc_start_call_main.h:58) ==62437== Address 0x1b4f33ac is 12 bytes after a block of size 3,056 alloc'd ==62437== at 0x4847899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==62437== by 0x522AA87: ImFontAtlasBuildWithStbTruetype(ImFontAtlas*) (in /usr/lib/x86_64-linux-gnu/libOpen3D.so.0.14.1) ==62437== by 0x52280EC: ImFontAtlas::GetTexDataAsAlpha8(unsigned char**, int*, int*, int*) (in /usr/lib/x86_64-linux-gnu/libOpen3D.so.0.14.1) ==62437== by 0x51E58BE: CreateFonts (Window.cpp:117) ==62437== by 0x51E58BE: open3d::visualization::gui::Window::CreateRenderer() (Window.cpp:413) ==62437== by 0x51E65F9: open3d::visualization::gui::Window::Window(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int, int, int) (Window.cpp:396)
The call to stbrp_pack_rects is here https://github.com/ocornut/imgui/blob/v1.86/imgui_draw.cpp#L2510 and the buffer that we're running off the end of is allocated here https://github.com/ocornut/imgui/blob/v1.86/imgui_draw.cpp#L2441 and the logic for how big that buffer is looks more or less OK to me but I've never seen this code before so take that with a big fat grain of salt :)
tl;dr: probably an imgui bug causing an out of bounds write trashing malloc's internal data structures
I spent a bit too long on this and I think this valgrind error is the core of the issue:
==62437== Invalid write of size 4 pack.h: 548) dWithStbTruetyp e(ImFontAtlas* ) (in /usr/lib/ x86_64- linux-gnu/ libOpen3D. so.0.14. 1) :GetTexDataAsAl pha8(unsigned char**, int*, int*, int*) (in /usr/lib/ x86_64- linux-gnu/ libOpen3D. so.0.14. 1) :visualization: :gui::Window: :CreateRenderer () (Window.cpp:413) :visualization: :gui::Window: :Window( std::__ cxx11:: basic_string< char, std::char_ traits< char>, std::allocator< char> > const&, int, int, int, int, int) (Window.cpp:396) :visualization: :gui::Window: :Window( std::__ cxx11:: basic_string< char, std::char_ traits< char>, std::allocator< char> > const&, int, int, int) (Window.cpp:262) :visualization: :GuiVisualizer: :GuiVisualizer( std::__ cxx11:: basic_string< char, std::char_ traits< char>, std::allocator< char> > const&, int, int) (GuiVisualizer. cpp:627) open3d: :visualization: :GuiVisualizer, char const (&)[7], int, int> (new_allocator. h:162) open3d: :visualization: :GuiVisualizer, char const (&)[7], int, int> (alloc_ traits. h:516) ptr_inplace< char const (&)[7], int, int> (shared_ ptr_base. h:519) count<open3d: :visualization: :GuiVisualizer, std::allocator< open3d: :visualization: :GuiVisualizer> , char const (&)[7], int, int> (shared_ ptr_base. h:650) ptr<std: :allocator< open3d: :visualization: :GuiVisualizer> , char const (&)[7], int, int> (shared_ ptr_base. h:1342) ptr<std: :allocator< open3d: :visualization: :GuiVisualizer> , char const (&)[7], int, int> (shared_ptr.h:409) shared< open3d: :visualization: :GuiVisualizer, std::allocator< open3d: :visualization: :GuiVisualizer> , char const (&)[7], int, int> (shared_ptr.h:863) open3d: :visualization: :GuiVisualizer, char const (&)[7], int, int> (shared_ptr.h:879) cpp:55) call_main. h:58) valgrind/ vgpreload_ memcheck- amd64-linux. so) dWithStbTruetyp e(ImFontAtlas* ) (in /usr/lib/ x86_64- linux-gnu/ libOpen3D. so.0.14. 1) :GetTexDataAsAl pha8(unsigned char**, int*, int*, int*) (in /usr/lib/ x86_64- linux-gnu/ libOpen3D. so.0.14. 1) :visualization: :gui::Window: :CreateRenderer () (Window.cpp:413) :visualization: :gui::Window: :Window( std::__ cxx11:: basic_string< char, std::char_ traits< char>, std::allocator< char> > const&, int, int, int, int, int) (Window.cpp:396)
==62437== at 0x685A6C0: stbrp_pack_rects (stb_rect_
==62437== by 0x522A2D9: ImFontAtlasBuil
==62437== by 0x52280EC: ImFontAtlas:
==62437== by 0x51E58BE: CreateFonts (Window.cpp:117)
==62437== by 0x51E58BE: open3d:
==62437== by 0x51E65F9: open3d:
==62437== by 0x51E668E: open3d:
==62437== by 0x5171AEC: open3d:
==62437== by 0x10A7DF: construct<
==62437== by 0x10A7DF: construct<
==62437== by 0x10A7DF: _Sp_counted_
==62437== by 0x10A7DF: __shared_
==62437== by 0x10A7DF: __shared_
==62437== by 0x10A7DF: shared_
==62437== by 0x10A7DF: allocate_
==62437== by 0x10A7DF: make_shared<
==62437== by 0x10A7DF: Run(int, char const**) (Open3DViewer.
==62437== by 0x5769D8F: (below main) (libc_start_
==62437== Address 0x1b4f33ac is 12 bytes after a block of size 3,056 alloc'd
==62437== at 0x4847899: malloc (in /usr/libexec/
==62437== by 0x522AA87: ImFontAtlasBuil
==62437== by 0x52280EC: ImFontAtlas:
==62437== by 0x51E58BE: CreateFonts (Window.cpp:117)
==62437== by 0x51E58BE: open3d:
==62437== by 0x51E65F9: open3d:
The call to stbrp_pack_rects is here https:/ /github. com/ocornut/ imgui/blob/ v1.86/imgui_ draw.cpp# L2510 and the buffer that we're running off the end of is allocated here https:/ /github. com/ocornut/ imgui/blob/ v1.86/imgui_ draw.cpp# L2441 and the logic for how big that buffer is looks more or less OK to me but I've never seen this code before so take that with a big fat grain of salt :)
tl;dr: probably an imgui bug causing an out of bounds write trashing malloc's internal data structures