Comment 3 for bug 1963556

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

I spent a bit too long on this and I think this valgrind error is the core of the issue:

==62437== Invalid write of size 4
==62437== at 0x685A6C0: stbrp_pack_rects (stb_rect_pack.h:548)
==62437== by 0x522A2D9: ImFontAtlasBuildWithStbTruetype(ImFontAtlas*) (in /usr/lib/x86_64-linux-gnu/libOpen3D.so.0.14.1)
==62437== by 0x52280EC: ImFontAtlas::GetTexDataAsAlpha8(unsigned char**, int*, int*, int*) (in /usr/lib/x86_64-linux-gnu/libOpen3D.so.0.14.1)
==62437== by 0x51E58BE: CreateFonts (Window.cpp:117)
==62437== by 0x51E58BE: open3d::visualization::gui::Window::CreateRenderer() (Window.cpp:413)
==62437== by 0x51E65F9: open3d::visualization::gui::Window::Window(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int, int, int) (Window.cpp:396)
==62437== by 0x51E668E: open3d::visualization::gui::Window::Window(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int) (Window.cpp:262)
==62437== by 0x5171AEC: open3d::visualization::GuiVisualizer::GuiVisualizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int) (GuiVisualizer.cpp:627)
==62437== by 0x10A7DF: construct<open3d::visualization::GuiVisualizer, char const (&)[7], int, int> (new_allocator.h:162)
==62437== by 0x10A7DF: construct<open3d::visualization::GuiVisualizer, char const (&)[7], int, int> (alloc_traits.h:516)
==62437== by 0x10A7DF: _Sp_counted_ptr_inplace<char const (&)[7], int, int> (shared_ptr_base.h:519)
==62437== by 0x10A7DF: __shared_count<open3d::visualization::GuiVisualizer, std::allocator<open3d::visualization::GuiVisualizer>, char const (&)[7], int, int> (shared_ptr_base.h:650)
==62437== by 0x10A7DF: __shared_ptr<std::allocator<open3d::visualization::GuiVisualizer>, char const (&)[7], int, int> (shared_ptr_base.h:1342)
==62437== by 0x10A7DF: shared_ptr<std::allocator<open3d::visualization::GuiVisualizer>, char const (&)[7], int, int> (shared_ptr.h:409)
==62437== by 0x10A7DF: allocate_shared<open3d::visualization::GuiVisualizer, std::allocator<open3d::visualization::GuiVisualizer>, char const (&)[7], int, int> (shared_ptr.h:863)
==62437== by 0x10A7DF: make_shared<open3d::visualization::GuiVisualizer, char const (&)[7], int, int> (shared_ptr.h:879)
==62437== by 0x10A7DF: Run(int, char const**) (Open3DViewer.cpp:55)
==62437== by 0x5769D8F: (below main) (libc_start_call_main.h:58)
==62437== Address 0x1b4f33ac is 12 bytes after a block of size 3,056 alloc'd
==62437== at 0x4847899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==62437== by 0x522AA87: ImFontAtlasBuildWithStbTruetype(ImFontAtlas*) (in /usr/lib/x86_64-linux-gnu/libOpen3D.so.0.14.1)
==62437== by 0x52280EC: ImFontAtlas::GetTexDataAsAlpha8(unsigned char**, int*, int*, int*) (in /usr/lib/x86_64-linux-gnu/libOpen3D.so.0.14.1)
==62437== by 0x51E58BE: CreateFonts (Window.cpp:117)
==62437== by 0x51E58BE: open3d::visualization::gui::Window::CreateRenderer() (Window.cpp:413)
==62437== by 0x51E65F9: open3d::visualization::gui::Window::Window(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int, int, int) (Window.cpp:396)

The call to stbrp_pack_rects is here https://github.com/ocornut/imgui/blob/v1.86/imgui_draw.cpp#L2510 and the buffer that we're running off the end of is allocated here https://github.com/ocornut/imgui/blob/v1.86/imgui_draw.cpp#L2441 and the logic for how big that buffer is looks more or less OK to me but I've never seen this code before so take that with a big fat grain of salt :)

tl;dr: probably an imgui bug causing an out of bounds write trashing malloc's internal data structures