Comment 4 for bug 1951126

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Debugging confirms this, tgt uses dlopen to load its optional storage modules and from there into glusterfs the it crashes

(gdb) frame 20
#20 0x0000555555596d3c in bs_init_signalfd () at bs.c:286
(gdb) p soname
$1 = 0x555555dbe150 "/usr/lib/tgt/backing-store/bs_glfs.so"
(gdb) l

This is slightly more tricky in this case, as there are two conditions that can break it:
1. loading tcmalloc late conflicting with existing code (that was LP: #1950777 Debian #999700
2. But on x86 in the cases we had #1 didn't happen, yet there is another problem and that is when multiple libs want to insert tcmalloc (which happens here)

Ceph (via tgt-rbd) already can use tcmalloc and tgt is ready to tolerate that.
But not ALSO glusterfs uses tcmalloc.
So effectively tgt does dlopen on its libs which now both pull in tcmalloc.

We can see that in GDB
(gdb) b dlopen
Breakpoint 1 at 0x14620
(gdb) run -f
Starting program: /usr/sbin/tgtd -f
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules?
tgtd: work_timer_start(146) use timer_fd based scheduler

Breakpoint 1, ___dlopen (file=0x5555555be4a0 "/usr/lib/tgt/backing-store/bs_glfs.so", mode=mode@entry=2) at dlopen.c:77
77 dlopen.c: No such file or directory.
(gdb) c
Continuing.

Breakpoint 1, ___dlopen (file=0x5555555be4a0 "/usr/lib/tgt/backing-store/bs_rbd.so", mode=mode@entry=2) at dlopen.c:77
77 in dlopen.c
(gdb) c
Continuing.
src/tcmalloc.cc:333] Attempt to free invalid pointer 0x5555555ecae0

And since this is name-based ordering we can turn the order around:

$ mv /usr/lib/tgt/backing-store/bs_glfs.so /usr/lib/tgt/backing-store/bs_xxglfs.so

And now:

(gdb) b dlopen
Breakpoint 1 at 0x14620
(gdb) run -f
Starting program: /usr/sbin/tgtd -f
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules?
tgtd: work_timer_start(146) use timer_fd based scheduler

Breakpoint 1, ___dlopen (file=0x5555555be4a0 "/usr/lib/tgt/backing-store/bs_xxglfs.so", mode=mode@entry=2) at dlopen.c:77
77 dlopen.c: No such file or directory.
(gdb) c
Continuing.

Breakpoint 1, ___dlopen (file=0x5555555be4a0 "/usr/lib/tgt/backing-store/bs_rbd.so", mode=mode@entry=2) at dlopen.c:77
77 in dlopen.c
(gdb) c
Continuing.
src/tcmalloc.cc:333] Attempt to free invalid pointer 0x5555555ecae0