Debugging confirms this, tgt uses dlopen to load its optional storage modules and from there into glusterfs the it crashes
(gdb) frame 20
#20 0x0000555555596d3c in bs_init_signalfd () at bs.c:286
(gdb) p soname
$1 = 0x555555dbe150 "/usr/lib/tgt/backing-store/bs_glfs.so"
(gdb) l
This is slightly more tricky in this case, as there are two conditions that can break it:
1. loading tcmalloc late conflicting with existing code (that was LP: #1950777 Debian #999700
2. But on x86 in the cases we had #1 didn't happen, yet there is another problem and that is when multiple libs want to insert tcmalloc (which happens here)
Ceph (via tgt-rbd) already can use tcmalloc and tgt is ready to tolerate that.
But not ALSO glusterfs uses tcmalloc.
So effectively tgt does dlopen on its libs which now both pull in tcmalloc.
We can see that in GDB
(gdb) b dlopen
Breakpoint 1 at 0x14620
(gdb) run -f
Starting program: /usr/sbin/tgtd -f
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules?
tgtd: work_timer_start(146) use timer_fd based scheduler
Breakpoint 1, ___dlopen (file=0x5555555be4a0 "/usr/lib/tgt/backing-store/bs_glfs.so", mode=mode@entry=2) at dlopen.c:77
77 dlopen.c: No such file or directory.
(gdb) c
Continuing.
Breakpoint 1, ___dlopen (file=0x5555555be4a0 "/usr/lib/tgt/backing-store/bs_rbd.so", mode=mode@entry=2) at dlopen.c:77
77 in dlopen.c
(gdb) c
Continuing.
src/tcmalloc.cc:333] Attempt to free invalid pointer 0x5555555ecae0
And since this is name-based ordering we can turn the order around:
(gdb) b dlopen
Breakpoint 1 at 0x14620
(gdb) run -f
Starting program: /usr/sbin/tgtd -f
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules?
tgtd: work_timer_start(146) use timer_fd based scheduler
Breakpoint 1, ___dlopen (file=0x5555555be4a0 "/usr/lib/tgt/backing-store/bs_xxglfs.so", mode=mode@entry=2) at dlopen.c:77
77 dlopen.c: No such file or directory.
(gdb) c
Continuing.
Breakpoint 1, ___dlopen (file=0x5555555be4a0 "/usr/lib/tgt/backing-store/bs_rbd.so", mode=mode@entry=2) at dlopen.c:77
77 in dlopen.c
(gdb) c
Continuing.
src/tcmalloc.cc:333] Attempt to free invalid pointer 0x5555555ecae0
Debugging confirms this, tgt uses dlopen to load its optional storage modules and from there into glusterfs the it crashes
(gdb) frame 20 tgt/backing- store/bs_ glfs.so"
#20 0x0000555555596d3c in bs_init_signalfd () at bs.c:286
(gdb) p soname
$1 = 0x555555dbe150 "/usr/lib/
(gdb) l
This is slightly more tricky in this case, as there are two conditions that can break it:
1. loading tcmalloc late conflicting with existing code (that was LP: #1950777 Debian #999700
2. But on x86 in the cases we had #1 didn't happen, yet there is another problem and that is when multiple libs want to insert tcmalloc (which happens here)
Ceph (via tgt-rbd) already can use tcmalloc and tgt is ready to tolerate that.
But not ALSO glusterfs uses tcmalloc.
So effectively tgt does dlopen on its libs which now both pull in tcmalloc.
We can see that in GDB 64-linux- gnu/libthread_ db.so.1" . start(146) use timer_fd based scheduler
(gdb) b dlopen
Breakpoint 1 at 0x14620
(gdb) run -f
Starting program: /usr/sbin/tgtd -f
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_
tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules?
tgtd: work_timer_
Breakpoint 1, ___dlopen (file=0x5555555 be4a0 "/usr/lib/ tgt/backing- store/bs_ glfs.so" , mode=mode@entry=2) at dlopen.c:77
77 dlopen.c: No such file or directory.
(gdb) c
Continuing.
Breakpoint 1, ___dlopen (file=0x5555555 be4a0 "/usr/lib/ tgt/backing- store/bs_ rbd.so" , mode=mode@entry=2) at dlopen.c:77 cc:333] Attempt to free invalid pointer 0x5555555ecae0
77 in dlopen.c
(gdb) c
Continuing.
src/tcmalloc.
And since this is name-based ordering we can turn the order around:
$ mv /usr/lib/ tgt/backing- store/bs_ glfs.so /usr/lib/ tgt/backing- store/bs_ xxglfs. so
And now:
(gdb) b dlopen 64-linux- gnu/libthread_ db.so.1" . start(146) use timer_fd based scheduler
Breakpoint 1 at 0x14620
(gdb) run -f
Starting program: /usr/sbin/tgtd -f
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_
tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules?
tgtd: work_timer_
Breakpoint 1, ___dlopen (file=0x5555555 be4a0 "/usr/lib/ tgt/backing- store/bs_ xxglfs. so", mode=mode@entry=2) at dlopen.c:77
77 dlopen.c: No such file or directory.
(gdb) c
Continuing.
Breakpoint 1, ___dlopen (file=0x5555555 be4a0 "/usr/lib/ tgt/backing- store/bs_ rbd.so" , mode=mode@entry=2) at dlopen.c:77 cc:333] Attempt to free invalid pointer 0x5555555ecae0
77 in dlopen.c
(gdb) c
Continuing.
src/tcmalloc.