Comment 22 for bug 1664302

Now it gets interesting: On the same machine, I just got this crash from caget:

Program terminated with signal 11, Segmentation fault.
#0 remove (this=0x1dc3040, mfmdpv=..., pChannelName=<value optimized out>, pAcc=<value optimized out>, pRej=<value optimized out>)
    at ../../../include/tsDLList.h:236
236 prevNode.pNext = theNode.pNext;
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.192.el6.x86_64 libgcc-4.4.7-17.el6.x86_64 libstdc++-4.4.7-17.el6.x86_64 ncurses-libs-5.7-4.20090207.el6.x86_64 readline-6.0-4.el6.x86_64
(gdb) bt
#0 remove (this=0x1dc3040, mfmdpv=..., pChannelName=<value optimized out>, pAcc=<value optimized out>, pRej=<value optimized out>)
    at ../../../include/tsDLList.h:236
#1 cac::pvMultiplyDefinedNotify (this=0x1dc3040, mfmdpv=..., pChannelName=<value optimized out>, pAcc=<value optimized out>, pRej=<value optimized out>)
    at ../cac.cpp:1309
#2 0x00007fe4b476b01e in ipAddrToAsciiEnginePrivate::run (this=0x1dc35d0) at ../../../src/libCom/misc/ipAddrToAsciiAsynchronous.cpp:276
#3 0x00007fe4b476d249 in epicsThreadCallEntryPoint (pPvt=0x1dc3a28) at ../../../src/libCom/osi/epicsThread.cpp:83
#4 0x00007fe4b4773ed3 in start_routine (arg=0x1dc3d90) at ../../../src/libCom/osi/os/posix/osdThread.c:389
#5 0x0000003111c07aa1 in start_thread () from /lib64/libpthread.so.0
#6 0x00000031114e8aad in clone () from /lib64/libc.so.6
(gdb) p prevNode
$1 = (tsDLNode<msgForMultiplyDefinedPV> &) @0x0: <error reading variable>
(gdb) p theNode
$2 = (tsDLNode<msgForMultiplyDefinedPV> &) @0x7fe4ac017038: {pNext = 0x7fe4ac0170d8, pPrev = 0x0}

I'm adding this here because of the similarity of the crash location in tsDLList.h, related to the node handling.

There are only two more threads, which makes this a lot easier to debug:
Thread 3 (Thread 0x7fe4b4734720 (LWP 14595)):
#0 0x0000003111c0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fe4b47747a9 in condWait (pevent=0x1dc3bd0) at ../../../src/libCom/osi/os/posix/osdEvent.c:75
#2 epicsEventWait (pevent=0x1dc3bd0) at ../../../src/libCom/osi/os/posix/osdEvent.c:137
#3 0x00007fe4b476dbac in epicsEvent::wait (this=<value optimized out>) at ../../../src/libCom/osi/epicsEvent.cpp:63
#4 0x00007fe4b476ad27 in ipAddrToAsciiTransactionPrivate::~ipAddrToAsciiTransactionPrivate (this=0x7fe4ac007d80, __in_chrg=<value optimized out>)
    at ../../../src/libCom/misc/ipAddrToAsciiAsynchronous.cpp:319
#5 0x00007fe4b476a96a in ipAddrToAsciiTransactionPrivate::release (this=0x7fe4ac007d80) at ../../../src/libCom/misc/ipAddrToAsciiAsynchronous.cpp:302
#6 0x00007fe4b49e19e2 in msgForMultiplyDefinedPV::~msgForMultiplyDefinedPV (this=0x7fe4ac017030, __in_chrg=<value optimized out>)
    at ../msgForMultiplyDefinedPV.cpp:53
#7 0x00007fe4b49be706 in cac::~cac (this=0x1dc3040, __in_chrg=<value optimized out>) at ../cac.cpp:338
#8 0x00007fe4b49bebb9 in cac::~cac (this=0x1dc3040, __in_chrg=<value optimized out>) at ../cac.cpp:349
#9 0x00007fe4b49d8aee in destroyTarget (this=0x1dc2c50, __in_chrg=<value optimized out>) at ../../../include/epicsMemory.h:52
#10 reset (this=0x1dc2c50, __in_chrg=<value optimized out>) at ../../../include/epicsMemory.h:111
#11 ca_client_context::~ca_client_context (this=0x1dc2c50, __in_chrg=<value optimized out>) at ../ca_client_context.cpp:188
#12 0x00007fe4b49d8eb9 in ca_client_context::~ca_client_context (this=0x1dc2c50, __in_chrg=<value optimized out>) at ../ca_client_context.cpp:193
#13 0x00007fe4b49c1d43 in ca_context_destroy () at ../access.cpp:252
#14 0x0000000000401c40 in main (argc=<value optimized out>, argv=<value optimized out>) at ../caget.c:551

Thread 2 (Thread 0x7fe4b3b30700 (LWP 14604)):
#0 0x0000003111c0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fe4b4774704 in condTimedwait (pevent=0x7fe4a4002900, timeout=0) at ../../../src/libCom/osi/os/posix/osdEvent.c:65
#2 epicsEventWaitWithTimeout (pevent=0x7fe4a4002900, timeout=0) at ../../../src/libCom/osi/os/posix/osdEvent.c:156
#3 0x00007fe4b475ec76 in errlogThread () at ../../../src/libCom/error/errlog.c:507
#4 0x00007fe4b4773ed3 in start_routine (arg=0x7fe4a4005cc0) at ../../../src/libCom/osi/os/posix/osdThread.c:389
#5 0x0000003111c07aa1 in start_thread () from /lib64/libpthread.so.0
#6 0x00000031114e8aad in clone () from /lib64/libc.so.6

I can actually reproduce this by just repeating the caget enough times. The PV is nothing special, just plain ao, r/w access.