orderly shutdown for soft IOC fails

[Jeff Hill]

From Mark Rivers:

I've run my simulation IOC detector and typed "exit". It hung up, but there was no error. This is completely reproducible, it never exits correctly.

This is on the same multi-processor Linux system with recent GCC that had the problem with epicsRingPointer.

Additional information:
I then attached to that process with gdb and did a bt for all threads.

I'm not really sure what to look for, but the Thread 1 and 11 look suspicious? Thread 1 seems to be in dbCaShutdown, waiting for an epicsEvent, and Thread 11 is waiting for a lock in ca_context_destroy?

Thread 1 (Thread 0xb808d6d0 (LWP 22878)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0x9f07740) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x0815f04e in dbCaShutdown (arg=0x0) at ../dbCa.c:179
#4 0x081b5f11 in epicsExitCallAtExits () at ../../../src/libCom/misc/epicsExit.c:80
#5 0x081c03b5 in myAtExit () at ../../../src/libCom/osi/os/posix/osdThread.c:127
#6 0x00225b39 in exit () from /lib/libc.so.6
#7 0x0020e5de in __libc_start_main () from /lib/libc.so.6
#8 0x08051231 in _start ()

Thread 11 (Thread 0xb79d1b90 (LWP 22898)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bfa19 in __lll_robust_lock_wait () from /lib/libpthread.so.0
#2 0x003bad95 in _L_robust_lock_335 () from /lib/libpthread.so.0
#3 0x003ba9a9 in pthread_mutex_lock () from /lib/libpthread.so.0
#4 0x081c1210 in epicsMutexOsdLock (pmutex=0xa049d98) at ../../../src/libCom/osi/os/posix/osdMutex.c:45
#5 0x081b99c0 in epicsMutexLock (pmutexNode=0xa01be68) at ../../../src/libCom/osi/epicsMutex.cpp:145
#6 0x081b99ef in epicsMutex::lock (this=0x9e915f8) at ../../../src/libCom/osi/epicsMutex.cpp:238
#7 0x081c51fc in epicsSingleton<timerQueueActiveMgr>::getReference (this=0x8218eb0) at ../../../src/libCom/cxxTemplates/epicsGuard.h:71
#8 0x081c4367 in epicsTimerQueueActiveForC::release (this=0xa074670) at ../../../src/libCom/timer/epicsTimer.cpp:79
#9 0x0818b908 in ~cac (this=0xa067ee8) at ../cac.cpp:287
#10 0x08162dbf in ~dbContext (this=0x9f07af8) at ../../../include/epicsMemory.h:52
#11 0x0817f01a in ~ca_client_context (this=0x9f07930) at ../../../include/epicsMemory.h:52
#12 0x08176f09 in ca_context_destroy () at ../access.cpp:251
#13 0x0815fc9a in dbCaTask (arg=0x0) at ../dbCa.c:986
#14 0x081bf9f8 in start_routine (arg=0x9f07790) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#15 0x003b932f in start_thread () from /lib/libpthread.so.0
#16 0x002d620e in clone () from /lib/libc.so.6

Here is the complete output:

(gdb) thread apply all bt

Thread 30 (Thread 0xb808cb90 (LWP 22879)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0x9e7d120) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x081ad503 in errlogThread () at ../../../src/libCom/error/errlog.c:483
#4 0x081bf9f8 in start_routine (arg=0x9e7d048) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 29 (Thread 0xb806bb90 (LWP 22880)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bced2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1a49 in epicsEventWaitWithTimeout (pevent=0x9e83990, timeout=1.7976931348623157e+308)
    at ../../../src/libCom/osi/os/posix/osdEvent.c:67
#3 0x081ba5b6 in epicsEvent::wait (this=0x9e8389c, timeOut=1.7976931348623157e+308) at ../../../src/libCom/osi/epicsEvent.cpp:72
#4 0x081c62f1 in timerQueueActive::run (this=0x9e83848) at ../../../src/libCom/timer/timerQueueActive.cpp:70
#5 0x081b96e2 in epicsThreadCallEntryPoint (pPvt=0x9e838a4) at ../../../src/libCom/osi/epicsThread.cpp:42
#6 0x081bf9f8 in start_routine (arg=0x9e83b08) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#7 0x003b932f in start_thread () from /lib/libpthread.so.0
#8 0x002d620e in clone () from /lib/libc.so.6

Thread 28 (Thread 0xb802ab90 (LWP 22881)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0x9ef82d8) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x080f24b7 in portThread (pport=0x9ef7fb8) at ../../asyn/asynDriver/asynManager.c:671
#4 0x081bf9f8 in start_routine (arg=0x9ef8328) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 27 (Thread 0xb7dffb90 (LWP 22882)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bced2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1a49 in epicsEventWaitWithTimeout (pevent=0x9e82378, timeout=6) at ../../../src/libCom/osi/os/posix/osdEvent.c:67
#3 0x081c3168 in twdTask (arg=0x0) at ../../../src/libCom/taskwd/taskwd.c:125
#4 0x081bf9f8 in start_routine (arg=0x9e82418) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 26 (Thread 0xb7f9db90 (LWP 22883)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0xb7e00c70) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x0805346b in simDetector::simTask (this=0x9e91410) at ../simDetector.cpp:325
#4 0x0805391d in simTaskC (drvPvt=0x9e91410) at ../simDetector.cpp:291
#5 0x081bf9f8 in start_routine (arg=0xb7e00e10) at ../../../src/libCom/osi/os/posix/osdThread.c:319
---Type <return> to continue, or q <return> to quit---
#6 0x003b932f in start_thread () from /lib/libpthread.so.0
#7 0x002d620e in clone () from /lib/libc.so.6

Thread 25 (Thread 0xb7f5cb90 (LWP 22884)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0xb7e01498) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x080f24b7 in portThread (pport=0xb7e177d8) at ../../asyn/asynDriver/asynManager.c:671
#4 0x081bf9f8 in start_routine (arg=0xb7e17940) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 24 (Thread 0xb7ddeb90 (LWP 22885)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0xb7e02138) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x0805346b in simDetector::simTask (this=0xb7e011a8) at ../simDetector.cpp:325
#4 0x0805391d in simTaskC (drvPvt=0xb7e011a8) at ../simDetector.cpp:291
#5 0x081bf9f8 in start_routine (arg=0xb7e022d8) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#6 0x003b932f in start_thread () from /lib/libpthread.so.0
#7 0x002d620e in clone () from /lib/libc.so.6

Thread 23 (Thread 0xb7d9db90 (LWP 22886)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0xb7e39698) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x080f24b7 in portThread (pport=0xb7e39530) at ../../asyn/asynDriver/asynManager.c:671
#4 0x081bf9f8 in start_routine (arg=0xb7e396e8) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 22 (Thread 0xb7d5cb90 (LWP 22887)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0x9e91318) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x081c25de in myReceive (pmsg=0xb7e02b90, message=0xb7d5c2c8, size=4, wait=true, haveTimeout=false, timeout=0)
    at ../../../src/libCom/osi/os/default/osdMessageQueue.cpp:289
#4 0x081c268e in epicsMessageQueueReceive (pmsg=0xb7e02b90, message=0xb7d5c2c8, size=4)
    at ../../../src/libCom/osi/os/default/osdMessageQueue.cpp:309
#5 0x0805c8ea in NDPluginDriver::processTask (this=0xb7e393d0) at ../NDPluginDriver.cpp:192
#6 0x0805b694 in processTask (drvPvt=0xb7e393d0) at ../NDPluginDriver.cpp:180
#7 0x081bf9f8 in start_routine (arg=0xb7e02c40) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#8 0x003b932f in start_thread () from /lib/libpthread.so.0
#9 0x002d620e in clone () from /lib/libc.so.6

Thread 21 (Thread 0xb7d1bb90 (LWP 22888)):
#0 0x00110416 in __kernel_vsyscall ()
---Type <return> to continue, or q <return> to quit---
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0xb7e40e18) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x080f24b7 in portThread (pport=0xb7e40cb0) at ../../asyn/asynDriver/asynManager.c:671
#4 0x081bf9f8 in start_routine (arg=0xb7e40e68) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 20 (Thread 0xb7cdab90 (LWP 22889)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0x9e825a8) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x081c25de in myReceive (pmsg=0xb7e288b8, message=0xb7cda2c8, size=4, wait=true, haveTimeout=false, timeout=0)
    at ../../../src/libCom/osi/os/default/osdMessageQueue.cpp:289
#4 0x081c268e in epicsMessageQueueReceive (pmsg=0xb7e288b8, message=0xb7cda2c8, size=4)
    at ../../../src/libCom/osi/os/default/osdMessageQueue.cpp:309
#5 0x0805c8ea in NDPluginDriver::processTask (this=0xb7e40b50) at ../NDPluginDriver.cpp:192
#6 0x0805b694 in processTask (drvPvt=0xb7e40b50) at ../NDPluginDriver.cpp:180
#7 0x081bf9f8 in start_routine (arg=0xb7e28958) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#8 0x003b932f in start_thread () from /lib/libpthread.so.0
#9 0x002d620e in clone () from /lib/libc.so.6

Thread 19 (Thread 0xb7c99b90 (LWP 22890)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0xb7e28ca8) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x080f24b7 in portThread (pport=0xb7e46f20) at ../../asyn/asynDriver/asynManager.c:671
#4 0x081bf9f8 in start_routine (arg=0xb7e47088) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 18 (Thread 0xb7c58b90 (LWP 22891)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0x9e82628) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x081c25de in myReceive (pmsg=0xb7e3aa60, message=0xb7c582c8, size=4, wait=true, haveTimeout=false, timeout=0)
    at ../../../src/libCom/osi/os/default/osdMessageQueue.cpp:289
#4 0x081c268e in epicsMessageQueueReceive (pmsg=0xb7e3aa60, message=0xb7c582c8, size=4)
    at ../../../src/libCom/osi/os/default/osdMessageQueue.cpp:309
#5 0x0805c8ea in NDPluginDriver::processTask (this=0xb7e46da0) at ../NDPluginDriver.cpp:192
#6 0x0805b694 in processTask (drvPvt=0xb7e46da0) at ../NDPluginDriver.cpp:180
#7 0x081bf9f8 in start_routine (arg=0xb7e3ab78) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#8 0x003b932f in start_thread () from /lib/libpthread.so.0
#9 0x002d620e in clone () from /lib/libc.so.6

Thread 17 (Thread 0xb7c17b90 (LWP 22892)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#2 0x081c1517 in epicsEventWait (pevent=0xb7e3aee8) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x080f24b7 in portThread (pport=0xb7e4d070) at ../../asyn/asynDriver/asynManager.c:671
#4 0x081bf9f8 in start_routine (arg=0xb7e4d1d8) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 16 (Thread 0xb7bd6b90 (LWP 22893)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0x9e826a8) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x081c25de in myReceive (pmsg=0xb7e54448, message=0xb7bd62c8, size=4, wait=true, haveTimeout=false, timeout=0)
    at ../../../src/libCom/osi/os/default/osdMessageQueue.cpp:289
#4 0x081c268e in epicsMessageQueueReceive (pmsg=0xb7e54448, message=0xb7bd62c8, size=4)
    at ../../../src/libCom/osi/os/default/osdMessageQueue.cpp:309
#5 0x0805c8ea in NDPluginDriver::processTask (this=0xb7e4cf00) at ../NDPluginDriver.cpp:192
#6 0x0805b694 in processTask (drvPvt=0xb7e4cf00) at ../NDPluginDriver.cpp:180
#7 0x081bf9f8 in start_routine (arg=0xb7e54548) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#8 0x003b932f in start_thread () from /lib/libpthread.so.0
#9 0x002d620e in clone () from /lib/libc.so.6

Thread 15 (Thread 0xb7b95b90 (LWP 22894)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bced2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1a49 in epicsEventWaitWithTimeout (pevent=0x9e830f8, timeout=299.99495557900002)
    at ../../../src/libCom/osi/os/posix/osdEvent.c:67
#3 0x081ba5b6 in epicsEvent::wait (this=0x9e8303c, timeOut=299.99495557900002) at ../../../src/libCom/osi/epicsEvent.cpp:72
#4 0x081c62f1 in timerQueueActive::run (this=0x9e82fe8) at ../../../src/libCom/timer/timerQueueActive.cpp:70
#5 0x081b96e2 in epicsThreadCallEntryPoint (pPvt=0x9e83044) at ../../../src/libCom/osi/epicsThread.cpp:42
#6 0x081bf9f8 in start_routine (arg=0x9e83250) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#7 0x003b932f in start_thread () from /lib/libpthread.so.0
#8 0x002d620e in clone () from /lib/libc.so.6

Thread 14 (Thread 0xb7b54b90 (LWP 22895)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0x9e833a8) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x0815d4f5 in callbackTask (arg=0x8211b6c) at ../callback.c:89
#4 0x081bf9f8 in start_routine (arg=0x9e83410) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 13 (Thread 0xb7ad3b90 (LWP 22896)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0x9e83568) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x0815d4f5 in callbackTask (arg=0x8211b70) at ../callback.c:89
#4 0x081bf9f8 in start_routine (arg=0x9e835d0) at ../../../src/libCom/osi/os/posix/osdThread.c:319
---Type <return> to continue, or q <return> to quit---
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 12 (Thread 0xb7a52b90 (LWP 22897)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0x9e83748) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x0815d4f5 in callbackTask (arg=0x8211b74) at ../callback.c:89
#4 0x081bf9f8 in start_routine (arg=0x9f074b8) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 11 (Thread 0xb79d1b90 (LWP 22898)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bfa19 in __lll_robust_lock_wait () from /lib/libpthread.so.0
#2 0x003bad95 in _L_robust_lock_335 () from /lib/libpthread.so.0
#3 0x003ba9a9 in pthread_mutex_lock () from /lib/libpthread.so.0
#4 0x081c1210 in epicsMutexOsdLock (pmutex=0xa049d98) at ../../../src/libCom/osi/os/posix/osdMutex.c:45
#5 0x081b99c0 in epicsMutexLock (pmutexNode=0xa01be68) at ../../../src/libCom/osi/epicsMutex.cpp:145
#6 0x081b99ef in epicsMutex::lock (this=0x9e915f8) at ../../../src/libCom/osi/epicsMutex.cpp:238
#7 0x081c51fc in epicsSingleton<timerQueueActiveMgr>::getReference (this=0x8218eb0) at ../../../src/libCom/cxxTemplates/epicsGuard.h:71
#8 0x081c4367 in epicsTimerQueueActiveForC::release (this=0xa074670) at ../../../src/libCom/timer/epicsTimer.cpp:79
#9 0x0818b908 in ~cac (this=0xa067ee8) at ../cac.cpp:287
#10 0x08162dbf in ~dbContext (this=0x9f07af8) at ../../../include/epicsMemory.h:52
#11 0x0817f01a in ~ca_client_context (this=0x9f07930) at ../../../include/epicsMemory.h:52
#12 0x08176f09 in ca_context_destroy () at ../access.cpp:251
#13 0x0815fc9a in dbCaTask (arg=0x0) at ../dbCa.c:986
#14 0x081bf9f8 in start_routine (arg=0x9f07790) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#15 0x003b932f in start_thread () from /lib/libpthread.so.0
#16 0x002d620e in clone () from /lib/libc.so.6

Thread 10 (Thread 0xb719cb90 (LWP 22909)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bfe58 in accept () from /lib/libpthread.so.0
#2 0x081bd9e3 in epicsSocketAccept (sock=7, pAddr=0xb719c2b0, addrlen=0xb719c2d0) at ../../../src/libCom/osi/os/posix/osdSock.c:99
#3 0x0813397f in req_server (pParm=0x0) at ../caservertask.c:194
#4 0x081bf9f8 in start_routine (arg=0xa04ab20) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 9 (Thread 0xb715bb90 (LWP 22910)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003c0436 in nanosleep () from /lib/libpthread.so.0
#2 0x081bf5db in epicsThreadSleep (seconds=15) at ../../../src/libCom/osi/os/posix/osdThread.c:609
#3 0x08139d86 in rsrv_online_notify_task (pParm=0x0) at ../online_notify.c:285
#4 0x081bf9f8 in start_routine (arg=0xa04acf8) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 8 (Thread 0xb713ab90 (LWP 22911)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bffd8 in recvfrom () from /lib/libpthread.so.0
#2 0x0813afba in cast_server (pParm=0x0) at ../cast_server.c:218
#3 0x081bf9f8 in start_routine (arg=0xa04aec0) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#4 0x003b932f in start_thread () from /lib/libpthread.so.0
#5 0x002d620e in clone () from /lib/libc.so.6

Thread 7 (Thread 0xb70f9b90 (LWP 22912)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003c0436 in nanosleep () from /lib/libpthread.so.0
#2 0x081bf5db in epicsThreadSleep (seconds=0.99999286700000001) at ../../../src/libCom/osi/os/posix/osdThread.c:609
#3 0x0817dfa9 in ca_client_context::pendEvent (this=0xa067c50, timeout=@0xb70f9188) at ../ca_client_context.cpp:583
#4 0x081777c6 in ca_pend_event (timeout=1) at ../access.cpp:454
#5 0x080eda58 in save_restore () at ../save_restore.c:827
#6 0x081bf9f8 in start_routine (arg=0xa04b9d8) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#7 0x003b932f in start_thread () from /lib/libpthread.so.0
#8 0x002d620e in clone () from /lib/libc.so.6

Thread 6 (Thread 0xb7078b90 (LWP 22913)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0xa0706a8) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x081ba62f in epicsEvent::wait (this=0xa070604) at ../../../src/libCom/osi/epicsEvent.cpp:63
#4 0x081b7392 in ipAddrToAsciiEnginePrivate::run (this=0xa0701e0) at ../../../src/libCom/misc/ipAddrToAsciiAsynchronous.cpp:305
#5 0x081b96e2 in epicsThreadCallEntryPoint (pPvt=0xa07060c) at ../../../src/libCom/osi/epicsThread.cpp:42
#6 0x081bf9f8 in start_routine (arg=0xa070820) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#7 0x003b932f in start_thread () from /lib/libpthread.so.0
#8 0x002d620e in clone () from /lib/libc.so.6

Thread 5 (Thread 0xb6ff7b90 (LWP 22914)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bced2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1a49 in epicsEventWaitWithTimeout (pevent=0xa0747f8, timeout=1.7976931348623157e+308)
    at ../../../src/libCom/osi/os/posix/osdEvent.c:67
#3 0x081ba5b6 in epicsEvent::wait (this=0xa0746c4, timeOut=1.7976931348623157e+308) at ../../../src/libCom/osi/epicsEvent.cpp:72
#4 0x081c62f1 in timerQueueActive::run (this=0xa074670) at ../../../src/libCom/timer/timerQueueActive.cpp:70
#5 0x081b96e2 in epicsThreadCallEntryPoint (pPvt=0xa0746cc) at ../../../src/libCom/osi/epicsThread.cpp:42
#6 0x081bf9f8 in start_routine (arg=0xa074970) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#7 0x003b932f in start_thread () from /lib/libpthread.so.0
#8 0x002d620e in clone () from /lib/libc.so.6

Thread 4 (Thread 0xb6df1b90 (LWP 22919)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#2 0x081c1517 in epicsEventWait (pevent=0xa10c058) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x08156421 in event_task (pParm=0xa0aafb8) at ../dbEvent.c:913
#4 0x081bf9f8 in start_routine (arg=0xa10c130) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 3 (Thread 0xb6d6fb90 (LWP 22920)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0xa12dbd8) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x08156421 in event_task (pParm=0xa0aa374) at ../dbEvent.c:913
#4 0x081bf9f8 in start_routine (arg=0xa12dcb0) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#5 0x003b932f in start_thread () from /lib/libpthread.so.0
#6 0x002d620e in clone () from /lib/libc.so.6

Thread 2 (Thread 0xb6d2eb90 (LWP 22921)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bff58 in recv () from /lib/libpthread.so.0
#2 0x0813460d in camsgtask (pParm=0xa04b450) at ../camsgtask.c:82
#3 0x081bf9f8 in start_routine (arg=0xa12ddf8) at ../../../src/libCom/osi/os/posix/osdThread.c:319
#4 0x003b932f in start_thread () from /lib/libpthread.so.0
#5 0x002d620e in clone () from /lib/libc.so.6

Thread 1 (Thread 0xb808d6d0 (LWP 22878)):
#0 0x00110416 in __kernel_vsyscall ()
#1 0x003bcba5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2 0x081c1517 in epicsEventWait (pevent=0x9f07740) at ../../../src/libCom/osi/os/posix/osdEvent.c:77
#3 0x0815f04e in dbCaShutdown (arg=0x0) at ../dbCa.c:179
#4 0x081b5f11 in epicsExitCallAtExits () at ../../../src/libCom/misc/epicsExit.c:80
#5 0x081c03b5 in myAtExit () at ../../../src/libCom/osi/os/posix/osdThread.c:127
#6 0x00225b39 in exit () from /lib/libc.so.6
#7 0x0020e5de in __libc_start_main () from /lib/libc.so.6
#8 0x08051231 in _start ()

OS: SMP Linux, recent gcc

Version: R3.14.10

Original Mantis Bug: mantis-328
    http://www.aps.anl.gov/epics/mantis/view_bug_page.php?f_id=328

[Jeff Hill (johill-lanl)]

This is what is occurring.

1) In Stroustrup "The C++ Programming Language" section 7.1.2 it says that C++ static variables, which are local to a function, are initialized only "the first time the thread of execution reaches its definition."

2) I see at this web site, <a href="http://publib.boulder.ibm.com/infocenter/zos/v1r9/index.jsp?topic=/com.ibm.zos.r9.bpxbd00/e016d.htm">http://publib.boulder.ibm.com/infocenter/zos/v1r9/index.jsp?topic=/com.ibm.zos.r9.bpxbd00/e016d.htm</a>, that an implementation might be calling atexit at the same place in the execution that the implementation is calling the, local to a function, static variable's ctor to establish an exit handler that runs this function-local static variable's dtor.

So probably this happened:

O iocInit calls atexit to enroll the epics-at-exit master atexit handler O Later on epicsSingletonPrivateMutex() is called for the first time and that causes the ctor for its local static variable "epicsSingletonMutex" to be called. At that same point in the execution the atexit handler that runs the dtor for "epicsSingletonMutex" is also installed.

Later in the execution the IOC shell exits. This causes exit () to be called and this happens.

O The atexit handler that runs the dtor for "epicsSingletonMutex" is called. As expected, this deletes the mutex.
O The epics-at-exit master atexit handler runs O The db ca link rundown calls ca_context_destroy() O Eventually epicsSingleton<T>::getReference() is called, and it tries to take ownership "epicsSingletonMutex" which has already been destroyed

RIP

[Jeff Hill (johill-lanl)]

committed a fix (but it isnt easy to test here because I have an old versionof linux)

[Andrew Johnson (anj)]

Hopefully fixed in R3.14.11.

[Andrew Johnson (anj)]

R3.14.11 released.