IOC segfaults when calling dbLoadRecords after iocInit

[Martin Konrad]

Diego Omitto reported the following issue:

I know that we are not supposed to load records after the iocInit but I think it should not segFault.

$cat dummy.db
record(ai,"dummy"){}

$ softIoc
epics> iocInit
Starting iocInit
############################################################################
## EPICS R3.15.6-1+frib3+0~20190507132128.11+debian09~1.gbp4c10ca
## EPICS Base built May 7 2019
############################################################################
cas warning: Configured TCP port was unavailable.
cas warning: Using dynamically assigned TCP port 39385,
cas warning: but now two or more servers share the same UDP port.
cas warning: Depending on your IP kernel this server may not be
cas warning: reachable with UDP unicast (a host's IP in EPICS_CA_ADDR_LIST)
iocRun: All initialization complete
epics> dbLoadRecords("dummy.db")
epics> #caget dummy from another terminal
epics> Segmentation fault

$ gdb /usr/lib/epics/bin/linux-x86_64-debug/softIoc
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/epics/bin/linux-x86_64-debug/softIoc...done.
(gdb) r
Starting program: /usr/lib/epics/bin/linux-x86_64-debug/softIoc
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff5d51700 (LWP 6677)]
[Thread 0x7ffff5d51700 (LWP 6677) exited]
epics> iocInit
[New Thread 0x7ffff5550700 (LWP 6687)]
Starting iocInit
############################################################################
## EPICS R3.15.6-1+frib3+0~20190507132128.11+debian09~1.gbp4c10ca
## EPICS Base built May 7 2019
############################################################################
[New Thread 0x7ffff54cf700 (LWP 6688)]
[New Thread 0x7ffff544e700 (LWP 6689)]
[New Thread 0x7ffff5d51700 (LWP 6690)]
[New Thread 0x7ffff534d700 (LWP 6691)]
[New Thread 0x7ffff514c700 (LWP 6692)]
[New Thread 0x7ffff4f4b700 (LWP 6693)]
[New Thread 0x7ffff4d4a700 (LWP 6694)]
[New Thread 0x7ffff4b49700 (LWP 6695)]
[New Thread 0x7ffff4948700 (LWP 6696)]
[New Thread 0x7ffff4747700 (LWP 6697)]
[New Thread 0x7ffff4546700 (LWP 6698)]
[New Thread 0x7ffff4345700 (LWP 6699)]
[New Thread 0x7fffc7fff700 (LWP 6700)]
[New Thread 0x7fffc7dfe700 (LWP 6701)]
[New Thread 0x7ffff4144700 (LWP 6702)]
[New Thread 0x7fffc7bfd700 (LWP 6703)]
[New Thread 0x7fffc7afc700 (LWP 6704)]
iocRun: All initialization complete
epics> dbLoadRecords("dummy.db")
epics> dbgf dummy
dbgf only works after iocInit
epics> [New Thread 0x7fffc7a7b700 (LWP 6913)]
[New Thread 0x7fffc797a700 (LWP 6914)]

Thread 22 "CAS-client" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffc797a700 (LWP 6914)]
dbScanLock (precord=0x5555558525f0) at ../../../src/ioc/db/dbLock.c:249
249 ../../../src/ioc/db/dbLock.c: No such file or directory.
(gdb) bt
#0 dbScanLock (precord=0x5555558525f0) at ../../../src/ioc/db/dbLock.c:249
#1 0x00007ffff7946464 in dbChannel_get_count (chan=chan@entry=0x7fffac004cb8, buffer_type=20,
    pbuffer=0x7ffff403f020, nRequest=nRequest@entry=0x7fffc7979cc0, pfl=pfl@entry=0x0)
    at ../../../src/ioc/db/db_access.c:157
#2 0x00007ffff796d29a in read_reply (pfl=0x0, eventsRemaining=1, dbch=0x7fffac004cb8, pArg=0x7fffc7979cd0)
    at ../../../src/ioc/rsrv/camessage.c:587
#3 read_notify_action (mp=<optimized out>, pPayload=<optimized out>, client=<optimized out>)
    at ../../../src/ioc/rsrv/camessage.c:804
#4 0x00007ffff796e08f in camessage (client=client@entry=0x7fffb0000f50) at ../../../src/ioc/rsrv/camessage.c:2640
#5 0x00007ffff796a981 in camsgtask (pParm=0x7fffb0000f50) at ../../../src/ioc/rsrv/camsgtask.c:117
#6 0x00007ffff76db410 in start_routine (arg=0x7fffbc0056c0) at ../../../src/libCom/osi/os/posix/osdThread.c:403
#7 0x00007ffff6b754a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007ffff73e0d0f in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)

[Martin Konrad (info-martin-konrad)]

Hi Diego,
Thanks for reporting this issue. I have a patch that I believe fixes the
problem:

git clone -b fix-crash-dbloadrecords-after-iocinit
https://git.launchpad.net/~info-martin-konrad/epics-base

Could you please take it for a test drive?

Thanks,

Martin

--
Martin Konrad
Facility for Rare Isotope Beams
Michigan State University
640 South Shaw Lane
East Lansing, MI 48824-1321, USA
Tel. [phone]
Email: <email address hidden>

[Martin Konrad (info-martin-konrad)]

-------- Forwarded Message --------
Subject: Re: [Bug 1829919] [NEW] IOC segfaults when calling
dbLoadRecords after iocInit
Date: Thu, 14 Nov 2019 11:15:16 -0500
From: Omitto, Diego <email address hidden>
To: Konrad, Martin <email address hidden>
CC: Bug 1829919 <email address hidden>

Hi Martin,

it works for me!

Thanks for looking into that,

Diego

[mdavidsaver (mdavidsaver)]

Martin, rather than adding a new variable, maybe better to use "iocState" global in iocInit.c. (eg. via a accessor function added in iocInit.h). The "interruptAccept" global would also work practically, though it is set from iocRun() not iocBuild().

[Martin Konrad (info-martin-konrad)]

Michael, thanks for your feedback. I have updated my branch accordingly.

I also thought about using interruptAccept but unfortunately we need to prevent loading databases after iocBuild() to prevent this problem reliably.

[Martin Konrad (info-martin-konrad)]

MR: https://code.launchpad.net/~info-martin-konrad/epics-base/+git/epics-base/+merge/375559