citserver segfaults on startup on 20.04 server

[donr]

This is on a new install of Server-20.04 with little extra.

From syslog:

Jun 2 02:35:53 cittest2 kernel: [ 393.264246] citserver[2431]: segfault at 10000014c ip 00007f3dba7f7b2d sp 00007fff479ac2b0 error 4 in libdb-5.3.so[7f3dba727000+142000]

From gdb with dgbsym package:

Starting program: /usr/sbin/citserver -lmail
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
citserver[627176]:
citserver[627176]:
citserver[627176]: *** Citadel server engine ***
citserver[627176]: Version 917 (build b47178f) ***
citserver[627176]: Copyright (C) 1987-2018 by the Citadel development team.
citserver[627176]:
citserver[627176]: This program is open source software: you can redistribute it and/or
citserver[627176]: modify it under the terms of the GNU General Public License, version 3.
citserver[627176]:
citserver[627176]: This program is distributed in the hope that it will be useful,
citserver[627176]: but WITHOUT ANY WARRANTY; without even the implied warranty of
citserver[627176]: MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
citserver[627176]: GNU General Public License for more details.
citserver[627176]:
citserver[627176]: libcitadel(unnumbered)
citserver[627176]: Checking directory access
citserver[627176]: Opening databases
citserver[627176]: db: mounting databases
citserver[627176]: Initializing configuration system

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff77b3b2d in __db_check_txn () from /lib/x86_64-linux-gnu/libdb-5.3.so
(gdb) bt
#0 0x00007ffff77b3b2d in __db_check_txn () from /lib/x86_64-linux-gnu/libdb-5.3.so
#1 0x00007ffff77b8ee0 in __db_get_pp () from /lib/x86_64-linux-gnu/libdb-5.3.so
#2 0x000055555556ec9a in cdb_fetch (cdb=cdb@entry=13, key=key@entry=0x5555555d4b85,
    keylen=keylen@entry=28) at database.c:657
#3 0x00005555555723d7 in CtdlGetConfigStr (
    key=key@entry=0x5555555d4b85 "c_config_created_or_migrated") at config.c:429
#4 0x000055555557290d in CtdlGetConfigLong (
    key=key@entry=0x5555555d4b85 "c_config_created_or_migrated") at config.c:459
#5 0x0000555555572a3c in initialize_config_system () at config.c:284
#6 0x000055555556939d in master_startup () at citserver.c:150
#7 0x0000555555564ca7 in main (argc=2, argv=0x7fffffffec68) at server_main.c:260
(gdb) f 2
#2 0x000055555556ec9a in cdb_fetch (cdb=cdb@entry=13, key=key@entry=0x5555555d4b85,
    keylen=keylen@entry=28) at database.c:657
657 database.c: No such file or directory.
(gdb) p cdb
$1 = 13
(gdb) p MyThread()->tid
$4 = (DB_TXN *) 0x100000000
(gdb) p dkey
$2 = {data = 0x5555555d4b85, size = 28, ulen = 0, dlen = 0, doff = 0, app_data = 0x0, flags = 0}
(gdb) p dret
$3 = {data = 0x0, size = 0, ulen = 0, dlen = 0, doff = 0, app_data = 0x0, flags = 16}
(gdb)

------------------------------
I tried to compile locally with debugging enabled but that compile fails also.

$ DEB_BUILD_OPTIONS='nostrip noopt debug' dpkg-buildpackage -b -uc -us
...
citserver.c: In function ‘cit_backtrace’:
citserver.c:50:18: error: ‘IOSTR’ undeclared (first use in this function); did you mean ‘ENOSTR’?
   50 | const char *p = IOSTR;
      | ^~~~~
      | ENOSTR

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: citadel-server 917-4
ProcVersionSignature: Ubuntu 5.4.0-33.37-generic 5.4.34
Uname: Linux 5.4.0-33-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.2
Architecture: amd64
CasperMD5CheckResult: pass
Date: Wed Jun 3 15:18:03 2020
InstallationDate: Installed on 2020-06-02 (1 days ago)
InstallationMedia: Ubuntu-Server 20.04 LTS "Focal Fossa" - Release amd64 (20200423)
ProcEnviron:
 SHELL=/bin/bash
 LANG=C.UTF-8
 TERM=xterm-256color
 PATH=(custom, no user)
SourcePackage: citadel
UpgradeStatus: No upgrade log present (probably fresh install)

[donr (don-db-reid)]

More details with libdb5.3-dbg also installed:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff77b3b2d in __db_check_txn (dbp=dbp@entry=0x5555557927e0, txn=0x100000000,
    assoc_locker=assoc_locker@entry=0x0, read_op=read_op@entry=1) at ../src/common/db_err.c:818
818 ../src/common/db_err.c: No such file or directory.
(gdb) bt
#0 0x00007ffff77b3b2d in __db_check_txn (dbp=dbp@entry=0x5555557927e0, txn=0x100000000,
    assoc_locker=assoc_locker@entry=0x0, read_op=read_op@entry=1) at ../src/common/db_err.c:818
#1 0x00007ffff77b8ee0 in __db_get_pp (dbp=0x5555557927e0, txn=<optimized out>,
    key=key@entry=0x7fffffffab00, data=data@entry=0x7fffffffab30, flags=flags@entry=0)
    at ../src/db/db_iface.c:710
#2 0x000055555556ec9a in cdb_fetch (cdb=cdb@entry=13, key=key@entry=0x5555555d4b85,
    keylen=keylen@entry=28) at database.c:657
#3 0x00005555555723d7 in CtdlGetConfigStr (key=key@entry=0x5555555d4b85 "c_config_created_or_migrated")
    at config.c:429
#4 0x000055555557290d in CtdlGetConfigLong (
    key=key@entry=0x5555555d4b85 "c_config_created_or_migrated") at config.c:459
#5 0x0000555555572a3c in initialize_config_system () at config.c:284
#6 0x000055555556939d in master_startup () at citserver.c:150
#7 0x0000555555564ca7 in main (argc=2, argv=0x7fffffffec68) at server_main.c:260

[donr (don-db-reid)]

This looks like a basic startup issue where the config stuff is trying to access the threading but the threading has not yet been initialized.

main() calls master_startup() which calls initialize_config_system() ...

but main() does not call go_threading() until much later.

So down in cdb_fetch() when it accesses "TSD->tid" it gets a value of 0x100000000 which is not a valid memory address, but this code only checks for NULL.

Perhaps it is simply initialization of tid to NULL that is needed, or should the threading
system be setup first????

[donr (don-db-reid)]

This appears to have been fixed in the upstream source where sysdep.c:init_sysdep() calls pthread_key_create(&ThreadKey). This was added after the current Ubuntu version 917.