While setting up a perl web application with mod_perl & apache, apache kept segfaulting. Broke out gdb, and found that it was segfaulting within perl itself Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7358ff5 in perl_parse () from /lib/x86_64-linux-gnu/libperl.so.5.30 (gdb) bt #0 0x00007ffff7358ff5 in perl_parse () from /lib/x86_64-linux-gnu/libperl.so.5.30 #1 0x00007ffff764cd0c in modperl_startup () from /usr/lib/apache2/modules/mod_perl.so #2 0x00007ffff764cc97 in modperl_startup () from /usr/lib/apache2/modules/mod_perl.so #3 0x00007ffff764d0fa in modperl_init () from /usr/lib/apache2/modules/mod_perl.so #4 0x00007ffff764d27b in modperl_hook_init () from /usr/lib/apache2/modules/mod_perl.so #5 0x00005555555b23d4 in ap_run_open_logs () #6 0x000055555558c440 in main () # valgrind apache2 -k start -X ==22529== Memcheck, a memory error detector ==22529== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==22529== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info ==22529== Command: apache2 -k start -X ==22529== ==22529== Invalid read of size 8 ==22529== at 0x564AFF5: perl_parse (in /usr/lib/x86_64-linux-gnu/libperl.so.5.30.0) ==22529== by 0x55A8D0B: modperl_startup (in /usr/lib/apache2/modules/mod_perl.so) ==22529== by 0x55A8C96: modperl_startup (in /usr/lib/apache2/modules/mod_perl.so) ==22529== by 0x55A90F9: modperl_init (in /usr/lib/apache2/modules/mod_perl.so) ==22529== by 0x55A927A: modperl_hook_init (in /usr/lib/apache2/modules/mod_perl.so) ==22529== by 0x1663D3: ap_run_open_logs (in /usr/sbin/apache2) ==22529== by 0x14043F: main (in /usr/sbin/apache2) ==22529== Address 0x5a44000 is not stack'd, malloc'd or (recently) free'd ==22529== ==22529== ==22529== Process terminating with default action of signal 11 (SIGSEGV) ==22529== Access not within mapped region at address 0x5A44000 ==22529== at 0x564AFF5: perl_parse (in /usr/lib/x86_64-linux-gnu/libperl.so.5.30.0) ==22529== by 0x55A8D0B: modperl_startup (in /usr/lib/apache2/modules/mod_perl.so) ==22529== by 0x55A8C96: modperl_startup (in /usr/lib/apache2/modules/mod_perl.so) ==22529== by 0x55A90F9: modperl_init (in /usr/lib/apache2/modules/mod_perl.so) ==22529== by 0x55A927A: modperl_hook_init (in /usr/lib/apache2/modules/mod_perl.so) ==22529== by 0x1663D3: ap_run_open_logs (in /usr/sbin/apache2) ==22529== by 0x14043F: main (in /usr/sbin/apache2) gdb indicated that it was erroring in very early in perl's runtime, before it had got to any perl code. When using debug symbols, the exact line it was failing on was `scriptname = argv[0];` (perl.c:2365) It wasn't possible to reason beyond that as stepping through optimised code even with debug symbols is next to impossible to make any sense of. I did find that building an unoptimised perl made the error go away. I found the following closed issue: https://github.com/Perl/perl5/issues/15806 which describes the same issue I was having. Looking at the source for mod_perl, I found that the argv array passed to perl_parse() is not NULL terminated as is required by perl - ( documentation: https://perldoc.perl.org/perlembed#Adding-a-Perl-interpreter-to-your-C-program ) After patching this, the problem went away and didn't come back. Patch is attached. However, for some reason or another, the problem is very inconsistent. Sometimes it will work fine, other times it will fail repeatedly. I suspect it has something to do with my environment (ubuntu-minimal:focal LXD container). It almost feels like the act of installing all the build dependencies for mod_perl made the crash disappear. However, with a clean container and copying the patched deb across, it does appear to be repeatable: # ls libapache2-mod-perl2_2.0.11-2_amd64.clean.deb libapache2-mod-perl2_2.0.11-2_amd64.patched.deb # dpkg -i libapache2-mod-perl2_2.0.11-2_amd64.clean.deb (Reading database ... 33224 files and directories currently installed.) Preparing to unpack libapache2-mod-perl2_2.0.11-2_amd64.clean.deb ... Unpacking libapache2-mod-perl2 (2.0.11-2) over (2.0.11-2) ... Setting up libapache2-mod-perl2 (2.0.11-2) ... apache2_invoke perl: already enabled # source /etc/apache2/envvars # apache2 -k start -X Segmentation fault (core dumped) # dpkg -i libapache2-mod-perl2_2.0.11-2_amd64.patched.deb (Reading database ... 33224 files and directories currently installed.) Preparing to unpack libapache2-mod-perl2_2.0.11-2_amd64.patched.deb ... Unpacking libapache2-mod-perl2 (2.0.11-2) over (2.0.11-2) ... Setting up libapache2-mod-perl2 (2.0.11-2) ... apache2_invoke perl: already enabled # apache2 -k start -X ^C # dpkg -i libapache2-mod-perl2_2.0.11-2_amd64.clean.deb (Reading database ... 33224 files and directories currently installed.) Preparing to unpack libapache2-mod-perl2_2.0.11-2_amd64.clean.deb ... Unpacking libapache2-mod-perl2 (2.0.11-2) over (2.0.11-2) ... Setting up libapache2-mod-perl2 (2.0.11-2) ... apache2_invoke perl: already enabled # apache2 -k start -X Segmentation fault (core dumped)