Apache2 mod-wsgi segfault on double reload

[Thomas Kirsch]

# Issue Description

Currently, Apache generates a segmentation fault with "libapache2-mod-wsgi-py3" enabled when the process receives multiple *USR1* signals within a short amount of time. The issue is only occuring when there is a wsgi process running (see attached crash report for sample vhost config).
The issue might arise when "apache2ctl graceful" is triggered by logrotate as a postrotate operation several times (i.e. used in several config files).

## System Infos

Description: Ubuntu 16.04.1 LTS
Release: 16.04
Kernel: 4.4.0-34-generic
Architecture: x86_64

## Package versions

apache2: 2.4.18-2ubuntu3
libapache2-mod-wsgi-py3: 4.3.0-1.1build1

## Steps to reproduce

 * Create vhost config with WSGIDaemonProcess directive and enable site
 * Issue double reload: apache2ctl graceful && apache2ctl graceful

## Outcome

 * Segmentation fault

## Core Dump of Apache

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.

#0 0x00007f6b6d010640 in ?? ()
No symbol table info available.
#1 0x00007f6b6fcbb3d0 in __libc_sigaction (sig=10, act=0x7fffaf18b430, oact=0x7fffaf18b300)
    at ../sysdeps/unix/sysv/linux/x86_64/sigaction.c:54
        result = <optimized out>
        kact = {k_sa_handler = 0x7fffaf18b3b0, sa_flags = 140099413790480,
          sa_restorer = 0x7f6b6fcaaf18, sa_mask = {__val = {140736131019476, 140736131019680,
              94201515593344, 140097538228224, 34, 140099423149124, 1, 0, 140099425322224,
              140099425321816, 140736131019840, 140099413849040, 1, 0, 0, 140097538228226}}}
        koact = {k_sa_handler = 0x7fffaf18b8a0, sa_flags = 1, sa_restorer = 0x0, sa_mask = {
            __val = {582, 93750, 1, 140099416110048, 94201486782229, 0, 0, 0, 140099424570760, 0,
              18446744073709551612, 140099410844915, 140736131020952, 140099410844915, 582, 51}}}
#2 0x0000000000000000 in ?? ()
No symbol table info available.

# Apport log attached

[Christian Ehrhardt  (paelzer)]

Add
        WSGIDaemonProcess example.com processes=2 threads=15 display-name=%{GROUP}
        WSGIProcessGroup example.com

to /etc/apache2/sites-enabled/000-default.conf of a default install.

Processes:
# ps axlf | grep wsg
0 0 26860 25750 20 0 14856 1100 pipe_w S+ ? 0:00 \_ grep --color=auto wsg
5 33 26765 26764 20 0 356176 12428 poll_s Sl ? 0:00 \_ (wsgi:example.com -k start
5 33 26766 26764 20 0 356168 12448 poll_s Sl ? 0:00 \_ (wsgi:example.com -k start

Looping over apache2ctl graceful triggers a crash.

I tried to get a better backtrace with more debug packages and such - attaching that...

This is from cosmic using 4.5.17-1 there is a changelog of wsgi 4.1 reading much like it at [1].
Good issues come back.

Never the less I'd ask you to report also upstream and mention the bug URL you created to track their activity on it as well.Especially since they already fixed the same in 4.1 they might be interested.

[1]: http://modwsgi.readthedocs.io/en/develop/release-notes/version-4.1.0.html

[Christian Ehrhardt  (paelzer)]

More detailed backtrace:

# apport-retrace /var/crash/_usr_sbin_apache2.0.crash --stdout -S system
Installing extra package apache2-bin to get ExecutablePath
dpkg-source: info: extracting apache2 in apache2-2.4.33
dpkg-source: info: unpacking apache2_2.4.33.orig.tar.bz2
dpkg-source: info: unpacking apache2_2.4.33-3ubuntu2.debian.tar.xz
dpkg-source: info: applying fhs_compliance.patch
dpkg-source: info: applying no_LD_LIBRARY_PATH.patch
dpkg-source: info: applying suexec-CVE-2007-1742.patch
dpkg-source: info: applying customize_apxs.patch
dpkg-source: info: applying build_suexec-custom.patch
dpkg-source: info: applying reproducible_builds.diff
dpkg-source: info: applying mod_http2_mem_usage_32bit.diff
dpkg-source: info: applying 086_svn_cross_compiles
W: Download is performed unsandboxed as root as file 'apache2_2.4.33-3ubuntu2.dsc' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
--- stack trace ---
#0 impl_pollset_remove (pollset=0x7feae06aa050, descriptor=0x0) at ./poll/unix/epoll.c:224
        ep = <optimized out>
        rv = 0
        ev = {events = 32766, data = {ptr = 0x56391f4e16a0, fd = 525211296, u32 = 525211296, u64 = 94803338335904}}
        ret = <optimized out>
#1 0x00007feadcf8cbe2 in disable_listensocks () at event.c:470
        i = 0
        i = <optimized out>
#2 0x00007feadcf8cc33 in wakeup_listener () at event.c:578
No locals.
#3 0x00007feadcf8cd74 in signal_threads (mode=1) at event.c:627
No locals.
#4 0x00007feadcf8a94a in child_main (child_num_arg=child_num_arg@entry=0, child_bucket=child_bucket@entry=0) at event.c:2579
        threads = 0x56391f4e16a0
        rv = 2
        ts = 0x7feae06a9af8
        thread_attr = 0x7feae06a9b18
        start_thread_id = 0x7feae06a9b78
        i = <optimized out>
#5 0x00007feadcf8d05a in make_child (s=0x7feae07414a0, slot=slot@entry=0, bucket=0) at event.c:2663
        pid = 0
#6 0x00007feadcf8dedd in server_main_loop (num_buckets=1, remaining_children_to_start=2) at event.c:2933
        ps = <optimized out>
        status = 0
        pid = {pid = 26968, in = 0x7feadfe8dcd6 <find_entry+134>, out = 0x0, err = 0x9}
        i = <optimized out>
        child_slot = 0
        exitwhy = APR_PROC_EXIT
        processed_status = 0
        child_slot = <optimized out>
        exitwhy = <optimized out>
        status = <optimized out>
        processed_status = <optimized out>
        pid = <optimized out>
        i = <optimized out>
        sr__ = <optimized out>
        ps = <optimized out>
        sr__ = <optimized out>
#7 event_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at event.c:3051
        num_buckets = 1
        remaining_children_to_start = <optimized out>
        i = <optimized out>
#8 0x000056391d475b7e in ap_run_mpm (pconf=0x7feae076c028, plog=0x7feae073a028, s=0x7feae07414a0) at mpm_common.c:94
        pHook = <optimized out>
        n = 0
        rv = -1
#9 0x000056391d46e46b in main (argc=<optimized out>, argv=<optimized out>) at main.c:818
        c = 0 '\000'
        showcompile = 0
        showdirectives = 0
        confname = 0x56391d4b57e5 "apache2.conf"
        def_server_root = 0x56391d4b57d8 "/etc/apache...

[Andreas Hasenack (ahasenack)]

Still happens on xenial, and disco as well, where we have:
apache2: 2.4.34-1ubuntu2
libapache2-mod-wsgi-py3: 4.5.17-1build2

It took longer to crash on disco, though.

[Andreas Hasenack (ahasenack)]

Still happening on xenial:

apache2 2.4.18-2ubuntu3.10
libapache2-mod-wsgi-py3 4.3.0-1.1build1

My apache default site config:
# cat /etc/apache2/sites-enabled/000-default.conf |grep -vE "^[[:blank:]]*(#|$)"
<VirtualHost *:80>
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
        WSGIDaemonProcess example.com processes=2 threads=15 display-name=%{GROUP}
        WSGIProcessGroup example.com
</VirtualHost>

Issued apache2ctl graceful in an infinite while loop, and it eventually crashed:
[Wed Jul 10 14:07:24.348925 2019] [mpm_event:notice] [pid 1363:tid 140450587101056] AH00493: SIGUSR1 received. Doing graceful restart
[Wed Jul 10 14:07:24.412595 2019] [wsgi:warn] [pid 1363:tid 140450587101056] mod_wsgi: Compiled for Python/3.5.1+.
[Wed Jul 10 14:07:24.412721 2019] [wsgi:warn] [pid 1363:tid 140450587101056] mod_wsgi: Runtime using Python/3.5.2.
[Wed Jul 10 14:07:24.414295 2019] [mpm_event:notice] [pid 1363:tid 140450587101056] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/3.5.2 configured -- resuming normal operations
[Wed Jul 10 14:07:24.414410 2019] [core:notice] [pid 1363:tid 140450587101056] AH00094: Command line: '/usr/sbin/apache2'
[Wed Jul 10 14:07:24.465601 2019] [mpm_event:notice] [pid 1363:tid 140450587101056] AH00493: SIGUSR1 received. Doing graceful restart
[Wed Jul 10 14:07:24.557427 2019] [core:notice] [pid 1363] AH00060: seg fault or similar nasty error detected in the parent process
[Wed Jul 10 14:07:24.781927 2019] [wsgi:warn] [pid 2427:tid 140220268152704] mod_wsgi: Compiled for Python/3.5.1+.
[Wed Jul 10 14:07:24.781998 2019] [wsgi:warn] [pid 2427:tid 140220268152704] mod_wsgi: Runtime using Python/3.5.2.
[Wed Jul 10 14:07:24.782018 2019] [core:warn] [pid 2427:tid 140220268152704] AH00098: pid file /var/run/apache2/apache2.pid overwritten -- Unclean shutdown of previous Apache run?

I couldn't reproduce it on eoan, and I left the reloads running for a few minutes.

Since my previous comment said it still happened on disco, let me try that now, also to validate my procedure.

...

I'm also having a hard time reproducing this on disco, same procedure as on xenial.

[Andreas Hasenack (ahasenack)]

No new info at the moment, bug still in our queue.

[Paride Legovini (paride)]

I tried Andrea's reproducer on Focal and couldn't get apache2 to crash. It's tempting to set it Fix Released, but we didn't really identify where the problem is, so Incomplete will be.