Random apache2 crash in Ubuntu 12.04 (libpthread-2.15.so)

[Juan Pablo Marco Cardona]

Hello,
in one of our ubuntu 12.04 x86_64 servers the apache2 crashes ramdomly.

This is the last apache2 crash in the kernel log:
[1110685.354122] /usr/sbin/apach[10514]: segfault at 6c ip 00007fde00a44e84 sp 00007fff1eb1f7e0 error 4 in libpthread-2.15.so[7fde00a3b000+18000]

Maybe it is not related, but when the apache2 process crashes, in the access log there are this entries:
127.0.0.1 - - [24/Mar/2013:06:42:29 +0100] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.2.22 (Ubuntu) (internal dummy connection)"
127.0.0.1 - - [24/Mar/2013:06:42:29 +0100] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.2.22 (Ubuntu) (internal dummy connection)"

I have attached an apport crash report of the last crash.

This are the ubuntu and apache2 package versions:

lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04

dpkg -l | grep -E " apache2 "
ii apache2 2.2.22-1ubuntu1.3 Apache HTTP Server metapackage

Just comment that today using aptitude the apache2 packages was upgraded and the last crash happened last Sunday at 6:45 A.M

Thanks in advance.

Regards,
Juan Pablo

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Please could you confirm the version of the apache2 package in use at the time the crash occurred? And have you reproduced this problem on any other hardware apart from the server affected?

Once you've answered, please change the bug status back to New.

Setting Importance to Medium for now. As apache2 is widely used on 12.04 and I haven't seen any other reports, I think it's reasonable to assume that this crash does not affect a wide number of Ubuntu users until others can confirm it.

[Juan Pablo Marco Cardona (juanpablo-marcocardona)]

Hello,
thanks for you quickly support.

I can confirm the apache2 package version, it was the previous one, 2.2.22-1ubuntu1.2

We are using the apache2 prefork version, not the worker.

This issue just happens in one of our servers.

Regards,
Juan Pablo

[Juan Pablo Marco Cardona (juanpablo-marcocardona)]

Hello again,
last sunday the apache2 process died again:
[1710511.756125] /usr/sbin/apach[4244]: segfault at 6c ip 00007f2e061b7e84 sp 00007fffbdc40060 error 4 in libpthread-2.15.so[7f2e061ae000+18000]

I attach the apache2 (CoreDumpDirectory in the apache configuration file ) core generate in this crash.

Thinking about this crash, as it happens regularly almost every sunday around 06:45 AM, maybe the crash is generated by the tasks crond runs weekly:
grep weekly /etc/crontab
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )

Weekly, crontab runs this tasks:

pwd
/etc/cron.weekly

ls
apt-xapian-index man-db

cat apt-xapian-index

#!/bin/sh

CMD=/usr/sbin/update-apt-xapian-index

# ionice should not be called in a virtual environment
# (similar to man-db cronjobs)
egrep -q '(envID|VxID):.*[1-9]' /proc/self/status || IONICE=/usr/bin/ionice

# Check if we're on battery
if which on_ac_power >/dev/null 2>&1; then
    on_ac_power >/dev/null 2>&1
    ON_BATTERY=$?

    # Here we use "-eq 1" instead of "-ne 0" because
    # on_ac_power could also return 255, which means
    # it can't tell whether we are on AC or not. In
    # that case, run update-a-x-i nevertheless.
    [ "$ON_BATTERY" -eq 1 ] && exit 0
fi

# Rebuild the index
if [ -x "$CMD" ]
then
        if [ -x "$IONICE" ]
        then
                nice -n 19 $IONICE -c 3 $CMD --quiet
        else
                nice -n 19 $CMD --quiet
        fi
fi

cat man-db
#!/bin/sh
#
# man-db cron weekly

set -e

iosched_idle=
# Don't try to change I/O priority in a vserver or OpenVZ.
if ! egrep -q '(envID|VxID):.*[1-9]' /proc/self/status && \
   ([ ! -d /proc/vz ] || [ -d /proc/bc ]); then
    dpkg_version="$(dpkg-query -W -f '${Version}' dpkg 2>/dev/null)"
    if dpkg --compare-versions "$dpkg_version" ge 1.15.0; then
        iosched_idle='--iosched idle'
    fi
fi

if ! [ -d /var/cache/man ]; then
    # Recover from deletion, per FHS.
    mkdir -p /var/cache/man
    chown man:root /var/cache/man || true
    chmod 2755 /var/cache/man
fi

# regenerate man database
if [ -x /usr/bin/mandb ]; then
    # --pidfile /dev/null so it always starts; mandb isn't really a daemon,
    # but we want to start it like one.
    start-stop-daemon --start --pidfile /dev/null \
                      --startas /usr/bin/mandb --oknodo --chuid man \
                      $iosched_idle \
                      -- --quiet
fi

exit 0

Then, i will deactivate this two crontab tasks (apt-xapian-index and man-db) and monitor the next Sunday if the apache crash happens again :)

Anyway, i have configured a watchdog that checks every 3 minutes if the apache2 process is running, if not, it starts it.

Regards,

[Juan Pablo Marco Cardona (juanpablo-marcocardona)]

Hello,
finally i didn't deactivate the weekly crond tasks, I configured the weekly crond tasks to be run at 05:47 AM every sunday.

Anyway, last sunday the apache2 process died again. This time at 05:15:13 AM.

Any update in this bug report?

Thanks in advanced!

Regards,

[Robie Basak (racb)]

Sorry, I don't think this bug report can make any progress until we've confirmed that the problem affects others (so we know it's not a hardware fault or local configuration issue) and we have steps a developer can use to reproduce the problem on his system.

[Juan Pablo Marco Cardona (juanpablo-marcocardona)]

Hello again,
finally i find the cause of this issue.

The problem happens every time logrotate rotate the apache logs.

cat /etc/logrotate.d/apache2

/var/log/apache2/*.log {
        weekly
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                /etc/init.d/apache2 reload > /dev/null
        endscript
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi; \
        endscript
}

In fact, every time a "/etc/init.d/apache2 reload" is run, the apache2 dies.
Then, executing "/etc/init.d/apache2 restart" instead every time logrotate rotate the apache2 logs, should fix the problem.

Anyway, this is just a workaround, because the apache shouldn't die every time a reload is executed, right?

Thanks in advance

Regards.
P.D: Maybe this bugs are related?
https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/669005
https://issues.apache.org/bugzilla/show_bug.cgi?id=50309

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dave Pooser (dave-ubuntu)]

I am seeing the same behavior on my system. Reproduceable every time I do /etc/init.d/apache2 reload. System in question is a VM running under VMware ESXi, so I could provide developers with a copy of the VM if they can't reproduce it themselves (but it's not a small file).

Apache is serving from three directories, using a Geotrust wildcard SSL cert: SVN over HTTPS, RequestTracker 4.0.10 (built from source), and a DokuWiki wiki. Version is 2.2.22 (Ubuntu) built 2013-03-08

I'll be glad to provide core dumps as needed, or a copy of the VM.

[Juan Pablo Marco Cardona (juanpablo-marcocardona)]

Hello,
when i opened this bug report, i forgot to comment that the server is virtual, it is running on Wvmware ESXi.

In fact, on Vmware ESXi 4.1 U1

Regards,

[Juan Pablo Marco Cardona (juanpablo-marcocardona)]

Hello again,
checking the Dave Posser's comment, i would like to add that in our server we are running RequestTracker too.
In this case, version 4.0.4

Regards,

[Dave Pooser (dave-ubuntu)]

OK, looks like RequestTracker is the problem-- specifically one of the perl modules. If I comment out the Perl handler lines in my site definition in /etc/apache2/sites-available/:
        <Perl>
                use Plack::Handler::Apache2;
                Plack::Handler::Apache2->preload("/opt/rt4/sbin/rt-server");
        </Perl>
then I get no segfault on reload. Uncomment those lines, and the segfault is back.

Plack::Handler::Apache2 version is 1.0018.tar.gz -- installed from CPAN

[Geoffrey Fairchild (gfairchild)]

I'm glad I stumbled onto this bug. My full information is available at http://stackoverflow.com/questions/16304844/php-apc-causing-apache-to-seg-fault.

- I am not using RequestTracker.

- This happens for me any time I issue "sudo service apache2 graceful". Issuing reload doesn't seem to cause mine to sigfault, but graceful does.

- Apache reliably goes down around 6:45am every day, which, as another poster noted, is when logrotate is set to run. In my logrotate scripts, I restart apache2 gracefully.

- When apache2 segfaults, I get the following error in /var/log/apache2/error.log:

[Tue Apr 30 11:05:33 2013] [notice] Graceful restart requested, doing restart
PHP Fatal error: PHP Startup: apc_mmap: mmap failed: in Unknown on line 0
[Tue Apr 30 11:05:34 2013] [notice] seg fault or similar nasty error detected in the parent process

Does anyone have any clue as to what's going on here?

[Geoffrey Fairchild (gfairchild)]

Actually, I think I spoke too soon. It looks like reload AND graceful cause apache2 to segfault. Only a full restart or stopping/starting apache2 seem to get it going again.

Any time this happens, I see lines like the following in /var/log/syslog:

kernel: apache2[11246]: segfault at ffffffff ip b7682bd8 sp bfbf9b90 error 7 in libpthread-2.15.so[b767a000+17000]

[Hector Perez (heperez)]

I have the same problem but with lighttpd. We are running Hawk in ubuntu 12.04.3 LTS in a VM running on Vmware Server 2.0.2

Sep 5 17:42:47 nodo02 kernel: [147211.549276] monitor[2300]: segfault at 6475 ip b7040cb7 sp bfc76b50 error 4 in libpthread-2.15.so[b7038000+17000]
Sep 5 17:43:06 nodo02 crmd: [1422]: info: handle_request: Current ping state: S_IDLE
Sep 5 17:44:06 nodo02 kernel: [147290.137124] monitor[2362]: segfault at 6475 ip b6fa7cb7 sp bfc44b90 error 4 in libpthread-2.15.so[b6f9f000+17000]

[Hector Perez (heperez)]

Sorry, false alarm. It was a bug in a process of hawk that made lighttpd fail with the error showed.

[Geoffrey Fairchild (gfairchild)]

I haven't actually experienced this bug in a while. Graceful shutdowns and logrotate have been working completely without issue. Unfortunately, my syslogs only goes back to 8/30/13, but there're no segfaults at all. I'm guessing a fix came out in some update.

[Dave Pooser (dave-ubuntu)]

I'm still seeing it after running apt-get upgrade this morning. Upgrading Plack::Handler to 1.0029 (using CPAN) does not resolve the issue either.

[Geoffrey Fairchild (gfairchild)]

All of a sudden, this bug has come back. The last couple weeks, Apache goes completely unresponsive almost daily. It takes a restart to get it going again.

[Geoffrey Fairchild (gfairchild)]

As an update to my last comment on 2013-12-17, my Apache non-responsiveness was actually due to mod_ssl, not this particular bug. I haven't encountered this bug in some time.

[Ryan Hsu (rhsu)]

This came out of nowhere for us and has happened twice in the last 2 weeks. This is on a server that has been running for months without any issue.

The server is running on VMware ESX 5.1u2 with the following versions:

$lsb_release -rd
Description: Ubuntu 12.04.3 LTS
Release: 12.04

$dpkg -l | grep -E " apache2 "
ii apache2 2.2.22-1ubuntu1.4 Apache HTTP Server metapackage

Logs:
syslog for 1st crash: http://paste.openstack.org/show/5234GGoeXthoVCdNjOr8/
syslog for 2nd crash: http://paste.openstack.org/show/4mY24f65lITgIAGgRVY4/