Random apache2 crash in Ubuntu 12.04 (libpthread-2.15.so)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| apache2 (Ubuntu) |
Medium
|
Unassigned |
Bug Description
Hello,
in one of our ubuntu 12.04 x86_64 servers the apache2 crashes ramdomly.
This is the last apache2 crash in the kernel log:
[1110685.354122] /usr/sbin/
Maybe it is not related, but when the apache2 process crashes, in the access log there are this entries:
127.0.0.1 - - [24/Mar/
127.0.0.1 - - [24/Mar/
I have attached an apport crash report of the last crash.
This are the ubuntu and apache2 package versions:
lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04
dpkg -l | grep -E " apache2 "
ii apache2 2.2.22-1ubuntu1.3 Apache HTTP Server metapackage
Just comment that today using aptitude the apache2 packages was upgraded and the last crash happened last Sunday at 6:45 A.M
Thanks in advance.
Regards,
Juan Pablo
description: | updated |
description: | updated |
Robie Basak (racb) wrote : | #2 |
Changed in apache2 (Ubuntu): | |
status: | New → Incomplete |
importance: | Undecided → Medium |
Hello,
thanks for you quickly support.
I can confirm the apache2 package version, it was the previous one, 2.2.22-1ubuntu1.2
We are using the apache2 prefork version, not the worker.
This issue just happens in one of our servers.
Regards,
Juan Pablo
Changed in apache2 (Ubuntu): | |
status: | Incomplete → New |
Hello again,
last sunday the apache2 process died again:
[1710511.756125] /usr/sbin/
I attach the apache2 (CoreDumpDirectory in the apache configuration file ) core generate in this crash.
Thinking about this crash, as it happens regularly almost every sunday around 06:45 AM, maybe the crash is generated by the tasks crond runs weekly:
grep weekly /etc/crontab
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
Weekly, crontab runs this tasks:
pwd
/etc/cron.weekly
ls
apt-xapian-index man-db
cat apt-xapian-index
#!/bin/sh
CMD=/usr/
# ionice should not be called in a virtual environment
# (similar to man-db cronjobs)
egrep -q '(envID|
# Check if we're on battery
if which on_ac_power >/dev/null 2>&1; then
on_ac_power >/dev/null 2>&1
ON_BATTERY=$?
# Here we use "-eq 1" instead of "-ne 0" because
# on_ac_power could also return 255, which means
# it can't tell whether we are on AC or not. In
# that case, run update-a-x-i nevertheless.
[ "$ON_BATTERY" -eq 1 ] && exit 0
fi
# Rebuild the index
if [ -x "$CMD" ]
then
if [ -x "$IONICE" ]
then
else
fi
fi
cat man-db
#!/bin/sh
#
# man-db cron weekly
set -e
iosched_idle=
# Don't try to change I/O priority in a vserver or OpenVZ.
if ! egrep -q '(envID|
([ ! -d /proc/vz ] || [ -d /proc/bc ]); then
dpkg_
if dpkg --compare-versions "$dpkg_version" ge 1.15.0; then
fi
fi
if ! [ -d /var/cache/man ]; then
# Recover from deletion, per FHS.
mkdir -p /var/cache/man
chown man:root /var/cache/man || true
chmod 2755 /var/cache/man
fi
# regenerate man database
if [ -x /usr/bin/mandb ]; then
# --pidfile /dev/null so it always starts; mandb isn't really a daemon,
# but we want to start it like one.
start-
fi
exit 0
Then, i will deactivate this two crontab tasks (apt-xapian-index and man-db) and monitor the next Sunday if the apache crash happens again :)
Anyway, i have configured a watchdog that checks every 3 minutes if the apache2 process is running, if not, it starts it.
Regards,
Hello,
finally i didn't deactivate the weekly crond tasks, I configured the weekly crond tasks to be run at 05:47 AM every sunday.
Anyway, last sunday the apache2 process died again. This time at 05:15:13 AM.
Any update in this bug report?
Thanks in advanced!
Regards,
Robie Basak (racb) wrote : | #7 |
Sorry, I don't think this bug report can make any progress until we've confirmed that the problem affects others (so we know it's not a hardware fault or local configuration issue) and we have steps a developer can use to reproduce the problem on his system.
Hello again,
finally i find the cause of this issue.
The problem happens every time logrotate rotate the apache logs.
cat /etc/logrotate.
/var/log/
weekly
missingok
rotate 52
compress
notifempty
create 640 root adm
postrotate
endscript
prerotate
if [ -d /etc/logrotate.
fi; \
endscript
}
In fact, every time a "/etc/init.
Then, executing "/etc/init.
Anyway, this is just a workaround, because the apache shouldn't die every time a reload is executed, right?
Thanks in advance
Regards.
P.D: Maybe this bugs are related?
https:/
https:/
Launchpad Janitor (janitor) wrote : | #9 |
Status changed to 'Confirmed' because the bug affects multiple users.
Changed in apache2 (Ubuntu): | |
status: | New → Confirmed |
Dave Pooser (dave-ubuntu) wrote : | #10 |
I am seeing the same behavior on my system. Reproduceable every time I do /etc/init.d/apache2 reload. System in question is a VM running under VMware ESXi, so I could provide developers with a copy of the VM if they can't reproduce it themselves (but it's not a small file).
Apache is serving from three directories, using a Geotrust wildcard SSL cert: SVN over HTTPS, RequestTracker 4.0.10 (built from source), and a DokuWiki wiki. Version is 2.2.22 (Ubuntu) built 2013-03-08
I'll be glad to provide core dumps as needed, or a copy of the VM.
Hello,
when i opened this bug report, i forgot to comment that the server is virtual, it is running on Wvmware ESXi.
In fact, on Vmware ESXi 4.1 U1
Regards,
Hello again,
checking the Dave Posser's comment, i would like to add that in our server we are running RequestTracker too.
In this case, version 4.0.4
Regards,
Dave Pooser (dave-ubuntu) wrote : | #13 |
OK, looks like RequestTracker is the problem-- specifically one of the perl modules. If I comment out the Perl handler lines in my site definition in /etc/apache2/
<Perl>
use Plack::
</Perl>
then I get no segfault on reload. Uncomment those lines, and the segfault is back.
Plack::
Geoffrey Fairchild (gfairchild) wrote : | #14 |
I'm glad I stumbled onto this bug. My full information is available at http://
- I am not using RequestTracker.
- This happens for me any time I issue "sudo service apache2 graceful". Issuing reload doesn't seem to cause mine to sigfault, but graceful does.
- Apache reliably goes down around 6:45am every day, which, as another poster noted, is when logrotate is set to run. In my logrotate scripts, I restart apache2 gracefully.
- When apache2 segfaults, I get the following error in /var/log/
[Tue Apr 30 11:05:33 2013] [notice] Graceful restart requested, doing restart
PHP Fatal error: PHP Startup: apc_mmap: mmap failed: in Unknown on line 0
[Tue Apr 30 11:05:34 2013] [notice] seg fault or similar nasty error detected in the parent process
Does anyone have any clue as to what's going on here?
Geoffrey Fairchild (gfairchild) wrote : | #15 |
Actually, I think I spoke too soon. It looks like reload AND graceful cause apache2 to segfault. Only a full restart or stopping/starting apache2 seem to get it going again.
Any time this happens, I see lines like the following in /var/log/syslog:
kernel: apache2[11246]: segfault at ffffffff ip b7682bd8 sp bfbf9b90 error 7 in libpthread-
Hector Perez (heperez) wrote : | #16 |
I have the same problem but with lighttpd. We are running Hawk in ubuntu 12.04.3 LTS in a VM running on Vmware Server 2.0.2
Sep 5 17:42:47 nodo02 kernel: [147211.549276] monitor[2300]: segfault at 6475 ip b7040cb7 sp bfc76b50 error 4 in libpthread-
Sep 5 17:43:06 nodo02 crmd: [1422]: info: handle_request: Current ping state: S_IDLE
Sep 5 17:44:06 nodo02 kernel: [147290.137124] monitor[2362]: segfault at 6475 ip b6fa7cb7 sp bfc44b90 error 4 in libpthread-
Hector Perez (heperez) wrote : | #17 |
Sorry, false alarm. It was a bug in a process of hawk that made lighttpd fail with the error showed.
Geoffrey Fairchild (gfairchild) wrote : | #18 |
I haven't actually experienced this bug in a while. Graceful shutdowns and logrotate have been working completely without issue. Unfortunately, my syslogs only goes back to 8/30/13, but there're no segfaults at all. I'm guessing a fix came out in some update.
Dave Pooser (dave-ubuntu) wrote : | #19 |
I'm still seeing it after running apt-get upgrade this morning. Upgrading Plack::Handler to 1.0029 (using CPAN) does not resolve the issue either.
summary: |
- Ramdom apache2 crash in Ubuntu 12.04 (libpthread-2.15.so) + Random apache2 crash in Ubuntu 12.04 (libpthread-2.15.so) |
Geoffrey Fairchild (gfairchild) wrote : | #20 |
All of a sudden, this bug has come back. The last couple weeks, Apache goes completely unresponsive almost daily. It takes a restart to get it going again.
Geoffrey Fairchild (gfairchild) wrote : | #21 |
As an update to my last comment on 2013-12-17, my Apache non-responsiveness was actually due to mod_ssl, not this particular bug. I haven't encountered this bug in some time.
Ryan Hsu (rhsu) wrote : | #22 |
This came out of nowhere for us and has happened twice in the last 2 weeks. This is on a server that has been running for months without any issue.
The server is running on VMware ESX 5.1u2 with the following versions:
$lsb_release -rd
Description: Ubuntu 12.04.3 LTS
Release: 12.04
$dpkg -l | grep -E " apache2 "
ii apache2 2.2.22-1ubuntu1.4 Apache HTTP Server metapackage
Logs:
syslog for 1st crash: http://
syslog for 2nd crash: http://
Bryce Harrington (bryce) wrote : | #23 |
Closing as expired since this was against Apache 2.2 and php5 but all supported Ubuntu releases are on Apache 2.4 and php7 now.
It doesn't look like a stack trace was collected but from the comments and log files it kind of sounds like an incompatibility between apache and some other components.
If anyone is having this or a similar problem on current versions of Ubuntu, please file a new bug report with fresh logs and stack trace.
Changed in apache2 (Ubuntu): | |
status: | Confirmed → Invalid |
Thank you for taking the time to report this bug and helping to make Ubuntu better.
Please could you confirm the version of the apache2 package in use at the time the crash occurred? And have you reproduced this problem on any other hardware apart from the server affected?
Once you've answered, please change the bug status back to New.
Setting Importance to Medium for now. As apache2 is widely used on 12.04 and I haven't seen any other reports, I think it's reasonable to assume that this crash does not affect a wide number of Ubuntu users until others can confirm it.