AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes - Can't access maas server URL

Bug #1628535 reported by Larry Michel on 2016-09-28
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Undecided
Unassigned
apache2 (Ubuntu)
Undecided
Unassigned

Bug Description

We stopped being able to access maas through its web address and this caused a crash:

  File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line 137, in get_tagged_nodes
    all_nodes_list = self.nodes_list(user)
  File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line 378, in nodes_list
    return json.loads(self.cmd(command))
  File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line 281, in cmd
    content = self.client.get(command).read()
  File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 237, in get
    url, method="GET", headers=headers)
  File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 116, in dispatch_query
    res = urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused>

On the apache2 error log, I saw:

ue Sep 27 06:25:12.159028 2016] [mpm_event:notice] [pid 1895:tid 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Tue Sep 27 06:25:12.159257 2016] [core:notice] [pid 1895:tid 140037883340672] AH00094: Command line: '/usr/sbin/apache2'
[Wed Sep 28 06:25:09.318844 2016] [mpm_event:notice] [pid 1895:tid 140037883340672] AH00493: SIGUSR1 received. Doing graceful restart
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
[Wed Sep 28 06:25:12.038905 2016] [mpm_event:notice] [pid 1895:tid 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Wed Sep 28 06:25:12.038944 2016] [core:notice] [pid 1895:tid 140037883340672] AH00094: Command line: '/usr/sbin/apache2'
[Wed Sep 28 06:25:14.040425 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:15.041505 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:16.042554 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:17.043605 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:18.045062 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:19.046130 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:20.047475 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:21.048295 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
... <repeats non-stop>
ull, not at MaxRequestWorkers
[Wed Sep 28 12:53:25.662213 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 12:53:26.663278 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 12:53:27.664325 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 12:53:28.665413 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 12:53:29.666492 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers

The errors stopped when I restarted the apache2 service with sudo service apache2 restart.

I couldn't tell where to find the core. There was only this file from a week ago:
$ ls -l _usr_sbin_apache2.0.crash
-rw-r----- 1 root root 2526598 Sep 21 22:33 _usr_sbin_apache2.0.crash

I tried to rerun it again and timestamp changed but core info did not appear to change.

I have collected access and error logs of the past couple of days as well as other logs including maas logs. Tarball is attached to this bug and it contains the 2 crash files:

$ ls -l _usr_sbin_apache2.0.crash.*
-rw-r----- 1 ubuntu ubuntu 2698755 Sep 28 13:42 _usr_sbin_apache2.0.crash.new
-rw-r----- 1 ubuntu ubuntu 2526598 Sep 21 22:33 _usr_sbin_apache2.0.crash.old

These are the package versions:

ii maas 2.0.0+bzr5189-0 all "Metal as a Service" is a physical cloud and I
ii apache2 2.4.18-2ubuntu3.1 amd64 Apache HTTP Server
ii apache2-bin 2.4.18-2ubuntu3.1 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.18-2ubuntu3.1 all Apache HTTP Server (common files)
ii apache2-utils 2.4.18-2ubuntu3.1 amd64 Apache HTTP Server (utility programs for web servers)
ii init 1.29ubuntu2 amd64 System-V-like init utilities - metapackage

$ lsb_release -rd
Description: Ubuntu 16.04.1 LTS
Release: 16.04

Tags: oil Edit Tag help
Larry Michel (lmic) wrote :
summary: - AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes
+ AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes -
+ Can't access maas server URL
ChristianEhrhardt (paelzer) wrote :

Hi Larry,
I can't spot anything out of the log other than what you listed.
Those errors usually indicate either:
- non thread safe software with worker
- just overloading with more threads than allowed per config

For the latter you could take a look at:
- https://www.tectut.com/2016/04/workaround-for-scoreboard-is-full-not-at-maxrequestworkers/
- http://serverfault.com/questions/516373/what-is-the-meaning-of-ah00485-scoreboard-is-full-not-at-maxrequestworkers

For the former depending on your config it might be https://bz.apache.org/bugzilla/show_bug.cgi?id=48471

Could you take a look and see if any of that resolves the issue for you?

Download full text (7.0 KiB)

Larry,

Can you discontinue accessing MAAS over port 80 and start accessing it
through port 5240? And see if you have issues.

Apache does a bad job proxying!

On Wednesday, September 28, 2016, Larry Michel <email address hidden>
wrote:

> Public bug reported:
>
> We stopped being able to access maas through its web address and this
> caused a crash:
>
> File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line
> 137, in get_tagged_nodes
> all_nodes_list = self.nodes_list(user)
> File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line
> 378, in nodes_list
> return json.loads(self.cmd(command))
> File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line
> 281, in cmd
> content = self.client.get(command).read()
> File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line
> 237, in get
> url, method="GET", headers=headers)
> File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line
> 116, in dispatch_query
> res = urllib2.urlopen(req)
> File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
> return _opener.open(url, data, timeout)
> File "/usr/lib/python2.7/urllib2.py", line 404, in open
> response = self._open(req, data)
> File "/usr/lib/python2.7/urllib2.py", line 422, in _open
> '_open', req)
> File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
> result = func(*args)
> File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
> return self.do_open(httplib.HTTPConnection, req)
> File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
> raise URLError(err)
> URLError: <urlopen error [Errno 111] Connection refused>
>
> On the apache2 error log, I saw:
>
> ue Sep 27 06:25:12.159028 2016] [mpm_event:notice] [pid 1895:tid
> 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming
> normal operations
> [Tue Sep 27 06:25:12.159257 2016] [core:notice] [pid 1895:tid
> 140037883340672] AH00094: Command line: '/usr/sbin/apache2'
> [Wed Sep 28 06:25:09.318844 2016] [mpm_event:notice] [pid 1895:tid
> 140037883340672] AH00493: SIGUSR1 received. Doing graceful restart
> AH00558: apache2: Could not reliably determine the server's fully
> qualified domain name, using 127.0.1.1. Set the 'ServerName' directive
> globally to suppress this message
> [Wed Sep 28 06:25:12.038905 2016] [mpm_event:notice] [pid 1895:tid
> 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming
> normal operations
> [Wed Sep 28 06:25:12.038944 2016] [core:notice] [pid 1895:tid
> 140037883340672] AH00094: Command line: '/usr/sbin/apache2'
> [Wed Sep 28 06:25:14.040425 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:15.041505 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:16.042554 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:17.043605 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28...

Read more...

Changed in maas:
status: New → Won't Fix
status: Won't Fix → Invalid
status: Invalid → Incomplete
milestone: none → 2.1.0
Changed in maas:
milestone: 2.1.0 → 2.1.1
Changed in maas:
milestone: 2.1.1 → 2.1.2
Changed in maas:
milestone: 2.1.2 → 2.1.3
Jason Hobbs (jason-hobbs) wrote :

marked as invalid since we were using the wrong port.

Changed in maas:
status: Incomplete → Invalid
Robie Basak (racb) on 2017-03-28
Changed in apache2 (Ubuntu):
status: New → Incomplete
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.