AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes - Can't access maas server URL
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| MAAS |
Undecided
|
Unassigned | ||
| apache2 (Ubuntu) |
Undecided
|
Unassigned |
Bug Description
We stopped being able to access maas through its web address and this caused a crash:
File "/usr/lib/
all_nodes_list = self.nodes_
File "/usr/lib/
return json.loads(
File "/usr/lib/
content = self.client.
File "/usr/lib/
url, method="GET", headers=headers)
File "/usr/lib/
res = urllib2.
File "/usr/lib/
return _opener.open(url, data, timeout)
File "/usr/lib/
response = self._open(req, data)
File "/usr/lib/
'_open', req)
File "/usr/lib/
result = func(*args)
File "/usr/lib/
return self.do_
File "/usr/lib/
raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused>
On the apache2 error log, I saw:
ue Sep 27 06:25:12.159028 2016] [mpm_event:notice] [pid 1895:tid 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Tue Sep 27 06:25:12.159257 2016] [core:notice] [pid 1895:tid 140037883340672] AH00094: Command line: '/usr/sbin/apache2'
[Wed Sep 28 06:25:09.318844 2016] [mpm_event:notice] [pid 1895:tid 140037883340672] AH00493: SIGUSR1 received. Doing graceful restart
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
[Wed Sep 28 06:25:12.038905 2016] [mpm_event:notice] [pid 1895:tid 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Wed Sep 28 06:25:12.038944 2016] [core:notice] [pid 1895:tid 140037883340672] AH00094: Command line: '/usr/sbin/apache2'
[Wed Sep 28 06:25:14.040425 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:15.041505 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:16.042554 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:17.043605 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:18.045062 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:19.046130 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:20.047475 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 06:25:21.048295 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
... <repeats non-stop>
ull, not at MaxRequestWorkers
[Wed Sep 28 12:53:25.662213 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 12:53:26.663278 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 12:53:27.664325 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 12:53:28.665413 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
[Wed Sep 28 12:53:29.666492 2016] [mpm_event:error] [pid 1895:tid 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
The errors stopped when I restarted the apache2 service with sudo service apache2 restart.
I couldn't tell where to find the core. There was only this file from a week ago:
$ ls -l _usr_sbin_
-rw-r----- 1 root root 2526598 Sep 21 22:33 _usr_sbin_
I tried to rerun it again and timestamp changed but core info did not appear to change.
I have collected access and error logs of the past couple of days as well as other logs including maas logs. Tarball is attached to this bug and it contains the 2 crash files:
$ ls -l _usr_sbin_
-rw-r----- 1 ubuntu ubuntu 2698755 Sep 28 13:42 _usr_sbin_
-rw-r----- 1 ubuntu ubuntu 2526598 Sep 21 22:33 _usr_sbin_
These are the package versions:
ii maas 2.0.0+bzr5189-0 all "Metal as a Service" is a physical cloud and I
ii apache2 2.4.18-2ubuntu3.1 amd64 Apache HTTP Server
ii apache2-bin 2.4.18-2ubuntu3.1 amd64 Apache HTTP Server (modules and other binary files)
ii apache2-data 2.4.18-2ubuntu3.1 all Apache HTTP Server (common files)
ii apache2-utils 2.4.18-2ubuntu3.1 amd64 Apache HTTP Server (utility programs for web servers)
ii init 1.29ubuntu2 amd64 System-V-like init utilities - metapackage
$ lsb_release -rd
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Larry Michel (lmic) wrote : | #1 |
summary: |
- AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes + AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes - + Can't access maas server URL |
Christian Ehrhardt (paelzer) wrote : | #2 |
Andres Rodriguez (andreserl) wrote : Re: [Bug 1628535] [NEW] AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes - Can't access maas server URL | #3 |
Larry,
Can you discontinue accessing MAAS over port 80 and start accessing it
through port 5240? And see if you have issues.
Apache does a bad job proxying!
On Wednesday, September 28, 2016, Larry Michel <email address hidden>
wrote:
> Public bug reported:
>
> We stopped being able to access maas through its web address and this
> caused a crash:
>
> File "/usr/lib/
> 137, in get_tagged_nodes
> all_nodes_list = self.nodes_
> File "/usr/lib/
> 378, in nodes_list
> return json.loads(
> File "/usr/lib/
> 281, in cmd
> content = self.client.
> File "/usr/lib/
> 237, in get
> url, method="GET", headers=headers)
> File "/usr/lib/
> 116, in dispatch_query
> res = urllib2.
> File "/usr/lib/
> return _opener.open(url, data, timeout)
> File "/usr/lib/
> response = self._open(req, data)
> File "/usr/lib/
> '_open', req)
> File "/usr/lib/
> result = func(*args)
> File "/usr/lib/
> return self.do_
> File "/usr/lib/
> raise URLError(err)
> URLError: <urlopen error [Errno 111] Connection refused>
>
> On the apache2 error log, I saw:
>
> ue Sep 27 06:25:12.159028 2016] [mpm_event:notice] [pid 1895:tid
> 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming
> normal operations
> [Tue Sep 27 06:25:12.159257 2016] [core:notice] [pid 1895:tid
> 140037883340672] AH00094: Command line: '/usr/sbin/apache2'
> [Wed Sep 28 06:25:09.318844 2016] [mpm_event:notice] [pid 1895:tid
> 140037883340672] AH00493: SIGUSR1 received. Doing graceful restart
> AH00558: apache2: Could not reliably determine the server's fully
> qualified domain name, using 127.0.1.1. Set the 'ServerName' directive
> globally to suppress this message
> [Wed Sep 28 06:25:12.038905 2016] [mpm_event:notice] [pid 1895:tid
> 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming
> normal operations
> [Wed Sep 28 06:25:12.038944 2016] [core:notice] [pid 1895:tid
> 140037883340672] AH00094: Command line: '/usr/sbin/apache2'
> [Wed Sep 28 06:25:14.040425 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:15.041505 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:16.042554 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:17.043605 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28...
Changed in maas: | |
status: | New → Won't Fix |
status: | Won't Fix → Invalid |
status: | Invalid → Incomplete |
milestone: | none → 2.1.0 |
Changed in maas: | |
milestone: | 2.1.0 → 2.1.1 |
Changed in maas: | |
milestone: | 2.1.1 → 2.1.2 |
Changed in maas: | |
milestone: | 2.1.2 → 2.1.3 |
Jason Hobbs (jason-hobbs) wrote : | #4 |
marked as invalid since we were using the wrong port.
Changed in maas: | |
status: | Incomplete → Invalid |
Changed in apache2 (Ubuntu): | |
status: | New → Incomplete |
Bryce Harrington (bryce) wrote : | #5 |
Closing as resolved per comment #4
Changed in apache2 (Ubuntu): | |
status: | Incomplete → Invalid |
Hi Larry,
I can't spot anything out of the log other than what you listed.
Those errors usually indicate either:
- non thread safe software with worker
- just overloading with more threads than allowed per config
For the latter you could take a look at: /www.tectut. com/2016/ 04/workaround- for-scoreboard- is-full- not-at- maxrequestworke rs/ serverfault. com/questions/ 516373/ what-is- the-meaning- of-ah00485- scoreboard- is-full- not-at- maxrequestworke rs
- https:/
- http://
For the former depending on your config it might be https:/ /bz.apache. org/bugzilla/ show_bug. cgi?id= 48471
Could you take a look and see if any of that resolves the issue for you?