Ubuntu
apache2 package

  1. Bug #1628535
  2. Comment #3

Comment 3 for bug 1628535

Revision history for this message
Andres Rodriguez (andreserl) wrote : Re: [Bug 1628535] [NEW] AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes - Can't access maas server URL

Larry,

Can you discontinue accessing MAAS over port 80 and start accessing it
through port 5240? And see if you have issues.

Apache does a bad job proxying!

On Wednesday, September 28, 2016, Larry Michel <email address hidden>
wrote:

> Public bug reported:
>
> We stopped being able to access maas through its web address and this
> caused a crash:
>
> File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line
> 137, in get_tagged_nodes
> all_nodes_list = self.nodes_list(user)
> File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line
> 378, in nodes_list
> return json.loads(self.cmd(command))
> File "/usr/lib/python2.7/dist-packages/oil_ci/common/maas.py", line
> 281, in cmd
> content = self.client.get(command).read()
> File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line
> 237, in get
> url, method="GET", headers=headers)
> File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line
> 116, in dispatch_query
> res = urllib2.urlopen(req)
> File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
> return _opener.open(url, data, timeout)
> File "/usr/lib/python2.7/urllib2.py", line 404, in open
> response = self._open(req, data)
> File "/usr/lib/python2.7/urllib2.py", line 422, in _open
> '_open', req)
> File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
> result = func(*args)
> File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
> return self.do_open(httplib.HTTPConnection, req)
> File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
> raise URLError(err)
> URLError: <urlopen error [Errno 111] Connection refused>
>
> On the apache2 error log, I saw:
>
> ue Sep 27 06:25:12.159028 2016] [mpm_event:notice] [pid 1895:tid
> 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming
> normal operations
> [Tue Sep 27 06:25:12.159257 2016] [core:notice] [pid 1895:tid
> 140037883340672] AH00094: Command line: '/usr/sbin/apache2'
> [Wed Sep 28 06:25:09.318844 2016] [mpm_event:notice] [pid 1895:tid
> 140037883340672] AH00493: SIGUSR1 received. Doing graceful restart
> AH00558: apache2: Could not reliably determine the server's fully
> qualified domain name, using 127.0.1.1. Set the 'ServerName' directive
> globally to suppress this message
> [Wed Sep 28 06:25:12.038905 2016] [mpm_event:notice] [pid 1895:tid
> 140037883340672] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming
> normal operations
> [Wed Sep 28 06:25:12.038944 2016] [core:notice] [pid 1895:tid
> 140037883340672] AH00094: Command line: '/usr/sbin/apache2'
> [Wed Sep 28 06:25:14.040425 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:15.041505 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:16.042554 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:17.043605 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:18.045062 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:19.046130 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:20.047475 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 06:25:21.048295 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> ... <repeats non-stop>
> ull, not at MaxRequestWorkers
> [Wed Sep 28 12:53:25.662213 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 12:53:26.663278 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 12:53:27.664325 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 12:53:28.665413 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Wed Sep 28 12:53:29.666492 2016] [mpm_event:error] [pid 1895:tid
> 140037883340672] AH00485: scoreboard is full, not at MaxRequestWorkers
>
>
> The errors stopped when I restarted the apache2 service with sudo service
> apache2 restart.
>
> I couldn't tell where to find the core. There was only this file from a
> week ago:
> $ ls -l _usr_sbin_apache2.0.crash
> -rw-r----- 1 root root 2526598 Sep 21 22:33 _usr_sbin_apache2.0.crash
>
> I tried to rerun it again and timestamp changed but core info did not
> appear to change.
>
> I have collected access and error logs of the past couple of days as
> well as other logs including maas logs. Tarball is attached to this bug
> and it contains the 2 crash files:
>
> $ ls -l _usr_sbin_apache2.0.crash.*
> -rw-r----- 1 ubuntu ubuntu 2698755 Sep 28 13:42
> _usr_sbin_apache2.0.crash.new
> -rw-r----- 1 ubuntu ubuntu 2526598 Sep 21 22:33
> _usr_sbin_apache2.0.crash.old
>
>
> These are the package versions:
>
> ii maas 2.0.0+bzr5189-0 all "Metal as a
> Service" is a physical cloud and I
> ii apache2 2.4.18-2ubuntu3.1
> amd64 Apache HTTP Server
> ii apache2-bin 2.4.18-2ubuntu3.1
> amd64 Apache HTTP Server (modules and other binary files)
> ii apache2-data 2.4.18-2ubuntu3.1
> all Apache HTTP Server (common files)
> ii apache2-utils 2.4.18-2ubuntu3.1
> amd64 Apache HTTP Server (utility programs for web servers)
> ii init 1.29ubuntu2
> amd64 System-V-like init utilities - metapackage
>
>
> $ lsb_release -rd
> Description: Ubuntu 16.04.1 LTS
> Release: 16.04
>
> ** Affects: maas
> Importance: Undecided
> Status: New
>
> ** Affects: apache2 (Ubuntu)
> Importance: Undecided
> Status: New
>
>
> ** Tags: oil
>
> ** Attachment added: "apache2.logs.tar.gz"
> https://bugs.launchpad.net/bugs/1628535/+attachment/
> 4750157/+files/apache2.logs.tar.gz
>
> ** Summary changed:
>
> - AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes
> + AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes -
> Can't access maas server URL
>
> ** Also affects: maas
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1628535
>
> Title:
> AH00493: SIGUSR1 received. Doing graceful restart - Apache2 crashes -
> Can't access maas server URL
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1628535/+subscriptions
>

--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer