wsgi occasionally hangs and needs apache to be graceful'ed

[David Ames]

The rnr site apache logs show many of these in its error log:

[Wed Jan 11 16:10:19 2012] [error] [client 91.189.89.105] Script timed out before returning headers: django.wsgi

[Wed Jan 11 16:10:21 2012] [error] [client 91.189.89.105] (4)Interrupted system call: mod_wsgi (pid=4923): Unable to connect to WSGI daemon process 'reviews.ubuntu.com' on '/var/run/apache2/wsgi.1135.58.4.sock' after multiple attempts.

[Wed Jan 11 16:11:58 2012] [error] [client 91.189.89.105] Premature end of script headers: django.wsgi

[Wed Jan 11 16:11:58 2012] [error] [client 91.189.89.106] (2)No such file or directory: mod_wsgi (pid=3319): Unable to connect to WSGI daemon process 'reviews.ubuntu.com' on '/var/run/apache2/wsgi.1135.58.4.sock' after multiple attempts.

Webops then needs to graceful apache

[Anthony Lenton (elachuni)]

The closest oops report I can find for this incident on satsuma is OOPS 2202satsuma76540 that provides the following traceback:

Traceback (most recent call last):
  File "/srv/reviews.ubuntu.com/production/rnr-server/sourcecode/django/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/srv/reviews.ubuntu.com/production/rnr-server/sourcecode/django/django/views/decorators/vary.py", line 22, in inner_func
    response = func(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/piston/resource.py", line 74, in __call__
    if not self.authentication.is_authenticated(request):
  File "/srv/reviews.ubuntu.com/production/rnr-server/src/reviewsapp/auth.py", line 77, in is_authenticated
    self.prefetch_oauth_consumer(orequest)
  File "/srv/reviews.ubuntu.com/production/rnr-server/src/reviewsapp/auth.py", line 101, in prefetch_oauth_consumer
    openid_identifier=consumer_key, signature=request.get_parameter('oauth_signature'))
  File "/srv/reviews.ubuntu.com/production/rnr-server/src/reviewsapp/utilities.py", line 149, in get_data_for_account
    result = self.identity_provider.authentications.validate_token(
  File "/usr/lib/pymodules/python2.6/lazr/restfulclient/resource.py", line 301, in __getattr__
    return self.lp_get_parameter(attr)
  File "/usr/lib/pymodules/python2.6/lazr/restfulclient/resource.py", line 196, in lp_get_parameter
    self._ensure_representation()
  File "/usr/lib/pymodules/python2.6/lazr/restfulclient/resource.py", line 331, in _ensure_representation
    representation = self._root._browser.get(self._wadl_resource)
  File "/usr/lib/pymodules/python2.6/lazr/restfulclient/_browser.py", line 316, in get
    response, content = self._request(url, extra_headers=headers)
  File "/usr/lib/pymodules/python2.6/lazr/restfulclient/_browser.py", line 260, in _request
    str(url), method=method, body=data, headers=headers)
  File "/usr/lib/pymodules/python2.6/httplib2/__init__.py", line 1129, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/lib/pymodules/python2.6/lazr/restfulclient/_browser.py", line 154, in _request
    redirections, cachekey)
  File "/usr/lib/pymodules/python2.6/httplib2/__init__.py", line 901, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/lib/pymodules/python2.6/lazr/restfulclient/_browser.py", line 159, in _conn_request
    conn, request_uri, method, body, headers)
  File "/usr/lib/pymodules/python2.6/httplib2/__init__.py", line 871, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.6/httplib.py", line 986, in getresponse
    response.begin()
  File "/usr/lib/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.6/httplib.py", line 349, in _read_status
    line = self.fp.readline()
  File "/usr/lib/python2.6/socket.py", line 397, in readline
    data = recv(1)
  File "/usr/lib/python2.6/ssl.py", line 96, in <lambda>
    self.recv = lambda buflen=1024, flags=0: SSLSocket.recv(self...

[Haw Loeung (hloeung)]

It's happening right now as we speak:

hloeung@satsuma:/var/log/apache2$ tail -n 20 pay.ubuntu.com-error.log
[Thu Jan 19 07:38:39 2012] [error] [client 91.189.89.105] mod_wsgi (pid=27305): Exception occurred processing WSGI script '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
[Thu Jan 19 07:38:39 2012] [error] [client 91.189.89.105] TypeError: 'NoneType' object is not iterable
[Thu Jan 19 07:38:41 2012] [error] [client 91.189.89.105] mod_wsgi (pid=27305): Exception occurred processing WSGI script '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
[Thu Jan 19 07:38:41 2012] [error] [client 91.189.89.105] TypeError: 'NoneType' object is not iterable
[Thu Jan 19 07:38:45 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305): Exception occurred processing WSGI script '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
[Thu Jan 19 07:38:45 2012] [error] [client 91.189.89.106] TypeError: 'NoneType' object is not iterable
[Thu Jan 19 07:38:47 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305): Exception occurred processing WSGI script '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
[Thu Jan 19 07:38:47 2012] [error] [client 91.189.89.106] TypeError: 'NoneType' object is not iterable
[Thu Jan 19 07:38:49 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305): Exception occurred processing WSGI script '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
[Thu Jan 19 07:38:49 2012] [error] [client 91.189.89.106] TypeError: 'NoneType' object is not iterable
[Thu Jan 19 07:38:51 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305): Exception occurred processing WSGI script '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
[Thu Jan 19 07:38:51 2012] [error] [client 91.189.89.106] TypeError: 'NoneType' object is not iterable
[Thu Jan 19 07:38:53 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305): Exception occurred processing WSGI script '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
[Thu Jan 19 07:38:53 2012] [error] [client 91.189.89.106] TypeError: 'NoneType' object is not iterable
[Thu Jan 19 07:38:55 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305): Exception occurred processing WSGI script '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
[Thu Jan 19 07:38:55 2012] [error] [client 91.189.89.106] TypeError: 'NoneType' object is not iterable
[Thu Jan 19 07:38:57 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305): Exception occurred processing WSGI script '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
[Thu Jan 19 07:38:57 2012] [error] [client 91.189.89.106] TypeError: 'NoneType' object is not iterable
[Thu Jan 19 07:39:00 2012] [error] [client 91.189.89.105] mod_wsgi (pid=27305): Exception occurred processing WSGI script '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
[Thu Jan 19 07:39:00 2012] [error] [client 91.189.89.105] TypeError: 'NoneType' object is not iterable

hloeung@satsuma:/srv/pay.ubuntu.com/production-logs/www-logs$ tail -n 100 *.log
    response = middleware_method(request, response)
  File "/usr/lib/pymodules/python2....

[Ricardo Kirkner (ricardokirkner)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This issue is not new (though the SSL cipher issue is), so I have done
some research and found this

http://notes.alexdong.com/how-to-configure-djangomodwsgi-to-avoid-a-fro

I was thinking we could try updating our apache configs to not use
threads, but processes to see if this fixes the issue for us.

As an alternative, I was also thinking of ditching modwsgi in favour
of gunicorn.

Both approaches still need some testing, but I'd like your input on
this (does it sound any of the two approaches is feasible?)

On 01/19/2012 04:40 AM, Haw Loeung wrote:
> It's happening right now as we speak:
>
> hloeung@satsuma:/var/log/apache2$ tail -n 20
> pay.ubuntu.com-error.log [Thu Jan 19 07:38:39 2012] [error] [client
> 91.189.89.105] mod_wsgi (pid=27305): Exception occurred processing
> WSGI script
> '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
> [Thu Jan 19 07:38:39 2012] [error] [client 91.189.89.105]
> TypeError: 'NoneType' object is not iterable [Thu Jan 19 07:38:41
> 2012] [error] [client 91.189.89.105] mod_wsgi (pid=27305):
> Exception occurred processing WSGI script
> '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
> [Thu Jan 19 07:38:41 2012] [error] [client 91.189.89.105]
> TypeError: 'NoneType' object is not iterable [Thu Jan 19 07:38:45
> 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305):
> Exception occurred processing WSGI script
> '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
> [Thu Jan 19 07:38:45 2012] [error] [client 91.189.89.106]
> TypeError: 'NoneType' object is not iterable [Thu Jan 19 07:38:47
> 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305):
> Exception occurred processing WSGI script
> '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
> [Thu Jan 19 07:38:47 2012] [error] [client 91.189.89.106]
> TypeError: 'NoneType' object is not iterable [Thu Jan 19 07:38:49
> 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305):
> Exception occurred processing WSGI script
> '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
> [Thu Jan 19 07:38:49 2012] [error] [client 91.189.89.106]
> TypeError: 'NoneType' object is not iterable [Thu Jan 19 07:38:51
> 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305):
> Exception occurred processing WSGI script
> '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
> [Thu Jan 19 07:38:51 2012] [error] [client 91.189.89.106]
> TypeError: 'NoneType' object is not iterable [Thu Jan 19 07:38:53
> 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305):
> Exception occurred processing WSGI script
> '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
> [Thu Jan 19 07:38:53 2012] [error] [client 91.189.89.106]
> TypeError: 'NoneType' object is not iterable [Thu Jan 19 07:38:55
> 2012] [error] [client 91.189.89.106] mod_wsgi (pid=27305):
> Exception occurred processing WSGI script
> '/srv/pay.ubuntu.com/production/code/payments-config/django.wsgi'.
> [Thu Jan 19 07:38:55 2012] [error] [client 91.189.89.106]
> TypeError: 'NoneType' object is not iterable [Thu Jan 19 07:38:57
> 2012] [error] [client 91.189....

[Tom Haddon (mthaddon)]

This happened again, and I was seeing the same errors in the logs:

https://pastebin.canonical.com/69045/

I graceful-ed apache and it seemed to fix things.

[Tom Haddon (mthaddon)]

Oh, although this time the problem was with USN, not RNR - sorry

[Haw Loeung (hloeung)]

This is also happening for UWC on seko and saci.

[Sat Dec 22 23:45:06 2012] [warn] NameVirtualHost *:80 has no VirtualHosts
hloeung@saci:/var/log/apache2$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.1 LTS"
NAME="Ubuntu"
VERSION="12.04.1 LTS, Precise Pangolin"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu precise (12.04.1 LTS)"
VERSION_ID="12.04"

hloeung@saci:/var/log/apache2$ apt-cache policy apache2 python-django
apache2:
  Installed: 2.2.22-1ubuntu1.2
  Candidate: 2.2.22-1ubuntu1.2
  Version table:
 *** 2.2.22-1ubuntu1.2 0
        500 http://archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.2.22-1ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
python-django:
  Installed: 1.3.1-4ubuntu1.4
  Candidate: 1.3.1-4ubuntu1.4
  Version table:
 *** 1.3.1-4ubuntu1.4 0
        500 http://archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.3.1-4ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
hloeung@saci:/var/log/apache2$