API Service errors during the load testing: status code 500 and 502

[Timur Nurlygayanov]

Magneto DB API service returns incorrect responses during the performance testing.

When we perform 800+ request per second to API service, we can see several responses with status code '500' and status code '502'.
As a result about 6 % of responses have statuses with error codes, when we perform 1000 request per second.

(please, see attached screenshot with performance tests results)

Error messages in magnetodb log files after the testing (also added to the bug description in comments):
http://paste.openstack.org/show/61928/
http://paste.openstack.org/show/61925/

We have send the following requests to MagnetoDB API service:
195
POST / HTTP/1.0
User-Agent: yandex-tank/1.1.1
Host: 172.18.169.205:8080
Content-Length:2
X-Amz-Target: DynamoDB_20120810.ListTables
Content-Type: application/x-amz-json-1.0
Connection: Close

{}

316
POST / HTTP/1.0
User-Agent: yandex-tank/1.1.1
Host: 172.18.169.205:8080
Content-Length:123
X-Amz-Target: DynamoDB_20120810.GetItem
Content-Type: application/x-amz-json-1.0
Connection: Close

{"TableName": "load_spam_table", "Key": {"user_id": {"S": "<email address hidden>"}, "date_message_id": {"S": "2013-12-31#123456"}}}

[Timur Nurlygayanov (tnurlygayanov)]

Errors in Magneto DB Logs:

2014-01-21 06:13:32.622 8448 TRACE gunicorn.error
2014-01-21 06:13:32.630 8448 ERROR gunicorn.error [-] Error handling request
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error Traceback (most recent call last):
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/gunicorn/workers/async.py", line 45, in handle
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error self.handle_request(listener, req, client, addr)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/gunicorn/workers/ggevent.py", line 151, in handle_request
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error super(GeventWorker, self).handle_request(*args)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/gunicorn/workers/async.py", line 93, in handle_request
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error respiter = self.wsgi(environ, resp.start_response)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error resp = self.call_func(req, *args, **self.kwargs)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error return self.func(req, *args, **kwargs)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/magnetodb/common/wsgi.py", line 50, in __call__
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error response = req.get_response(self.application)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error application, catch_exc_info=False)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error app_iter = application(self.environ, start_response)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error return resp(environ, start_response)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error response = self.app(environ, start_response)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error return resp(environ, start_response)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error resp = self.call_func(req, *args, **self.kwargs)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
2014-01-21 06:13:32.630 8...