MagnetoDB

operation timeout on update item request

Bug #1415478 reported by Andrii Ostapenko on 2015-01-28

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	MagnetoDB	Triaged	High	Unassigned	MagnetoDB liberty-1 "l1"

Bug Description

Running this test:
https://review.openstack.org/#/c/150943/7/contrib/tempest/tempest/api/keyvalue/in_progress/rest/test_concurrent_writes.py

I periodically get on server side:

2015-01-28 13:35:28.366 2868 ERROR magnetodb.common.middleware.fault [req-5a39db2f-78fc-4d49-8bfc-4f7430c6087e 6a8093949b4a414bb2fcdc7d0926c1a8 dc845364ad194c50a0242f8d61d1ee80] code=1100 [Coordinator node
timed out waiting for replica nodes' responses] message="Operation timed out - received only 1 responses." info={'received_responses': 1, 'required_responses': 2, 'consistency': 'SERIAL'}
Traceback (most recent call last):
  File "/opt/stack/magnetodb/magnetodb/common/middleware/fault.py", line 115, in process_request
    return req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 61, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 823, in __call__
    return self._call_app(env, start_response)
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 758, in _call_app
    return self._app(env, _fake_start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 61, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 244, in __call__
    action_result = self.execute_action(action, request, **action_args)
  File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 258, in execute_action
    return self.dispatch(self.controller, action, request, **action_args)
  File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 267, in dispatch
    return method(*args, **kwargs)
  File "/opt/stack/magnetodb/magnetodb/api/__init__.py", line 35, in wrapper
    return f(self, *args, **kwargs)
  File "/opt/stack/magnetodb/magnetodb/common/probe.py", line 94, in wrapper
    return f(*args, **kwargs)
  File "/opt/stack/magnetodb/magnetodb/api/openstack/v1/data/update_item.py", line 84, in process_request
    expected_condition_map=expected_item_conditions)
  File "/opt/stack/magnetodb/magnetodb/storage/__init__.py", line 304, in update_item
    expected_condition_map
  File "/opt/stack/magnetodb/magnetodb/storage/manager/simple_impl.py", line 668, in update_item
    expected_condition_map
  File "/opt/stack/magnetodb/magnetodb/common/probe.py", line 94, in wrapper
    return f(*args, **kwargs)
  File "/opt/stack/magnetodb/magnetodb/storage/driver/cassandra/cassandra_with_custom_lsi_impl.py", line 915, in update_item
    "".join(query_builder), consistent=True
  File "/opt/stack/magnetodb/magnetodb/common/cassandra/cluster_handler.py", line 165, in execute_query
    raise ex
WriteTimeout: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 1 responses." info={'received_responses': 1, 'required_responses': 2, 'consistency': 'SERIAL'}

See original description

Revision history for this message

Ilya Sviridov (isviridov) wrote on 2015-01-28:

oastapenko, please provide request details

Changed in magnetodb:
status:	New → Incomplete

Andrii Ostapenko (aostapenko) on 2015-01-28

description:

updated

Revision history for this message

Ilya Sviridov (isviridov) wrote on 2015-01-28: Re: [Bug 1415478] Re: operation timeout on update item request

Download full text (15.9 KiB)

aostapenko, could you please specify json request in format you sending it
via http or code sample of library you are using
On Jan 28, 2015 5:10 PM, "Andrei V. Ostapenko" <
<email address hidden>> wrote:

aostapenko, could you please specify json request in format you sending it
via http or code sample of library you are using
On Jan 28, 2015 5:10 PM, "Andrei V. Ostapenko" <
Andrey_Ostapenko@symantec.com> wrote:

> ** Description changed:
>
> + Send multiple update item requests concurrently:
> +
> + {'extra_attr1': {'action': 'PUT', 'value': {'S': 'extra_value1'}}},
> + 'time_to_live': None, 'return_values': None, 'key': {'range_attr': {'S':
> + 'range_value'}, 'hash_attr': {'S': 'hash_value'}}}
> +
> + I'm periodically getting:
> +
>   2015-01-28 13:35:28.366 2868 ERROR magnetodb.common.middleware.fault
> [req-5a39db2f-78fc-4d49-8bfc-4f7430c6087e 6a8093949b4a414bb2fcdc7d0926c1a8
> dc845364ad194c50a0242f8d61d1ee80] code=1100 [Coordinator node
> -  timed out waiting for replica nodes' responses] message="Operation
> timed out - received only 1 responses." info={'received_responses': 1,
> 'required_responses': 2, 'consistency': 'SERIAL'}
> +  timed out waiting for replica nodes' responses] message="Operation
> timed out - received only 1 responses." info={'received_responses': 1,
> 'required_responses': 2, 'consistency': 'SERIAL'}
>   Traceback (most recent call last):
> -   File "/opt/stack/magnetodb/magnetodb/common/middleware/fault.py", line
> 115, in process_request
> -     return req.get_response(self.application)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1320, in send
> -     application, catch_exc_info=False)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1284, in call_application
> -     app_iter = application(self.environ, start_response)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130,
> in __call__
> -     resp = self.call_func(req, *args, **self.kwargs)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195,
> in call_func
> -     return self.func(req, *args, **kwargs)
> -   File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 61, in
> __call__
> -     response = req.get_response(self.application)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1320, in send
> -     application, catch_exc_info=False)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1284, in call_application
> -     app_iter = application(self.environ, start_response)
> -   File
> "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py",
> line 823, in __call__
> -     return self._call_app(env, start_response)
> -   File
> "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py",
> line 758, in _call_app
> -     return self._app(env, _fake_start_response)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130,
> in __call__
> -     resp = self.call_func(req, *args, **self.kwargs)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195,
> in call_func
> -     return self.func(req, *args, **kwargs)
> -   File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 61, in
> __call__
> -     response = req.get_response(self.application)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1320, in send
> -     application, catch_exc_info=False)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1284, in call_application
> -     app_iter = application(self.environ, start_response)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144,
> in __call__
> -     return resp(environ, start_response)
> -   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py",
> line 131, in __call__
> -     response = self.app(environ, start_response)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144,
> in __call__
> -     return resp(environ, start_response)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130,
> in __call__
> -     resp = self.call_func(req, *args, **self.kwargs)
> -   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195,
> in call_func
> -     return self.func(req, *args, **kwargs)
> -   File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 244, in
> __call__
> -     action_result = self.execute_action(action, request, **action_args)
> -   File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 258, in
> execute_action
> -     return self.dispatch(self.controller, action, request, **action_args)
> -   File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 267, in
> dispatch
> -     return method(*args, **kwargs)
> -   File "/opt/stack/magnetodb/magnetodb/api/__init__.py", line 35, in
> wrapper
> -     return f(self, *args, **kwargs)
> -   File "/opt/stack/magnetodb/magnetodb/common/probe.py", line 94, in
> wrapper
> -     return f(*args, **kwargs)
> -   File
> "/opt/stack/magnetodb/magnetodb/api/openstack/v1/data/update_item.py", line
> 84, in process_request
> -     expected_condition_map=expected_item_conditions)
> -   File "/opt/stack/magnetodb/magnetodb/storage/__init__.py", line 304,
> in update_item
> -     expected_condition_map
> -   File "/opt/stack/magnetodb/magnetodb/storage/manager/simple_impl.py",
> line 668, in update_item
> -     expected_condition_map
> -   File "/opt/stack/magnetodb/magnetodb/common/probe.py", line 94, in
> wrapper
> -     return f(*args, **kwargs)
> -   File
> "/opt/stack/magnetodb/magnetodb/storage/driver/cassandra/cassandra_with_custom_lsi_impl.py",
> line 915, in update_item
> -     "".join(query_builder), consistent=True
> -   File
> "/opt/stack/magnetodb/magnetodb/common/cassandra/cluster_handler.py", line
> 165, in execute_query
> -     raise ex
> +   File "/opt/stack/magnetodb/magnetodb/common/middleware/fault.py", line
> 115, in process_request
> +     return req.get_response(self.application)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1320, in send
> +     application, catch_exc_info=False)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1284, in call_application
> +     app_iter = application(self.environ, start_response)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130,
> in __call__
> +     resp = self.call_func(req, *args, **self.kwargs)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195,
> in call_func
> +     return self.func(req, *args, **kwargs)
> +   File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 61, in
> __call__
> +     response = req.get_response(self.application)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1320, in send
> +     application, catch_exc_info=False)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1284, in call_application
> +     app_iter = application(self.environ, start_response)
> +   File
> "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py",
> line 823, in __call__
> +     return self._call_app(env, start_response)
> +   File
> "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py",
> line 758, in _call_app
> +     return self._app(env, _fake_start_response)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130,
> in __call__
> +     resp = self.call_func(req, *args, **self.kwargs)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195,
> in call_func
> +     return self.func(req, *args, **kwargs)
> +   File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 61, in
> __call__
> +     response = req.get_response(self.application)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1320, in send
> +     application, catch_exc_info=False)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1284, in call_application
> +     app_iter = application(self.environ, start_response)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144,
> in __call__
> +     return resp(environ, start_response)
> +   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py",
> line 131, in __call__
> +     response = self.app(environ, start_response)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144,
> in __call__
> +     return resp(environ, start_response)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130,
> in __call__
> +     resp = self.call_func(req, *args, **self.kwargs)
> +   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195,
> in call_func
> +     return self.func(req, *args, **kwargs)
> +   File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 244, in
> __call__
> +     action_result = self.execute_action(action, request, **action_args)
> +   File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 258, in
> execute_action
> +     return self.dispatch(self.controller, action, request, **action_args)
> +   File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 267, in
> dispatch
> +     return method(*args, **kwargs)
> +   File "/opt/stack/magnetodb/magnetodb/api/__init__.py", line 35, in
> wrapper
> +     return f(self, *args, **kwargs)
> +   File "/opt/stack/magnetodb/magnetodb/common/probe.py", line 94, in
> wrapper
> +     return f(*args, **kwargs)
> +   File
> "/opt/stack/magnetodb/magnetodb/api/openstack/v1/data/update_item.py", line
> 84, in process_request
> +     expected_condition_map=expected_item_conditions)
> +   File "/opt/stack/magnetodb/magnetodb/storage/__init__.py", line 304,
> in update_item
> +     expected_condition_map
> +   File "/opt/stack/magnetodb/magnetodb/storage/manager/simple_impl.py",
> line 668, in update_item
> +     expected_condition_map
> +   File "/opt/stack/magnetodb/magnetodb/common/probe.py", line 94, in
> wrapper
> +     return f(*args, **kwargs)
> +   File
> "/opt/stack/magnetodb/magnetodb/storage/driver/cassandra/cassandra_with_custom_lsi_impl.py",
> line 915, in update_item
> +     "".join(query_builder), consistent=True
> +   File
> "/opt/stack/magnetodb/magnetodb/common/cassandra/cluster_handler.py", line
> 165, in execute_query
> +     raise ex
>   WriteTimeout: code=1100 [Coordinator node timed out waiting for replica
> nodes' responses] message="Operation timed out - received only 1
> responses." info={'received_responses': 1, 'required_responses': 2,
> 'consistency': 'SERIAL'}
>
> --
> You received this bug notification because you are subscribed to
> MagnetoDB.
> Matching subscriptions: magnetodb-bugs
> https://bugs.launchpad.net/bugs/1415478
>
> Title:
>   operation timeout on update item request
>
> Status in MagnetoDB - key-value storage service for OpenStack:
>   Incomplete
>
> Bug description:
>   Send multiple update item requests concurrently:
>
>   {'extra_attr1': {'action': 'PUT', 'value': {'S': 'extra_value1'}}},
>   'time_to_live': None, 'return_values': None, 'key': {'range_attr':
>   {'S': 'range_value'}, 'hash_attr': {'S': 'hash_value'}}}
>
>   I'm periodically getting:
>
>   2015-01-28 13:35:28.366 2868 ERROR magnetodb.common.middleware.fault
> [req-5a39db2f-78fc-4d49-8bfc-4f7430c6087e 6a8093949b4a414bb2fcdc7d0926c1a8
> dc845364ad194c50a0242f8d61d1ee80] code=1100 [Coordinator node
>    timed out waiting for replica nodes' responses] message="Operation
> timed out - received only 1 responses." info={'received_responses': 1,
> 'required_responses': 2, 'consistency': 'SERIAL'}
>   Traceback (most recent call last):
>     File "/opt/stack/magnetodb/magnetodb/common/middleware/fault.py", line
> 115, in process_request
>       return req.get_response(self.application)
>     File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1320, in send
>       application, catch_exc_info=False)
>     File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1284, in call_application
>       app_iter = application(self.environ, start_response)
>     File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130,
> in __call__
>       resp = self.call_func(req, *args, **self.kwargs)
>     File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195,
> in call_func
>       return self.func(req, *args, **kwargs)
>     File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 61, in
> __call__
>       response = req.get_response(self.application)
>     File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1320, in send
>       application, catch_exc_info=False)
>     File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1284, in call_application
>       app_iter = application(self.environ, start_response)
>     File
> "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py",
> line 823, in __call__
>       return self._call_app(env, start_response)
>     File
> "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py",
> line 758, in _call_app
>       return self._app(env, _fake_start_response)
>     File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130,
> in __call__
>       resp = self.call_func(req, *args, **self.kwargs)
>     File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195,
> in call_func
>       return self.func(req, *args, **kwargs)
>     File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 61, in
> __call__
>       response = req.get_response(self.application)
>     File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1320, in send
>       application, catch_exc_info=False)
>     File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line
> 1284, in call_application
>       app_iter = application(self.environ, start_response)
>     File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144,
> in __call__
>       return resp(environ, start_response)
>     File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py",
> line 131, in __call__
>       response = self.app(environ, start_response)
>     File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144,
> in __call__
>       return resp(environ, start_response)
>     File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130,
> in __call__
>       resp = self.call_func(req, *args, **self.kwargs)
>     File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195,
> in call_func
>       return self.func(req, *args, **kwargs)
>     File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 244, in
> __call__
>       action_result = self.execute_action(action, request, **action_args)
>     File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 258, in
> execute_action
>       return self.dispatch(self.controller, action, request, **action_args)
>     File "/opt/stack/magnetodb/magnetodb/common/wsgi.py", line 267, in
> dispatch
>       return method(*args, **kwargs)
>     File "/opt/stack/magnetodb/magnetodb/api/__init__.py", line 35, in
> wrapper
>       return f(self, *args, **kwargs)
>     File "/opt/stack/magnetodb/magnetodb/common/probe.py", line 94, in
> wrapper
>       return f(*args, **kwargs)
>     File
> "/opt/stack/magnetodb/magnetodb/api/openstack/v1/data/update_item.py", line
> 84, in process_request
>       expected_condition_map=expected_item_conditions)
>     File "/opt/stack/magnetodb/magnetodb/storage/__init__.py", line 304,
> in update_item
>       expected_condition_map
>     File "/opt/stack/magnetodb/magnetodb/storage/manager/simple_impl.py",
> line 668, in update_item
>       expected_condition_map
>     File "/opt/stack/magnetodb/magnetodb/common/probe.py", line 94, in
> wrapper
>       return f(*args, **kwargs)
>     File
> "/opt/stack/magnetodb/magnetodb/storage/driver/cassandra/cassandra_with_custom_lsi_impl.py",
> line 915, in update_item
>       "".join(query_builder), consistent=True
>     File
> "/opt/stack/magnetodb/magnetodb/common/cassandra/cluster_handler.py", line
> 165, in execute_query
>       raise ex
>   WriteTimeout: code=1100 [Coordinator node timed out waiting for replica
> nodes' responses] message="Operation timed out - received only 1
> responses." info={'received_responses': 1, 'required_responses': 2,
> 'consistency': 'SERIAL'}
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/magnetodb/+bug/1415478/+subscriptions
>

Andrii Ostapenko (aostapenko) on 2015-01-29

description:

updated

Ilya Sviridov (isviridov) on 2015-02-04

Changed in magnetodb:
status:	Incomplete → Triaged
importance:	Undecided → High
milestone:	none → kilo-2

Ilya Sviridov (isviridov) on 2015-02-05

Changed in magnetodb:
milestone:	kilo-2 → kilo-3

Ilya Sviridov (isviridov) on 2015-03-20

Changed in magnetodb:
milestone:	kilo-3 → l-1

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Related blueprints

Add tests for testing concurrent writes

Remote bug watches

Bug watches keep track of this bug in other bug trackers.