commit reservation failed due to size too large

Bug #1669449 reported by TommyLike
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Undecided
Valeriy Ponomaryov

Bug Description

Currently we don't raise exception if the reservation.commit operation failed, such as size exceed the max value of integer, what's more, these dirty reservation could obstuct the future requests.

```
File "/opt/stack/manila/manila/api/middleware/fault.py", line 72, in __call__
  return req.get_response(self.application)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
  application, catch_exc_info=False)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
  app_iter = application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  return self.func(req, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
  response = req.get_response(self.application)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
  application, catch_exc_info=False)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
  app_iter = application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  return resp(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  return self.func(req, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 335, in __call__
  response = req.get_response(self._app)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
  application, catch_exc_info=False)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
  app_iter = application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  return resp(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  return resp(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 141, in __call__
  response = self.app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  return resp(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  return self.func(req, *args, **kwargs)
File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 761, in __call__
  content_type, body, accept)
File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 819, in _process_stack
  action_result = self.dispatch(meth, request, action_args)
File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 909, in dispatch
  return method(req=request, **action_args)
File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 1051, in version_select
  return func.func(self, *args, **kwargs)
File "/opt/stack/manila/manila/api/v2/shares.py", line 183, in create
  req, body, check_create_share_from_snapshot_support=True)
File "/opt/stack/manila/manila/api/v1/shares.py", line 329, in _create
  **kwargs)
File "/opt/stack/manila/manila/share/api.py", line 141, in create
  reservations = QUOTAS.reserve(context, shares=1, gigabytes=size)
File "/opt/stack/manila/manila/quota.py", line 966, in reserve
  user_id=user_id)
File "/opt/stack/manila/manila/quota.py", line 469, in reserve
  project_id=project_id, user_id=user_id)
File "/opt/stack/manila/manila/db/api.py", line 239, in quota_reserve
  project_id=project_id, user_id=user_id)
File "/opt/stack/manila/manila/db/sqlalchemy/api.py", line 164, in wrapper
  return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 144, in wrapper
  ectxt.reraise = not expected
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  self.force_reraise()
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  six.reraise(self.type_, self.value, self.tb)
File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
  return f(*args, **kwargs)
File "/opt/stack/manila/manila/db/sqlalchemy/api.py", line 955, in quota_reserve
  session=session)
File "/opt/stack/manila/manila/db/sqlalchemy/api.py", line 751, in _reservation_create
  reservation_ref.save(session=session)
File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/models.py", line 50, in save
  session.flush()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2080, in flush
  self._flush(objects)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2198, in _flush
  transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2162, in _flush
  flush_context.execute()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
  rec.execute(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
  uow
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 178, in save_obj
  mapper, table, insert)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 804, in _emit_insert_statement
  execute(statement, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
  return meth(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
  return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
  compiled_sql, distilled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
  context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
  util.raise_from_cause(newraise, exc_info)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
  reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
  context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
  cursor.execute(statement, parameters)
File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 166, in execute
  result = self._query(query)
File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 322, in _query
  conn.query(q)
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 852, in query
  self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1053, in _read_query_result
  result.read()
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1336, in read
  first_packet = self.connection._read_packet()
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1010, in _read_packet
  packet.check_error()
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 393, in check_error
  err.raise_mysql_exception(self._data)
File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
  raise errorclass(errno, errval)
DBDataError: (pymysql.err.DataError) (1264, u"Out of range value for column 'delta' at row 1") [SQL: u'INSERT INTO reservations (created_at, updated_at, deleted_at, deleted, uuid, usage_id, project_id, user_id, resource, delta, expire) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(uuid)s, %(usage_id)s, %(project_id)s, %(user_id)s, %(resource)s, %(delta)s, %(expire)s)'] [parameters: {'deleted_at': None, 'resource': 'gigabytes', 'uuid': 'a0000d0a-d247-4754-9c18-f73dce3e3c18', 'deleted': 0, 'created_at': datetime.datetime(2017, 3, 2, 6, 31, 26, 476109), 'updated_at': None, 'expire': datetime.datetime(2017, 3, 3, 6, 31, 25, 850753), 'delta': 2247483646, 'user_id': u'2f8d19caf32b4a8981f1689b0a3e1cd4', 'project_id': u'c73b285fec9e47af951732804a3d0a3a', 'usage_id': 21}]
```

TommyLike (hu-husheng)
description: updated
Changed in manila:
assignee: nobody → TommyLike (hu-husheng)
status: New → In Progress
Changed in manila:
assignee: TommyLike (hu-husheng) → Tom Barron (tpb)
Changed in manila:
assignee: Tom Barron (tpb) → TommyLike (hu-husheng)
Changed in manila:
assignee: TommyLike (hu-husheng) → Valeriy Ponomaryov (vponomaryov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.openstack.org/440155
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=563e06c3c69888e4166b10c1d9f50fe0e820c02d
Submitter: Jenkins
Branch: master

commit 563e06c3c69888e4166b10c1d9f50fe0e820c02d
Author: TommyLike <email address hidden>
Date: Thu Mar 2 14:43:15 2017 +0800

    Add periodic task to clean up expired reservation

    Currently we would have uncommit reservations in database if
    the commit operation failed. This could obstruct creating new
    shares or others which would consume quota. This patch adds
    a perodic task to clean this expired reservations.

    Also fix bug in db.reservation_expire method(We dropped
    attribute 'usage' from Reservation, but it's still used
    in this method).

    Closes-Bug: #1669449

    Change-Id: I3dc0973ebac5eb33832e242c72059be1eb954369

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 5.0.0.0b2

This issue was fixed in the openstack/manila 5.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.