@Dmitry Komarov: As per me, issue I was facing is same as that mentioned in the ticket. Please check once I have observed similar issue while running openstack tempest suites where multiple volumes creation happens at the same time. Then, sometimes Cinder-Volume loses the MySQL connection during the queries. The result is that the volume hangs in the creation process forever. root@gayathri:/home/# openstack volume list --all-projects +--------------------------------------+---------------------------------------------------------+-----------+------+-------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+---------------------------------------------------------+-----------+------+-------------+ | a7f7b561-ab60-4c9d-b19b-ef08d174e529 | tempest-VolumesBackupsAdminTest-Volume-1899881623 | creating | 5 | | +--------------------------------------+---------------------------------------------------------+-----------+------+-------------+ Found below error in the cinder-volume log: 2020-08-07 12:23:25.726 11 WARNING cinder.volume.manager [req-4e1c1dc7-2527-4b9f-9123-2b36b1c5eb21 4f31cff2a6294d0981269cc58eb34d3e b8c2c93ad3eb404cbf83e2ac0b200cea - f57d52f23c0748129c77ae8bb84778b7 f57d52f23c0748129c77ae8bb84778b7] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1188197b-e955-4ef0-9a0c-927dc0cacf74) transitioned into state 'FAILURE' from state 'RUNNING' 1 predecessors (most recent first): Flow 'volume_create_manager': oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager Traceback (most recent call last): 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return fn() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 425, in connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return _ConnectionFairy._checkout(self) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 822, in _checkout 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager fairy = _ConnectionRecord.checkout(pool) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 554, in checkout 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager rec = pool._do_get() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1250, in _do_get 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager self._dec_overflow() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__ 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager compat.reraise(exc_type, exc_value, exc_tb) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager raise value 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1247, in _do_get 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return self._create_connection() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 370, in _create_connection 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return _ConnectionRecord(self) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 499, in __init__ 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager self.__connect(first_connect_check=True) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 701, in __connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager connection = pool._invoke_creator(self) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return dialect.connect(*cargs, **cparams) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return self.dbapi.connect(*cargs, **cparams) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return Connection(*args, **kwargs) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/pymysql/connections.py", line 325, in __init__ 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager self.connect() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/pymysql/connections.py", line 599, in connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager self._request_authentication() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/pymysql/connections.py", line 861, in _request_authentication 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager auth_packet = self._read_packet() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager packet_header = self._read_bytes(4) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query') 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager The above exception was the direct cause of the following exception: 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager Traceback (most recent call last): 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager result = task.execute(**arguments) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 229, in execute 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager volume.refresh() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/cinder/objects/cleanable.py", line 238, in refresh 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager super(CinderCleanableObject, self).refresh() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/cinder/objects/base.py", line 487, in refresh 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager current = self.get_by_id(self._context, self.id) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/cinder/objects/base.py", line 347, in get_by_id 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/cinder/db/api.py", line 1794, in get_by_id 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return IMPL.get_by_id(context, model, id, *args, **kwargs) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return f(*args, **kwargs) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 7067, in get_by_id 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return _GET_METHODS[model](context, id, *args, **kwargs) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return f(*args, **kwargs) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 2081, in volume_get 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return _volume_get(context, volume_id) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return f(*args, **kwargs) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 1841, in _volume_get 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager result = result.filter_by(id=volume_id).first() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2979, in first 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager ret = list(self[0:1]) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2771, in __getitem__ 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return list(res) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__ 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return self._execute_and_instances(context) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3103, in _execute_and_instances 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager querycontext, self._connection_from_session, close_with_result=True 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3111, in _get_bind_args 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager mapper=self._bind_mapper(), clause=querycontext.statement, **kw 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3096, in _connection_from_session 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager conn = self.session.connection(**kw) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1120, in connection 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager execution_options=execution_options, 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1129, in _connection_for_bind 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager conn = engine.contextual_connect(**kw) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2192, in contextual_connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager self._wrap_pool_connect(self.pool.connect, None), 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2232, in _wrap_pool_connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager e, dialect, self 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1526, in _handle_dbapi_exception_noconnection 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager util.raise_from_cause(newraise, exc_info) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager reraise(type(exception), exception, tb=exc_tb, cause=cause) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 276, in reraise 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager raise value.with_traceback(tb) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return fn() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 425, in connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return _ConnectionFairy._checkout(self) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 822, in _checkout 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager fairy = _ConnectionRecord.checkout(pool) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 554, in checkout 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager rec = pool._do_get() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1250, in _do_get 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager self._dec_overflow() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__ 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager compat.reraise(exc_type, exc_value, exc_tb) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager raise value 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1247, in _do_get 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return self._create_connection() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 370, in _create_connection 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return _ConnectionRecord(self) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 499, in __init__ 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager self.__connect(first_connect_check=True) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 701, in __connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager connection = pool._invoke_creator(self) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return dialect.connect(*cargs, **cparams) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return self.dbapi.connect(*cargs, **cparams) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return Connection(*args, **kwargs) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/pymysql/connections.py", line 325, in __init__ 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager self.connect() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/pymysql/connections.py", line 599, in connect 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager self._request_authentication() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/pymysql/connections.py", line 861, in _request_authentication 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager auth_packet = self._read_packet() 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager packet_header = self._read_bytes(4) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) 2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager 2020-08-07 12:23:25.732 11 WARNING cinder.volume.manager [req-4e1c1dc7-2527-4b9f-9123-2b36b1c5eb21 4f31cff2a6294d0981269cc58eb34d3e b8c2c93ad3eb404cbf83e2ac0b200cea - f57d52f23c0748129c77ae8bb84778b7 f57d52f23c0748129c77ae8bb84778b7] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1188197b-e955-4ef0-9a0c-927dc0cacf74) transitioned into state 'REVERTED' from state 'REVERTING' 2020-08-07 12:23:25.734 11 WARNING cinder.volume.manager [req-4e1c1dc7-2527-4b9f-9123-2b36b1c5eb21 4f31cff2a6294d0981269cc58eb34d3e b8c2c93ad3eb404cbf83e2ac0b200cea - f57d52f23c0748129c77ae8bb84778b7 f57d52f23c0748129c77ae8bb84778b7] Flow 'volume_create_manager' (79177cca-2779-42fa-9598-f48861a431f9) transitioned into state 'REVERTED' from state 'RUNNING'