Cinder Volume lose MySQL connection during query under heavy load

[DMITRY KOMAROV]

If multiple volumes creation happens at the same time Cinder-Volume sometime loses the MySQL connection during the queries. The result is that the volume hangs in the creation process forever. We faced with this bug in Stein, Train, Ussuri releases, in Rocky release we didn't face this bug.
Tuning of mysql parameters and cinder db connection parameters doesn't affect this bug.

We used this script to reproduce the bug:
####
source /root/admin-openrc.sh

for _ID in $(seq 1 80)
do
        if [[ "$((${_ID}%10))" == "0" ]]
        then
           openstack volume create --image 7c6e074b-f7d6-4360-9091-ffaac877dab2 --size 40 VOL-${_ID} 2>/dev/null &
        else
           openstack volume create --size 40 VOL-${_ID} 2>/dev/null &
        fi
done
####

Error that we can see in cinder-volume.log when we face the bug:

2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server [req-7da71b7d-eb62-4196-8768-6f2c824ed047 7c60a827d0cd42e9a0124dc24d139da2 8a69fa7463a7468f85a5f022dccd29a5 - default default] Exception during message handling: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
(Background on this error at: http://sqlalche.me/e/e3q8)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "</var/lib/kolla/venv/lib/python2.7/site-packages/decorator.pyc:decorator-gen-673>", line 2, in create_volume
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 212, in wrapper
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 749, in create_volume
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server _run_flow()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 741, in _run_flow
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server flow_engine.run()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server failures[0].reraise()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/types/failure.py", line 346, in reraise
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1135, in execute
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server **volume_spec)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/utils.py", line 696, in _wrapper
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 206, in call
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return attempt.get(self._wrap_exception)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 247, in get
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2])
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/retrying.py", line 200, in call
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1039, in _create_from_image
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server image_meta=image_meta)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 412, in _handle_bootable_volume_glance_meta
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server volume_utils.enable_bootable_flag(volume)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/volume_utils.py", line 1091, in enable_bootable_flag
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server volume.save()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/objects/volume.py", line 399, in save
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server db.volume_update(self._context, self.id, updates)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/api.py", line 326, in volume_update
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return IMPL.volume_update(context, volume_id, values)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 241, in wrapper
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 2590, in volume_update
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server result = query.filter_by(id=volume_id).update(values)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3835, in update
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server update_op.exec_()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1694, in exec_
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server self._do_exec()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1887, in _do_exec
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server self._execute_stmt(update_stmt)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1699, in _execute_stmt
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server self.result = self.query._execute_crud(stmt, self.mapper)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3364, in _execute_crud
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server mapper=mapper, clause=stmt, close_with_result=True
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3349, in _connection_from_session
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server conn = self.session.connection(**kw)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1124, in connection
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server execution_options=execution_options,
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1130, in _connection_for_bind
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server engine, execution_options
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 431, in _connection_for_bind
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server conn = bind._contextual_connect()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2239, in _contextual_connect
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server self._wrap_pool_connect(self.pool.connect, None),
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2279, in _wrap_pool_connect
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server e, dialect, self
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1542, in _handle_dbapi_exception_noconnection
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server util.raise_from_cause(newraise, exc_info)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server reraise(type(exception), exception, tb=exc_tb, cause=cause)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2275, in _wrap_pool_connect
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return fn()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/pool/base.py", line 363, in connect
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return _ConnectionFairy._checkout(self)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server fairy = _ConnectionRecord.checkout(pool)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server rec = pool._do_get()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server self._dec_overflow()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server compat.reraise(exc_type, exc_value, exc_tb)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return self._create_connection()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return _ConnectionRecord(self)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server self.__connect(first_connect_check=True)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server connection = pool._invoke_creator(self)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return dialect.connect(*cargs, **cparams)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 453, in connect
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return self.dbapi.connect(*cargs, **cparams)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/__init__.py", line 94, in Connect
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server return Connection(*args, **kwargs)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 325, in __init__
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server self.connect()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 599, in connect
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server self._request_authentication()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 861, in _request_authentication
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server auth_packet = self._read_packet()
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 657, in _read_packet
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server packet_header = self._read_bytes(4)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 707, in _read_bytes
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server (Background on this error at: http://sqlalche.me/e/e3q8)
2020-06-30 19:08:06.697 39 ERROR oslo_messaging.rpc.server

[Gayathri Devi Kathiri (gayathridevikathiri)]

@Dmitry Komarov: As per me, issue I was facing is same as that mentioned in the ticket. Please check once

I have observed similar issue while running openstack tempest suites where multiple volumes creation happens at the same time. Then, sometimes Cinder-Volume loses the MySQL connection during the queries. The result is that the volume hangs in the creation process forever.

root@gayathri:/home/# openstack volume list --all-projects
+--------------------------------------+---------------------------------------------------------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+---------------------------------------------------------+-----------+------+-------------+
| a7f7b561-ab60-4c9d-b19b-ef08d174e529 | tempest-VolumesBackupsAdminTest-Volume-1899881623 | creating | 5 | |
+--------------------------------------+---------------------------------------------------------+-----------+------+-------------+

Found below error in the cinder-volume log:

2020-08-07 12:23:25.726 11 WARNING cinder.volume.manager [req-4e1c1dc7-2527-4b9f-9123-2b36b1c5eb21 4f31cff2a6294d0981269cc58eb34d3e b8c2c93ad3eb404cbf83e2ac0b200cea - f57d52f23c0748129c77ae8bb84778b7 f57d52f23c0748129c77ae8bb84778b7] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (1188197b-e955-4ef0-9a0c-927dc0cacf74) transitioned into state 'FAILURE' from state 'RUNNING'
1 predecessors (most recent first):
  Flow 'volume_create_manager': oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager Traceback (most recent call last):
2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect
2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return fn()
2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 425, in connect
2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager return _ConnectionFairy._checkout(self)
2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 822, in _checkout
2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager fairy = _ConnectionRecord.checkout(pool)
2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 554, in checkout
2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager rec = pool._do_get()
2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1250, in _do_get
2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager self._dec_overflow()
2020-08-07 12:23:25.726 11 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.6/s...

[Gayathri Devi Kathiri (gayathridevikathiri)]

Reference for cinder volume creation time:
root@gayathri:/home/# openstack volume show a7f7b561-ab60-4c9d-b19b-ef08d174e529
+--------------------------------+---------------------------------------------------------+
| Field | Value |
+--------------------------------+---------------------------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-08-07T12:23:13.000000 |
| description | None |
| encrypted | False |
| id | a7f7b561-ab60-4c9d-b19b-ef08d174e529 |
| migration_status | None |
| multiattach | False |
| name | tempest-e4e980ee-VolumesBackupsAdminTest-Volume-1899881623 |
| os-vol-host-attr:host | cinder-volume-worker@rbd1#rbd1 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | b8c2c93ad3eb404cbf83e2ac0b200cea |
| properties | |
| replication_status | None |
| size | 5 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | rbd1 |
| updated_at | 2020-08-07T12:23:14.000000 |
| user_id | 4f31cff2a6294d0981269cc58eb34d3e |
+--------------------------------+---------------------------------------------------------+