Cinder

Blank volume's attach_status causes the volume list to fail

Bug #1954613 reported by ZhaoYixin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Incomplete
Low
Unassigned

Bug Description

openstack volume list --all

The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-35350006-e121-4d69-8d84-3ddceb198e5d)

Logs:

2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault [None req-0602542e-6e37-4dea-8e34-50ca18cab42c 484e61647b5e4dff8719956bc86b74bc 5749de0c98f94688aaf9700730cf237b - default default] Caught error: <class 'ValueError'> Field value is invalid: ValueError: Field value is invalid
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault Traceback (most recent call last):
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/cinder/api/middleware/fault.py", line 85, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return req.get_response(self.application)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/request.py", line 1313, in send
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault status, headers, app_iter = self.call_application(
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/request.py", line 1278, in call_application
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/dec.py", line 143, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return resp(environ, start_response)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/dec.py", line 129, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault resp = self.call_func(req, *args, **kw)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/dec.py", line 193, in call_func
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return self.func(req, *args, **kwargs)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/osprofiler/web.py", line 112, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return request.get_response(self.application)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/request.py", line 1313, in send
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault status, headers, app_iter = self.call_application(
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/request.py", line 1278, in call_application
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/dec.py", line 129, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault resp = self.call_func(req, *args, **kw)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/dec.py", line 193, in call_func
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return self.func(req, *args, **kwargs)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault response = req.get_response(self._app)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/request.py", line 1313, in send
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault status, headers, app_iter = self.call_application(
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/request.py", line 1278, in call_application
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/dec.py", line 129, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault resp = self.call_func(req, *args, **kw)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/dec.py", line 193, in call_func
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return self.func(req, *args, **kwargs)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/keystonemiddleware/audit/__init__.py", line 159, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault response = req.get_response(self._application)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/request.py", line 1313, in send
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault status, headers, app_iter = self.call_application(
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/request.py", line 1278, in call_application
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/dec.py", line 143, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return resp(environ, start_response)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/dec.py", line 143, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return resp(environ, start_response)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/routes/middleware.py", line 141, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault response = self.app(environ, start_response)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/dec.py", line 143, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return resp(environ, start_response)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/dec.py", line 129, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault resp = self.call_func(req, *args, **kw)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/webob/dec.py", line 193, in call_func
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return self.func(req, *args, **kwargs)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/cinder/api/openstack/wsgi.py", line 838, in __call__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return self._process_stack(request, action, action_args,
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/cinder/api/openstack/wsgi.py", line 899, in _process_stack
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault action_result = self.dispatch(meth, request, action_args)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/cinder/api/openstack/wsgi.py", line 994, in dispatch
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return method(req=request, **action_args)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/cinder/api/v2/volumes.py", line 91, in detail
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return self._get_volumes(req, is_detail=True)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/cinder/api/v2/volumes.py", line 119, in _get_volumes
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault volumes = self.volume_api.get_all(context, marker, limit,
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/cinder/volume/api.py", line 634, in get_all
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault volumes = objects.VolumeList.get_all(context, marker, limit,
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/cinder/objects/volume.py", line 618, in get_all
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return base.obj_make_list(context, cls(context), objects.Volume,
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 1112, in obj_make_list
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault item = item_cls._from_db_object(context, item_cls(), db_item,
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/cinder/objects/volume.py", line 285, in _from_db_object
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault attachments = base.obj_make_list(
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 1112, in obj_make_list
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault item = item_cls._from_db_object(context, item_cls(), db_item,
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/cinder/objects/volume_attachment.py", line 97, in _from_db_object
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault attachment[name] = value
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 758, in __setitem__
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault setattr(self, name, value)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 72, in setter
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault field_value = field.coerce(self, name, value)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/oslo_versionedobjects/fields.py", line 204, in coerce
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault return self._type.coerce(obj, attr, value)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault File "/var/lib/openstack/lib/python3.8/site-packages/oslo_versionedobjects/fields.py", line 325, in coerce
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault raise ValueError(msg)
2021-12-09 08:21:52.985 13 ERROR cinder.api.middleware.fault ValueError: Field value is invalid

By querying the database, it is found that the attach_status of a volume(8e8e18ed-90ae-417e-99cf-83a3bd67f754) is blank.

MariaDB [cinder]> select id, volume_id, attach_time, detach_time, attach_mode, attach_status from volume_attachment where attach_status!='detached' ;
+--------------------------------------+--------------------------------------+---------------------+-------------+-------------+---------------+
| id | volume_id | attach_time | detach_time | attach_mode | attach_status |
+--------------------------------------+--------------------------------------+---------------------+-------------+-------------+---------------+
| 27e87311-7a11-4e40-b661-d821ca0d7c03 | da587315-d5dd-4576-be72-22e8a6e86c7f | 2021-12-08 00:52:20 | NULL | rw | attached |
| 341d5b1d-63b5-4689-bbad-6c8154c420e3 | 8e8e18ed-90ae-417e-99cf-83a3bd67f754 | 2021-12-09 08:18:47 | NULL | rw | |
| 6bcbe5fa-67b5-48db-a757-170eecf8f0c8 | 569b466b-0ca7-4ca6-9ca3-6147d4e107ab | 2021-12-09 01:37:35 | NULL | rw | attached |
| 859a9da2-599e-4e56-bb0f-9f50f1be9c12 | 39e7ebb5-ba47-4fd3-97f6-3dd3debe72cf | 2021-12-02 06:37:25 | NULL | null | attached |
+--------------------------------------+--------------------------------------+---------------------+-------------+-------------+---------------+

Revision history for this message
Eric Harney (eharney) wrote :

Do you know how to reproduce this situation?

Revision history for this message
ZhaoYixin (zhaoyixin) wrote :

I do not know.

Whether to add exception handling in the volume list?

Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

Greetings,
Please provide the cinder version you're using and steps to reproduce this.
Thanks in advance

Changed in cinder:
status: New → Incomplete
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.