Image upload is failing with NoFibreChannelVolumeDeviceFound error after configuring cinder backend (HP3Par FC storage) for the glance service.
The issue here we are facing is that usually cinder calls os_brick with multipath info but in this case glance is doing the call (without passing any multipath info). Cinder driver of glance_store passes 'hard-coded' False as multipath is not used to os_brick library.
In order to fix this we need to add new config option for cinder driver of glance_store 'multipath_enabled' True|False to avoid such issues.
Steps to reproduce:
1. Configure cinder back end(HP3Par FC storage) for the Image service
2. Create image
$ glance image-create --name "cirros2" --file cirros-0.3.0-i386-disk.img --disk-format qcow2 --container-format bare
Output:
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
glance-api.log:-
~~~
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi Traceback (most recent call last):
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1227, in __call__
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi request, **action_args)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1270, in dispatch
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return method(*args, **kwargs)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/utils.py", line 414, in wrapped
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 278, in upload
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self._restore(image_repo, image)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.force_reraise()
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 143, in upload
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi image.set_data(data, size)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.base.set_data(data, size)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 480, in set_data
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.force_reraise()
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 427, in set_data
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.repo.set_data(data, size)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 202, in set_data
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.image.set_data(data, size=size)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/location.py", line 439, in set_data
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi verifier=verifier)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 453, in add_to_backend
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi verifier)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 426, in store_add_to_backend
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi verifier=verifier)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/_drivers/cinder.py", line 677, in add
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f:
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return self.gen.next()
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/_drivers/cinder.py", line 497, in _open_cinder_volume
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi device = conn.connect_volume(connection_info['data'])
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi result = f(*args, **kwargs)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return f(*args, **kwargs)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 245, in connect_volume
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi timer.start(interval=2).wait()
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return hubs.get_hub().switch()
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return self.greenlet.switch()
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 137, in _run_loop
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi result = func(*self.args, **self.kw)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 231, in _wait_for_device_discovery
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi raise exception.NoFibreChannelVolumeDeviceFound()
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi NoFibreChannelVolumeDeviceFound: Unable to find a Fibre Channel volume
~~~
Image upload is failing with NoFibreChannelV olumeDeviceFoun d error after configuring cinder backend (HP3Par FC storage) for the glance service.
The issue here we are facing is that usually cinder calls os_brick with multipath info but in this case glance is doing the call (without passing any multipath info). Cinder driver of glance_store passes 'hard-coded' False as multipath is not used to os_brick library.
In order to fix this we need to add new config option for cinder driver of glance_store 'multipath_enabled' True|False to avoid such issues.
Steps to reproduce:
1. Configure cinder back end(HP3Par FC storage) for the Image service 0.3.0-i386- disk.img --disk-format qcow2 --container-format bare
2. Create image
$ glance image-create --name "cirros2" --file cirros-
Output:
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
glance-api.log:- python2. 7/site- packages/ glance/ common/ wsgi.py" , line 1227, in __call__ python2. 7/site- packages/ glance/ common/ wsgi.py" , line 1270, in dispatch python2. 7/site- packages/ glance/ common/ utils.py" , line 414, in wrapped python2. 7/site- packages/ glance/ api/v2/ image_data. py", line 278, in upload image_repo, image) python2. 7/site- packages/ oslo_utils/ excutils. py", line 220, in __exit__ reraise( ) python2. 7/site- packages/ oslo_utils/ excutils. py", line 196, in force_reraise self.type_ , self.value, self.tb) python2. 7/site- packages/ glance/ api/v2/ image_data. py", line 143, in upload data(data, size) python2. 7/site- packages/ glance/ domain/ proxy.py" , line 195, in set_data set_data( data, size) python2. 7/site- packages/ glance/ notifier. py", line 480, in set_data ion(notify_ error, 'image.upload', msg) python2. 7/site- packages/ oslo_utils/ excutils. py", line 220, in __exit__ reraise( ) python2. 7/site- packages/ oslo_utils/ excutils. py", line 196, in force_reraise self.type_ , self.value, self.tb) python2. 7/site- packages/ glance/ notifier. py", line 427, in set_data set_data( data, size) python2. 7/site- packages/ glance/ api/policy. py", line 202, in set_data set_data( *args, **kwargs) python2. 7/site- packages/ glance/ quota/_ _init__ .py", line 304, in set_data set_data( data, size=size) python2. 7/site- packages/ glance/ location. py", line 439, in set_data python2. 7/site- packages/ glance_ store/backend. py", line 453, in add_to_backend python2. 7/site- packages/ glance_ store/backend. py", line 426, in store_add_ to_backend python2. 7/site- packages/ glance_ store/capabilit ies.py" , line 225, in op_checker python2. 7/site- packages/ glance_ store/_ drivers/ cinder. py", line 677, in add cinder_ volume( client, volume, 'wb') as f: python2. 7/contextlib. py", line 17, in __enter__ python2. 7/site- packages/ glance_ store/_ drivers/ cinder. py", line 497, in _open_cinder_volume volume( connection_ info['data' ]) python2. 7/site- packages/ os_brick/ utils.py" , line 150, in trace_logging_ wrapper python2. 7/site- packages/ oslo_concurrenc y/lockutils. py", line 274, in inner python2. 7/site- packages/ os_brick/ initiator/ connectors/ fibre_channel. py", line 245, in connect_volume interval= 2).wait( ) python2. 7/site- packages/ eventlet/ event.py" , line 121, in wait hub().switch( ) python2. 7/site- packages/ eventlet/ hubs/hub. py", line 294, in switch switch( ) python2. 7/site- packages/ oslo_service/ loopingcall. py", line 137, in _run_loop python2. 7/site- packages/ os_brick/ initiator/ connectors/ fibre_channel. py", line 231, in _wait_for_ device_ discovery NoFibreChannelV olumeDeviceFoun d() olumeDeviceFoun d: Unable to find a Fibre Channel volume
~~~
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi Traceback (most recent call last):
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi request, **action_args)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return method(*args, **kwargs)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self._restore(
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.force_
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi six.reraise(
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi image.set_
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.base.
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi _send_notificat
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.force_
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi six.reraise(
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.repo.
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return self.image.
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.image.
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi verifier=verifier)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi verifier)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi verifier=verifier)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi with self._open_
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib64/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return self.gen.next()
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi device = conn.connect_
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi result = f(*args, **kwargs)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return f(*args, **kwargs)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi timer.start(
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return hubs.get_
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return self.greenlet.
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi result = func(*self.args, **self.kw)
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi raise exception.
2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi NoFibreChannelV
~~~