glance_store

Image upload is failing with NoFibreChannelVolumeDeviceFound after configuring Cinder(HP3Par FC storage) as glance backend

Bug #1863983 reported by Abhishek Kekane
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
glance_store
Fix Released
Medium
Abhishek Kekane

Bug Description

Image upload is failing with NoFibreChannelVolumeDeviceFound error after configuring cinder backend (HP3Par FC storage) for the glance service.

The issue here we are facing is that usually cinder calls os_brick with multipath info but in this case glance is doing the call (without passing any multipath info). Cinder driver of glance_store passes 'hard-coded' False as multipath is not used to os_brick library.

In order to fix this we need to add new config option for cinder driver of glance_store 'cinder_use_multipath' True|False to avoid such issues.

Steps to reproduce:

1. Configure cinder back end(HP3Par FC storage) for the Image service
2. Create image
$ glance image-create --name "cirros2" --file cirros-0.3.0-i386-disk.img --disk-format qcow2 --container-format bare

Output:
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

glance-api.log:-
 ~~~
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi Traceback (most recent call last):
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1227, in __call__
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi request, **action_args)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1270, in dispatch
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return method(*args, **kwargs)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/utils.py", line 414, in wrapped
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 278, in upload
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self._restore(image_repo, image)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.force_reraise()
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 143, in upload
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi image.set_data(data, size)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.base.set_data(data, size)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 480, in set_data
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.force_reraise()
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 427, in set_data
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.repo.set_data(data, size)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 202, in set_data
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi self.image.set_data(data, size=size)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/location.py", line 439, in set_data
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi verifier=verifier)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 453, in add_to_backend
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi verifier)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 426, in store_add_to_backend
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi verifier=verifier)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/_drivers/cinder.py", line 677, in add
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi with self._open_cinder_volume(client, volume, 'wb') as f:
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return self.gen.next()
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance_store/_drivers/cinder.py", line 497, in _open_cinder_volume
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi device = conn.connect_volume(connection_info['data'])
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi result = f(*args, **kwargs)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return f(*args, **kwargs)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 245, in connect_volume
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi timer.start(interval=2).wait()
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return hubs.get_hub().switch()
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi return self.greenlet.switch()
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 137, in _run_loop
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi result = func(*self.args, **self.kw)
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 231, in _wait_for_device_discovery
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi raise exception.NoFibreChannelVolumeDeviceFound()
 2019-12-24 07:10:30.869 28 ERROR glance.common.wsgi NoFibreChannelVolumeDeviceFound: Unable to find a Fibre Channel volume
 ~~~

See original description
Abhishek Kekane (abhishek-kekane)
Changed in glance-store:
importance: Undecided → Medium
assignee: nobody → Abhishek Kekane (abhishek-kekane)
Abhishek Kekane (abhishek-kekane)
description: updated
Revision history for this message
Abhishek Kekane (abhishek-kekane) wrote :

In addition to the hard-code use of multipath, cinder driver is also hard-coded enforce_multipath to False. So we should add two new config options here,

1. cinder_use_multipath
2. cinder_enforce_multipath

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (master)

Fix proposed to branch: master
Review: https://review.opendev.org/708811

Changed in glance-store:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (master)

Reviewed: https://review.opendev.org/708811
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=ae73287c7b4db8d5894f72d8a5b1b89b42697545
Submitter: Zuul
Branch: master

commit ae73287c7b4db8d5894f72d8a5b1b89b42697545
Author: Abhishek Kekane <email address hidden>
Date: Thu Feb 20 07:21:14 2020 +0000

    Image upload fails if cinder multipath is enabled

    Image upload is failing with NoFibreChannelVolumeDeviceFound error after
    configuring cinder backend (HP3Par FC storage) for the glance service. The
    issue here is that usually cinder calls os_brick with multipath information
    but in this case cinder driver of glance is doing the call (without passing
    any multipath information). with hard-coded' False to let os_brick library
    know that multipath is not configured.

    In order to fix this, added two new boolean config option for cinder
    driver default to False;

    'cinder_use_multipath'
    'cinder_enforce_multipath'

    Change-Id: I07064f3cb1a33ac4ac7e4b572f8e1b5c688bd9a3
    Closes-Bug: #1863983

Changed in glance-store:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/709048

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/709049

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/709050

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance_store (stable/train)

Change abandoned by Abhishek Kekane (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/709048

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance_store (stable/stein)

Change abandoned by Abhishek Kekane (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/709049

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance_store (stable/rocky)

Change abandoned by Abhishek Kekane (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/709050

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 2.0.0

This issue was fixed in the openstack/glance_store 2.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (stable/train)

Reviewed: https://review.opendev.org/709048
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=d052ce57b5cfc66a4b3b8030a14db3d46c312990
Submitter: Zuul
Branch: stable/train

commit d052ce57b5cfc66a4b3b8030a14db3d46c312990
Author: Abhishek Kekane <email address hidden>
Date: Thu Feb 20 07:21:14 2020 +0000

    Image upload fails if cinder multipath is enabled

    Image upload is failing with NoFibreChannelVolumeDeviceFound error after
    configuring cinder backend (HP3Par FC storage) for the glance service. The
    issue here is that usually cinder calls os_brick with multipath information
    but in this case cinder driver of glance is doing the call (without passing
    any multipath information). with hard-coded' False to let os_brick library
    know that multipath is not configured.

    In order to fix this, added two new boolean config option for cinder
    driver default to False;

    'cinder_use_multipath'
    'cinder_enforce_multipath'

    Change-Id: I07064f3cb1a33ac4ac7e4b572f8e1b5c688bd9a3
    Closes-Bug: #1863983
    (cherry picked from commit ae73287c7b4db8d5894f72d8a5b1b89b42697545)

tags: added: in-stable-train
Revision history for this message
Rajat Dhasmana (whoami-rajat) wrote :

Related fix: https://review.opendev.org/c/openstack/glance_store/+/786271

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to glance_store (master)

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/786271
Committed: https://opendev.org/openstack/glance_store/commit/04e5ead7c000211a4c10104ed2bb65c9df7681ae
Submitter: "Zuul (22348)"
Branch: master

commit 04e5ead7c000211a4c10104ed2bb65c9df7681ae
Author: whoami-rajat <email address hidden>
Date: Wed Apr 14 10:40:45 2021 -0400

    Pass multipath config while creating connector object

    Currently we only pass the configured multipath option when getting
    the connector properties and do not pass it when creating the
    connector object which in turn does the multipathing in os-brick.
    So we are only passing multipathing option to cinder for exporting &
    mapping the volume to paths but not while creating the connector object
    and hence doing multipathing partially.
    This patch passes that option and fixes the stated issue.

    Related-Bug: #1863983

    Co-Authored-By: Gorka Eguileor <email address hidden>

    Change-Id: I82979c47ec9ec1dd6cc864d3827ab219177251f8

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to glance_store (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/glance_store/+/786420

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to glance_store (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/786420
Committed: https://opendev.org/openstack/glance_store/commit/9bb857097e8847bf8c9c722f7ff5990dccb391cf
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 9bb857097e8847bf8c9c722f7ff5990dccb391cf
Author: whoami-rajat <email address hidden>
Date: Wed Apr 14 10:40:45 2021 -0400

    Pass multipath config while creating connector object

    Currently we only pass the configured multipath option when getting
    the connector properties and do not pass it when creating the
    connector object which in turn does the multipathing in os-brick.
    So we are only passing multipathing option to cinder for exporting &
    mapping the volume to paths but not while creating the connector object
    and hence doing multipathing partially.
    This patch passes that option and fixes the stated issue.

    Related-Bug: #1863983

    Co-Authored-By: Gorka Eguileor <email address hidden>

    Change-Id: I82979c47ec9ec1dd6cc864d3827ab219177251f8
    (cherry picked from commit 04e5ead7c000211a4c10104ed2bb65c9df7681ae)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to glance_store (stable/victoria)

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/glance_store/+/786789

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance_store (stable/victoria)

Change abandoned by "Rajat Dhasmana <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/glance_store/+/786789

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to glance_store (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/786789
Committed: https://opendev.org/openstack/glance_store/commit/e43a56fb9b772703214e39f3371cc2ff00e06d4d
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit e43a56fb9b772703214e39f3371cc2ff00e06d4d
Author: whoami-rajat <email address hidden>
Date: Wed Apr 14 10:40:45 2021 -0400

    Pass multipath config while creating connector object

    Currently we only pass the configured multipath option when getting
    the connector properties and do not pass it when creating the
    connector object which in turn does the multipathing in os-brick.
    So we are only passing multipathing option to cinder for exporting &
    mapping the volume to paths but not while creating the connector object
    and hence doing multipathing partially.
    This patch passes that option and fixes the stated issue.

    Related-Bug: #1863983

    Co-Authored-By: Gorka Eguileor <email address hidden>

    Change-Id: I82979c47ec9ec1dd6cc864d3827ab219177251f8
    (cherry picked from commit 04e5ead7c000211a4c10104ed2bb65c9df7681ae)
    (cherry picked from commit 9bb857097e8847bf8c9c722f7ff5990dccb391cf)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to glance_store (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/glance_store/+/822841

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance_store (stable/ussuri)

Change abandoned by "Abhishek Kekane <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/glance_store/+/822841
Reason: This does not fit with our backport criteria [0]

[0] https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store train-eol

This issue was fixed in the openstack/glance_store train-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.