solidfire driver crashes during ensure export if solidfire not available

[Vish Ishaya]

If the solidfire device isn't available for some reason and volumes have been created, the solidfire driver will crash during startup.

2013-08-20 03:08:06 DEBUG cinder.volume.drivers.solidfire Executing SolidFire ensure_export...
2013-08-20 03:08:06 DEBUG cinder.volume.drivers.solidfire Payload for SolidFire API call: {"params":
{"username": "690534e8073a40d1b027c35808feb6da"}
, "method": "GetAccountByName", "id": 190619619316511908876065095856189853893}
2013-08-20 03:08:06 ERROR cinder.volume.drivers.solidfire Failed to make httplib connection SolidFire Cluster: (verify san_ip settings)
2013-08-20 03:08:06 ERROR cinder.service Unhandled exception
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 227, in _start_child
self._child_process(wrap.server)
File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 204, in _child_process
launcher.run_server(server)
File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 95, in run_server
server.start()
File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 355, in start
self.manager.init_host()
File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 149, in init_host
self.driver.ensure_export(ctxt, volume)
File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/solidfire.py", line 545, in ensure_export
return self._do_export(volume)
File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/solidfire.py", line 267, in _do_export
sfaccount = self._get_sfaccount(volume'project_id')
File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/solidfire.py", line 225, in _get_sfaccount
sfaccount = self._get_sfaccount_by_name(sf_account_name)
File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/solidfire.py", line 211, in _get_sfaccount_by_name
data = self._issue_api_request('GetAccountByName', params)
File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/solidfire.py", line 146, in _issue_api_request
raise exception.SolidFireAPIException(msg)
SolidFireAPIException: Failed to make httplib connection:
2013-08-20 03:08:06 INFO cinder.service Child 13752 exited with status 2
2013-08-20 03:08:06 INFO cinder.service _wait_child 1
2013-08-20 03:08:06 INFO cinder.service wait wrap.failed True

Something better than this should occur, for example retrying the export indefinitely or simply ignoring the error.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/43181

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/43181
Committed: http://github.com/openstack/cinder/commit/e13b7d8a46cd54e61bdc20500ad597c2545ffaed
Submitter: Jenkins
Branch: master

commit e13b7d8a46cd54e61bdc20500ad597c2545ffaed
Author: John Griffith <email address hidden>
Date: Wed Aug 21 13:05:30 2013 -0600

    Dont crash service if sf cluster isnt available

    Currently if the SolidFire driver is configured but the cluster
    isn't available for some reason the ensure_export call will raise
    an unhandled exception and crash the volume-service.

    We should be able to handle things like loosing connectivity to
    a single back-end without impacting the other volume-service
    backends.

    We'll wrap the ensure_export call in a try block here and
    return None in the case that the connection can't be made.
    This will keep the service from crashing and log an error
    message that the connection timed out.

    Additional work would include adding a periodic retry task
    to the manager to try and start the backend service for us
    on some regular interval incase the device comes back.

    Fixes: bug 1215064

    Change-Id: Ice3f517d220c40113074bb77adbb10d5e32abd0b

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/43526

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/43526
Committed: http://github.com/openstack/cinder/commit/3329158d784d5966fa2fdccd1c097295ebe37a67
Submitter: Jenkins
Branch: stable/grizzly

commit 3329158d784d5966fa2fdccd1c097295ebe37a67
Author: John Griffith <email address hidden>
Date: Wed Aug 21 13:05:30 2013 -0600

    Dont crash service if sf cluster isnt available

    Currently if the SolidFire driver is configured but the cluster
    isn't available for some reason the ensure_export call will raise
    an unhandled exception and crash the volume-service.

    We should be able to handle things like loosing connectivity to
    a single back-end without impacting the other volume-service
    backends.

    We'll wrap the ensure_export call in a try block here and
    return None in the case that the connection can't be made.
    This will keep the service from crashing and log an error
    message that the connection timed out.

    Additional work would include adding a periodic retry task
    to the manager to try and start the backend service for us
    on some regular interval incase the device comes back.

    Fixes: bug 1215064

    Change-Id: Ice3f517d220c40113074bb77adbb10d5e32abd0b
    (cherry picked from commit e13b7d8a46cd54e61bdc20500ad597c2545ffaed)