3par driver failure in get_volume_stats

[Gerald McBrearty]

With the Queens release 3PAR driver I'm seeing this failure:

2018-08-30 14:53:05.118 6940 DEBUG cinder.volume.drivers.hpe.hpe_3par_fc [req-49ccfbd0-6ca9-46ea-8381-22bbcb9c60b8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 9defcbc367ab4d51954432de54be4add - default default] <== get_volume_stats: exception (60120ms) KeyError('array_id',) trace_logging_wrapper /usr/lib/python2.7/site-packages/cinder/utils.py:908
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task [req-49ccfbd0-6ca9-46ea-8381-22bbcb9c60b8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 9defcbc367ab4d51954432de54be4add - default default] Error during PowerVCVolumeManager.publish_service_capabilities: KeyError: 'array_id'
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task Traceback (most recent call last):
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task task(self, context)
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2567, in publish_service_capabilities
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task self._report_driver_status(context)
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2464, in _report_driver_status
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task volume_stats = self.driver.get_volume_stats(refresh=True)
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 902, in trace_logging_wrapper
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task result = f(*args, **kwargs)
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hpe/hpe_3par_base.py", line 119, in get_volume_stats
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task common = self._login()
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hpe/hpe_3par_base.py", line 84, in _login
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task array_id=array_id)
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hpe/hpe_3par_common.py", line 498, in do_setup
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task self.client.id = stats['array_id']
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task KeyError: 'array_id'
2018-08-30 14:53:05.118 6940 ERROR oslo_service.periodic_task
2018-08-30 14:53:05.120 6940 DEBUG oslo_service.periodic_task [req-49ccfbd0-6ca9-46ea-8381-22bbcb9c60b8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 9defcbc367ab4d51954432de54be4add - default default] Running periodic task PowerVCVolumeManager._refresh_driver_stats_if_needed run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215
2018-08-30 14:53:05.121 6940 DEBUG cinder.volume.drivers.hpe.hpe_3par_fc [req-49ccfbd0-6ca9-46ea-8381-22bbcb9c60b8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 9defcbc367ab4d51954432de54be4add - default default] ==> get_volume_stats: call {'self': <cinder.volume.drivers.hpe.hpe_3par_fc.HPE3PARFCDriver object at 0x7f991373b590>, 'refresh': True} trace_logging_wrapper /usr/lib/python2.7/site-packages/cinder/utils.py:898

[Gerald McBrearty (gfm-r)]

Digging a little deeper the last get_volume_stats call before this failure returned:

2018-08-30 14:33:54.636 6940 DEBUG cinder.volume.drivers.hpe.hpe_3par_fc [req-49ccfbd0-6ca9-46ea-8381-22bbcb9c60b8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 9defcbc367ab4d51954432de54be4add - default default] <== get_volume_stats: return (120938ms) {'driver_version': u'4.0.5', 'volume_backend_name': u'generic0', 'storage_protocol': u'FC'} trace_logging_wrapper /usr/lib/python2.7/site-packages/cinder/utils.py:922

and the one right before that looks good
2018-08-30 14:31:53.073 6940 DEBUG cinder.volume.drivers.hpe.hpe_3par_fc [req-164bc93b-4f74-438e-8b4f-a8ab9922e8aa - - - - -] <== get_volume_stats: return (214665ms) {'array_id': 12965, 'vendor_name': u'Hewlett Packard Enterprise', 'volume_backend_name': u'generic0', 'driver_version': u'4.0.5', 'replication_targets': [], 'pools': [{'thick_provisioning_support': True, 'bandwidth': 232926.9, 'consistent_group_snapshot_enabled': True, 'latency': 8.24, 'filter_function': None, 'compression': False, 'total_volumes': 324, 'consistent_group_replication_enabled': False, 'free_capacity_gb': 13716, 'queue_length': 6.4, 'replication_enabled': False, 'pool_name': 'FC_r5', 'goodness_function': None, 'multiattach': False, 'provisioned_capacity_gb': 49345, 'thin_provisioning_support': True, 'io_size': 140.6, 'reserved_percentage': 0, 'QoS_support': True, 'capacity_utilization': 78.03788448913583, 'replication_count': 0, 'replication_type': ['sync', 'periodic'], 'total_capacity_gb': 62453, 'location_info': u'HPE3PARDriver:1612965:FC_r5', 'avg_busy_perc': 1.4, 'throughput': 1657.0, 'max_over_subscription_ratio': '20.0'}], 'replication_enabled': False, 'storage_protocol': u'FC'} trace_logging_wrapper /usr/lib/python2.7/site-packages/cinder/utils.py:922

[Gerald McBrearty (gfm-r)]

This got around the failure ... but I'm not sure this is the correct resolution.

diff --git a/cinder/volume/drivers/hpe/hpe_3par_common.py b/cinder/volume/drivers/hpe/hpe_3par_common.py
index 95dd2b9..8130668 100644
--- a/cinder/volume/drivers/hpe/hpe_3par_common.py
+++ b/cinder/volume/drivers/hpe/hpe_3par_common.py
@@ -485,7 +485,7 @@ class HPE3PARCommon(object):

        # Get the client ID for provider_location. We only need to retrieve
        # the ID directly from the array if the driver stats are not provided.
- if not stats:
+ if not stats or 'array_id' not in stats:
            try:
                self.client_login()
                info = self.client.getStorageSystemInfo()

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/612884

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/612884
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=ec7977b7615097fb56c5c1640fb5268dba9b26d4
Submitter: Zuul
Branch: master

commit ec7977b7615097fb56c5c1640fb5268dba9b26d4
Author: chhagarw <email address hidden>
Date: Wed Oct 24 01:36:09 2018 -0400

    hpe 3par driver initialization failure

    do_setup call fails if the stats passed has the missing
    array_id, in such cases it should use the client login
    to fetch the array_id information.

    Change-Id: I36d75fbe18770d6e3ae03c05d5392f8141082a0f
    Closes-bug: #1790005

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/639756

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 14.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/639756
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=5618041bfc224af0a1cfe1bee57382ae9b46b04c
Submitter: Zuul
Branch: stable/rocky

commit 5618041bfc224af0a1cfe1bee57382ae9b46b04c
Author: chhagarw <email address hidden>
Date: Wed Oct 24 01:36:09 2018 -0400

    hpe 3par driver initialization failure

    do_setup call fails if the stats passed has the missing
    array_id, in such cases it should use the client login
    to fetch the array_id information.

    Change-Id: I36d75fbe18770d6e3ae03c05d5392f8141082a0f
    Closes-bug: #1790005
    (cherry picked from commit ec7977b7615097fb56c5c1640fb5268dba9b26d4)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 13.0.5 release.