| 2014-05-07 21:29:08 |
Yehia Beyh |
description |
When an HP 3par host in host set, the cinder volume detach will hang. Looking at the log files, the exception shows the following exception after deleting all the vluns, the host entry is a pre-existing one since the host storage is using the same 3par system:
cinder.openstack.common.rpc.amqp HTTPConflict: Conflict (HTTP 409) 77 - host is a member of a set
in master/cinder/volume/drivers/san/hp/hp_3par_common.py
def delete_vlun(self, volume, hostname):
volume_name = self._get_3par_vol_name(volume['id'])
vlun = self._get_vlun(volume_name, hostname)
if vlun is not None:
# VLUN Type of MATCHED_SET 4 requires the port to be provided
if self.VLUN_TYPE_MATCHED_SET == vlun['type']:
self.client.deleteVLUN(volume_name, vlun['lun'], hostname,
vlun['portPos'])
else:
self.client.deleteVLUN(volume_name, vlun['lun'], hostname)
try:
self._delete_3par_host(hostname)
self._remove_hosts_naming_dict_host(hostname)
>> except hpexceptions.HTTPConflict as ex:
>> # host will only be removed after all vluns
>> # have been removed
>> if 'has exported VLUN' in ex.get_description():
>> pass
>> else:
>> raise
Yes, the problem occurred in Havana. I believe the problem is in Icehouse as well, but I have not tested it yet.
Log info:
cinder.volume.drivers.san.hp.hp_3par_common [req-681562e9-8142-472d-92ef-206cc95b9146 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] VOLUME (andyvol5 : volume-f7ff0c63-8dc0-4e3d-b37a-fd2ea18390c6 osv-9-8MY43ATj2zev0uoYOQxg) Updating KEY-VALUE pair: (instance_uuid : 034995fc-bd0d-4e7a-bffc-a21013cef55f) update_volume_key_value_pair /usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py:809
2014-05-02 17:30:34.063 22459 DEBUG cinder.volume.drivers.san.hp.hp_3par_common [req-681562e9-8142-472d-92ef-206cc95b9146 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] SSH CMD = ['setvv', '-setkv', u'instance_uuid=034995fc-bd0d-4e7a-bffc-a21013cef55f', 'osv-9-8MY43ATj2zev0uoYOQxg'] _cli_run /usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py:319
2014-05-02 17:30:34.064 22459 DEBUG cinder.volume.drivers.san.hp.hp_3par_common [req-681562e9-8142-472d-92ef-206cc95b9146 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Running cmd (SSH): setvv -setkv instance_uuid=034995fc-bd0d-4e7a-bffc-a21013cef55f osv-9-8MY43ATj2zev0uoYOQxg _ssh_execute /usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py:333
2014-05-02 17:30:34.065 22459 DEBUG paramiko.transport [-] [chan 7] Max packet in: 34816 bytes _log /usr/lib64/python2.6/site-packages/paramiko/channel.py:1100
2014-05-02 17:30:34.068 22459 DEBUG paramiko.transport [-] [chan 7] Max packet out: 32768 bytes _log /usr/lib64/python2.6/site-packages/paramiko/channel.py:1100
2014-05-02 17:30:34.068 22459 INFO paramiko.transport [-] Secsh channel 7 opened.
2014-05-02 17:30:34.080 22459 DEBUG paramiko.transport [-] [chan 7] Sesch channel 7 request ok _log /usr/lib64/python2.6/site-packages/paramiko/channel.py:1100
2014-05-02 17:30:34.082 22459 DEBUG paramiko.transport [-] [chan 7] Sesch channel 7 request ok _log /usr/lib64/python2.6/site-packages/paramiko/channel.py:1100
2014-05-02 17:30:34.416 22459 DEBUG paramiko.transport [-] [chan 7] EOF received (7) _log /usr/lib64/python2.6/site-packages/paramiko/channel.py:1100
2014-05-02 17:30:34.417 22459 DEBUG paramiko.transport [-] [chan 7] EOF sent (7) _log /usr/lib64/python2.6/site-packages/paramiko/channel.py:1100
2014-05-02 17:30:34.420 22459 DEBUG cinder.volume.drivers.san.hp.hp_3par_common [req-681562e9-8142-472d-92ef-206cc95b9146 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Result was 0 _ssh_execute /usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py:360
2014-05-02 17:30:34.421 22459 DEBUG cinder.openstack.common.lockutils [req-681562e9-8142-472d-92ef-206cc95b9146 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Released file lock "3par" at /var/lib/cinder/tmp/cinder-3par for method "attach_volume"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:239
2014-05-02 17:30:34.458 22459 DEBUG cinder.openstack.common.lockutils [req-681562e9-8142-472d-92ef-206cc95b9146 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Released file lock "f7ff0c63-8dc0-4e3d-b37a-fd2ea18390c6" at /var/lib/cinder/tmp/cinder-f7ff0c63-8dc0-4e3d-b37a-fd2ea18390c6 for method "do_attach"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:239
2014-05-02 17:30:34.460 22459 DEBUG cinder.openstack.common.rpc.amqp [req-681562e9-8142-472d-92ef-206cc95b9146 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] UNIQUE_ID is 3653b3c10d3543c7b19901b7c016df8d. _add_unique_id /usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py:345
2014-05-02 17:30:34.464 22459 DEBUG cinder.openstack.common.rpc.amqp [req-681562e9-8142-472d-92ef-206cc95b9146 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] UNIQUE_ID is 0b6be42a32194830bf1c8b474c94100e. _add_unique_id /usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py:345
2014-05-02 17:30:56.281 22440 WARNING cinder.service [-] Signal initiated backend status output:
2014-05-02 17:30:56.283 22440 WARNING cinder.service [-] Backend: ci-5254002ABE33@3PAR-FC-1635483 (pid: 22459, failed: False, forks: [1399051099.3576369], workers: 1)
2014-05-02 17:31:23.150 22459 DEBUG cinder.openstack.common.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_msg_id': u'167948f2784a4e1d8b8742cc84eb92c9', u'_context_quota_class': None, u'_context_timestamp': u'2014-05-02T17:31:23.093771', u'_context_request_id': u'req-2118f4aa-4468-41cd-bfba-2c055d2d3275', u'_context_service_catalog': [{u'endpoints': [{u'adminURL': u'https://135.1.243.60/rest/compute/v1.1/66c49057ea4d478e975c576a926a1415', u'region': u'public', u'internalURL': u'http://127.0.0.1:8774/v1.1/66c49057ea4d478e975c576a926a1415', u'id': u'f011e2ceb14248b8847eae4a52ef8fa4', u'publicURL': u'https://135.1.243.60/rest/compute/v1.1/66c49057ea4d478e975c576a926a1415'}], u'endpoints_links': [], u'type': u'compute', u'name': u'nova'}], u'_context_tenant': u'66c49057ea4d478e975c576a926a1415', u'args': {u'connector': {u'ip': u'127.0.0.1', u'host': u'kvm1-4.hpiscmgmt.local', u'wwnns': [u'1000569b06d00011', u'1000569b06d00013'], u'initiator': u'iqn.1994-05.com.redhat:6a37bb893c6', u'wwpns': [u'1000569b06d00010', u'1000569b06d00012']}, u'force': False, u'volume_id': u'f7ff0c63-8dc0-4e3d-b37a-fd2ea18390c6'}, u'namespace': None, u'_context_auth_token': '<SANITIZED>', u'_context_project_name': u'administrator', u'_context_is_admin': True, u'version': u'1.0', u'_context_project_id': u'66c49057ea4d478e975c576a926a1415', u'_context_user': u'e08faaac660b4cefa8371d37e2717113', u'_unique_id': u'4af8c6fbad9b429fb5e46a858be45223', u'_context_read_deleted': u'no', u'_context_user_id': u'e08faaac660b4cefa8371d37e2717113', u'method': u'terminate_connection', u'_context_remote_address': u'192.168.4.11'} _safe_log /usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/common.py:290
2014-05-02 17:31:23.153 22459 DEBUG cinder.openstack.common.rpc.amqp [-] unpacked context: {'project_name': u'administrator', 'user_id': u'e08faaac660b4cefa8371d37e2717113', 'roles': [u'admin'], 'timestamp': u'2014-05-02T17:31:23.093771', 'auth_token': '<SANITIZED>', 'remote_address': u'192.168.4.11', 'quota_class': None, 'is_admin': True, 'user': u'e08faaac660b4cefa8371d37e2717113', 'service_catalog': [{u'endpoints_links': [], u'endpoints': [{u'adminURL': u'https://135.1.243.60/rest/compute/v1.1/66c49057ea4d478e975c576a926a1415', u'region': u'public', u'publicURL': u'https://135.1.243.60/rest/compute/v1.1/66c49057ea4d478e975c576a926a1415', u'internalURL': u'http://127.0.0.1:8774/v1.1/66c49057ea4d478e975c576a926a1415', u'id': u'f011e2ceb14248b8847eae4a52ef8fa4'}], u'type': u'compute', u'name': u'nova'}], 'request_id': u'req-2118f4aa-4468-41cd-bfba-2c055d2d3275', 'project_id': u'66c49057ea4d478e975c576a926a1415', 'read_deleted': u'no', 'tenant': u'66c49057ea4d478e975c576a926a1415'} _safe_log /usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/common.py:290
2014-05-02 17:31:23.182 22459 DEBUG cinder.openstack.common.lockutils [req-2118f4aa-4468-41cd-bfba-2c055d2d3275 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Got semaphore "3par" for method "terminate_connection"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:191
2014-05-02 17:31:23.183 22459 DEBUG cinder.openstack.common.lockutils [req-2118f4aa-4468-41cd-bfba-2c055d2d3275 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Attempting to grab file lock "3par" for method "terminate_connection"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:202
2014-05-02 17:31:23.185 22459 DEBUG cinder.openstack.common.lockutils [req-2118f4aa-4468-41cd-bfba-2c055d2d3275 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Got file lock "3par" at /var/lib/cinder/tmp/cinder-3par for method "terminate_connection"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:232
2014-05-02 17:31:23.186 22459 DEBUG cinder.volume.drivers.san.hp.hp_3par_common [req-2118f4aa-4468-41cd-bfba-2c055d2d3275 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Connecting to 3PAR client_login /usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py:190
2014-05-02 17:31:23.187 22459 DEBUG hp3parclient.http [-]
REQ: curl -i https://192.168.3.20:8080/api/v1/credentials -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-3parclient"
_http_log_req /usr/lib/python2.6/site-packages/hp3parclient/http.py:159
2014-05-02 17:31:23.188 22459 DEBUG hp3parclient.http [-] REQ BODY: {"password": "3pardata", "user": "3paradm"}
_http_log_req /usr/lib/python2.6/site-packages/hp3parclient/http.py:161
2014-05-02 17:31:23.477 22459 DEBUG hp3parclient.http [-] RESP:{'cache-control': 'no-cache',
'connection': 'close',
'content-type': 'application/json',
'date': 'Fri, 02 May 2014 17:32:17 GMT',
'location': '/api/v1/credentials/272d1-c810c89243580a0047fec3b7c713b650-a1d66353',
'pragma': 'no-cache',
'server': 'hp3par-wsapi',
'status': '201'}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:166
2014-05-02 17:31:23.477 22459 DEBUG hp3parclient.http [-] RESP BODY:{"key":"272d1-c810c89243580a0047fec3b7c713b650-a1d66353"}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:167
2014-05-02 17:31:23.478 22459 DEBUG hp3parclient.http [-]
REQ: curl -i https://192.168.3.20:8080/api/v1/hosts/kvm1-4 -X GET -H "X-Hp3Par-Wsapi-Sessionkey: 272d1-c810c89243580a0047fec3b7c713b650-a1d66353" -H "Accept: application/json" -H "User-Agent: python-3parclient"
_http_log_req /usr/lib/python2.6/site-packages/hp3parclient/http.py:159
2014-05-02 17:31:23.552 22459 DEBUG hp3parclient.http [-] RESP:{'cache-control': 'no-cache',
'connection': 'close',
'content-location': u'https://192.168.3.20:8080/api/v1/hosts/kvm1-4',
'content-type': 'application/json',
'date': 'Fri, 02 May 2014 17:32:17 GMT',
'pragma': 'no-cache',
'server': 'hp3par-wsapi',
'status': '200'}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:166
2014-05-02 17:31:23.553 22459 DEBUG hp3parclient.http [-] RESP BODY:{"id":4,"name":"kvm1-4","FCPaths":[{"wwn":"1000569B06D00012","portPos":{"node":2,"slot":2,"cardPort":4},"hostSpeed":0},{"wwn":"1000569B06D00012","portPos":{"node":3,"slot":2,"cardPort":4},"hostSpeed":0},{"wwn":"1000569B06D00012","portPos":{"node":2,"slot":2,"cardPort":2},"hostSpeed":0},{"wwn":"1000569B06D00010","portPos":{"node":2,"slot":2,"cardPort":1},"hostSpeed":0},{"wwn":"1000569B06D00010","portPos":{"node":3,"slot":2,"cardPort":3},"hostSpeed":0},{"wwn":"1000569B06D00010","portPos":{"node":2,"slot":2,"cardPort":3},"hostSpeed":0},{"wwn":"1000569B06D00012","portPos":{"node":3,"slot":2,"cardPort":2},"hostSpeed":0},{"wwn":"1000569B06D00010","portPos":{"node":3,"slot":2,"cardPort":1},"hostSpeed":0}],"iSCSIPaths":[],"persona":1,"initiatorChapEnabled":false,"targetChapEnabled":false}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:167
2014-05-02 17:31:23.555 22459 DEBUG hp3parclient.http [-]
REQ: curl -i https://192.168.3.20:8080/api/v1/vluns -X GET -H "X-Hp3Par-Wsapi-Sessionkey: 272d1-c810c89243580a0047fec3b7c713b650-a1d66353" -H "Accept: application/json" -H "User-Agent: python-3parclient"
_http_log_req /usr/lib/python2.6/site-packages/hp3parclient/http.py:159
2014-05-02 17:31:23.629 22459 DEBUG hp3parclient.http [-] RESP:{'cache-control': 'no-cache',
'connection': 'close',
'content-location': 'https://192.168.3.20:8080/api/v1/vluns',
'content-type': 'application/json',
'date': 'Fri, 02 May 2014 17:32:17 GMT',
'pragma': 'no-cache',
'server': 'hp3par-wsapi',
'status': '200'}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:166
2014-05-02 17:31:23.629 22459 DEBUG hp3parclient.http [-] RESP BODY:{"total":39,"members":[{"lun":0,"volumeName":"alkvmmgmt_dataLUN","hostname":"alkvmmgmt.alu.local","remoteName":"500143802423C38A","portPos":{"node":2,"slot":1,"cardPort":1},"type":3,"volumeWWN":"60002AC0000000000000000800008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"alkvmmgmt_dataLUN","hostname":"alkvmmgmt.alu.local","remoteName":"500143802423C388","portPos":{"node":3,"slot":1,"cardPort":1},"type":3,"volumeWWN":"60002AC0000000000000000800008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-9-8MY43ATj2zev0uoYOQxg","hostname":"kvm1-4","remoteName":"1000569B06D00010","portPos":{"node":3,"slot":2,"cardPort":1},"type":3,"volumeWWN":"60002AC0000000000000005900008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-9-8MY43ATj2zev0uoYOQxg","hostname":"kvm1-4","remoteName":"1000569B06D00012","portPos":{"node":3,"slot":2,"cardPort":2},"type":3,"volumeWWN":"60002AC0000000000000005900008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-nvaKw1lCRoqYXq6c8hc4Jg","hostname":"kvm1-12.alu.local","remoteName":"1000569B06D00028","portPos":{"node":3,"slot":2,"cardPort":1},"type":3,"volumeWWN":"60002AC0000000000000002500008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-nvaKw1lCRoqYXq6c8hc4Jg","hostname":"kvm1-12.alu.local","remoteName":"1000569B06D0002A","portPos":{"node":3,"slot":2,"cardPort":2},"type":3,"volumeWWN":"60002AC0000000000000002500008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-4zuEZ6y0SK6Rr.-Z2DU2tQ","hostname":"kvm1-14.alu.local","remoteName":"1000569B06D00030","portPos":{"node":3,"slot":2,"cardPort":1},"type":3,"volumeWWN":"60002AC0000000000000001800008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-4zuEZ6y0SK6Rr.-Z2DU2tQ","hostname":"kvm1-14.alu.local","remoteName":"1000569B06D00032","portPos":{"node":3,"slot":2,"cardPort":2},"type":3,"volumeWWN":"60002AC0000000000000001800008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-BchPkdC6RTC1yCto9FeqTQ","hostname":"kvm1-2.alu.local","remoteName":"1000569B06D00008","portPos":{"node":3,"slot":2,"cardPort":1},"type":3,"volumeWWN":"60002AC0000000000000001700008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-BchPkdC6RTC1yCto9FeqTQ","hostname":"kvm1-2.alu.local","remoteName":"1000569B06D0000A","portPos":{"node":3,"slot":2,"cardPort":2},"type":3,"volumeWWN":"60002AC0000000000000001700008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-9-8MY43ATj2zev0uoYOQxg","hostname":"kvm1-4","remoteName":"1000569B06D00010","portPos":{"node":2,"slot":2,"cardPort":3},"type":3,"volumeWWN":"60002AC0000000000000005900008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-9-8MY43ATj2zev0uoYOQxg","hostname":"kvm1-4","remoteName":"1000569B06D00010","portPos":{"node":3,"slot":2,"cardPort":3},"type":3,"volumeWWN":"60002AC0000000000000005900008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-9-8MY43ATj2zev0uoYOQxg","hostname":"kvm1-4","remoteName":"1000569B06D00010","portPos":{"node":2,"slot":2,"cardPort":1},"type":3,"volumeWWN":"60002AC0000000000000005900008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-9-8MY43ATj2zev0uoYOQxg","hostname":"kvm1-4","remoteName":"1000569B06D00012","portPos":{"node":2,"slot":2,"cardPort":2},"type":3,"volumeWWN":"60002AC0000000000000005900008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-9-8MY43ATj2zev0uoYOQxg","hostname":"kvm1-4","remoteName":"1000569B06D00012","portPos":{"node":3,"slot":2,"cardPort":4},"type":3,"volumeWWN":"60002AC0000000000000005900008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-9-8MY43ATj2zev0uoYOQxg","hostname":"kvm1-4","remoteName":"1000569B06D00012","portPos":{"node":2,"slot":2,"cardPort":4},"type":3,"volumeWWN":"60002AC0000000000000005900008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-nvaKw1lCRoqYXq6c8hc4Jg","hostname":"kvm1-12.alu.local","remoteName":"1000569B06D00028","portPos":{"node":2,"slot":2,"cardPort":3},"type":3,"volumeWWN":"60002AC0000000000000002500008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-nvaKw1lCRoqYXq6c8hc4Jg","hostname":"kvm1-12.alu.local","remoteName":"1000569B06D00028","portPos":{"node":2,"slot":2,"cardPort":1},"type":3,"volumeWWN":"60002AC0000000000000002500008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-nvaKw1lCRoqYXq6c8hc4Jg","hostname":"kvm1-12.alu.local","remoteName":"1000569B06D00028","portPos":{"node":3,"slot":2,"cardPort":3},"type":3,"volumeWWN":"60002AC0000000000000002500008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-nvaKw1lCRoqYXq6c8hc4Jg","hostname":"kvm1-12.alu.local","remoteName":"1000569B06D0002A","portPos":{"node":2,"slot":2,"cardPort":2},"type":3,"volumeWWN":"60002AC0000000000000002500008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-nvaKw1lCRoqYXq6c8hc4Jg","hostname":"kvm1-12.alu.local","remoteName":"1000569B06D0002A","portPos":{"node":2,"slot":2,"cardPort":4},"type":3,"volumeWWN":"60002AC0000000000000002500008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-nvaKw1lCRoqYXq6c8hc4Jg","hostname":"kvm1-12.alu.local","remoteName":"1000569B06D0002A","portPos":{"node":3,"slot":2,"cardPort":4},"type":3,"volumeWWN":"60002AC0000000000000002500008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-4zuEZ6y0SK6Rr.-Z2DU2tQ","hostname":"kvm1-14.alu.local","remoteName":"1000569B06D00030","portPos":{"node":2,"slot":2,"cardPort":3},"type":3,"volumeWWN":"60002AC0000000000000001800008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-4zuEZ6y0SK6Rr.-Z2DU2tQ","hostname":"kvm1-14.alu.local","remoteName":"1000569B06D00030","portPos":{"node":2,"slot":2,"cardPort":1},"type":3,"volumeWWN":"60002AC0000000000000001800008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-4zuEZ6y0SK6Rr.-Z2DU2tQ","hostname":"kvm1-14.alu.local","remoteName":"1000569B06D00030","portPos":{"node":3,"slot":2,"cardPort":3},"type":3,"volumeWWN":"60002AC0000000000000001800008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-4zuEZ6y0SK6Rr.-Z2DU2tQ","hostname":"kvm1-14.alu.local","remoteName":"1000569B06D00032","portPos":{"node":2,"slot":2,"cardPort":2},"type":3,"volumeWWN":"60002AC0000000000000001800008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-4zuEZ6y0SK6Rr.-Z2DU2tQ","hostname":"kvm1-14.alu.local","remoteName":"1000569B06D00032","portPos":{"node":2,"slot":2,"cardPort":4},"type":3,"volumeWWN":"60002AC0000000000000001800008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-4zuEZ6y0SK6Rr.-Z2DU2tQ","hostname":"kvm1-14.alu.local","remoteName":"1000569B06D00032","portPos":{"node":3,"slot":2,"cardPort":4},"type":3,"volumeWWN":"60002AC0000000000000001800008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-BchPkdC6RTC1yCto9FeqTQ","hostname":"kvm1-2.alu.local","remoteName":"1000569B06D00008","portPos":{"node":3,"slot":2,"cardPort":3},"type":3,"volumeWWN":"60002AC0000000000000001700008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-BchPkdC6RTC1yCto9FeqTQ","hostname":"kvm1-2.alu.local","remoteName":"1000569B06D00008","portPos":{"node":2,"slot":2,"cardPort":3},"type":3,"volumeWWN":"60002AC0000000000000001700008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-BchPkdC6RTC1yCto9FeqTQ","hostname":"kvm1-2.alu.local","remoteName":"1000569B06D00008","portPos":{"node":2,"slot":2,"cardPort":1},"type":3,"volumeWWN":"60002AC0000000000000001700008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-BchPkdC6RTC1yCto9FeqTQ","hostname":"kvm1-2.alu.local","remoteName":"1000569B06D0000A","portPos":{"node":3,"slot":2,"cardPort":4},"type":3,"volumeWWN":"60002AC0000000000000001700008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-BchPkdC6RTC1yCto9FeqTQ","hostname":"kvm1-2.alu.local","remoteName":"1000569B06D0000A","portPos":{"node":2,"slot":2,"cardPort":2},"type":3,"volumeWWN":"60002AC0000000000000001700008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"osv-BchPkdC6RTC1yCto9FeqTQ","hostname":"kvm1-2.alu.local","remoteName":"1000569B06D0000A","portPos":{"node":2,"slot":2,"cardPort":4},"type":3,"volumeWWN":"60002AC0000000000000001700008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":true},{"lun":0,"volumeName":"alkvmmgmt_dataLUN","hostname":"alkvmmgmt.alu.local","type":3,"volumeWWN":"60002AC0000000000000000800008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":false},{"lun":0,"volumeName":"osv-BchPkdC6RTC1yCto9FeqTQ","hostname":"kvm1-2.alu.local","type":3,"volumeWWN":"60002AC0000000000000001700008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":false},{"lun":0,"volumeName":"osv-4zuEZ6y0SK6Rr.-Z2DU2tQ","hostname":"kvm1-14.alu.local","type":3,"volumeWWN":"60002AC0000000000000001800008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":false},{"lun":0,"volumeName":"osv-nvaKw1lCRoqYXq6c8hc4Jg","hostname":"kvm1-12.alu.local","type":3,"volumeWWN":"60002AC0000000000000002500008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":false},{"lun":0,"volumeName":"osv-9-8MY43ATj2zev0uoYOQxg","hostname":"kvm1-4","type":3,"volumeWWN":"60002AC0000000000000005900008A9B","multipathing":1,"failedPathPol":1,"failedPathInterval":0,"active":false}]}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:167
2014-05-02 17:31:23.644 22459 DEBUG hp3parclient.http [-]
REQ: curl -i https://192.168.3.20:8080/api/v1/vluns/osv-9-8MY43ATj2zev0uoYOQxg,0,kvm1-4 DELETE -H "X-Hp3Par-Wsapi-Sessionkey: 272d1-c810c89243580a0047fec3b7c713b650-a1d66353" -H "Accept: application/json" -H "User-Agent: python-3parclient"
_http_log_req /usr/lib/python2.6/site-packages/hp3parclient/http.py:159
2014-05-02 17:31:23.697 22459 DEBUG hp3parclient.http [-] RESP:{'cache-control': 'no-cache',
'connection': 'close',
'date': 'Fri, 02 May 2014 17:32:17 GMT',
'pragma': 'no-cache',
'server': 'hp3par-wsapi',
'status': '200'}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:166
2014-05-02 17:31:23.698 22459 DEBUG hp3parclient.http [-] RESP BODY:
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:167
2014-05-02 17:31:23.699 22459 DEBUG hp3parclient.http [-]
REQ: curl -i https://192.168.3.20:8080/api/v1/hosts/kvm1-4 DELETE -H "X-Hp3Par-Wsapi-Sessionkey: 272d1-c810c89243580a0047fec3b7c713b650-a1d66353" -H "Accept: application/json" -H "User-Agent: python-3parclient"
_http_log_req /usr/lib/python2.6/site-packages/hp3parclient/http.py:159
2014-05-02 17:31:23.866 22459 DEBUG hp3parclient.http [-] RESP:{'connection': 'close',
'content-type': 'application/json',
'date': 'Fri, 02 May 2014 17:32:17 GMT',
'server': 'hp3par-wsapi',
'status': '409'}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:166
2014-05-02 17:31:23.867 22459 DEBUG hp3parclient.http [-] RESP BODY:{"code":77,"desc":"host is a member of a set"}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:167
2014-05-02 17:31:23.868 22459 DEBUG hp3parclient.http [-]
REQ: curl -i https://192.168.3.20:8080/api/v1/credentials/272d1-c810c89243580a0047fec3b7c713b650-a1d66353 DELETE -H "X-Hp3Par-Wsapi-Sessionkey: 272d1-c810c89243580a0047fec3b7c713b650-a1d66353" -H "Accept: application/json" -H "User-Agent: python-3parclient"
_http_log_req /usr/lib/python2.6/site-packages/hp3parclient/http.py:159
2014-05-02 17:31:23.887 22459 DEBUG hp3parclient.http [-] RESP:{'cache-control': 'no-cache',
'connection': 'close',
'date': 'Fri, 02 May 2014 17:32:17 GMT',
'pragma': 'no-cache',
'server': 'hp3par-wsapi',
'status': '200'}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:166
2014-05-02 17:31:23.888 22459 DEBUG hp3parclient.http [-] RESP BODY:
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:167
2014-05-02 17:31:23.889 22459 DEBUG cinder.volume.drivers.san.hp.hp_3par_common [req-2118f4aa-4468-41cd-bfba-2c055d2d3275 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Disconnect from 3PAR client_logout /usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py:201
2014-05-02 17:31:23.890 22459 DEBUG cinder.openstack.common.lockutils [req-2118f4aa-4468-41cd-bfba-2c055d2d3275 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Released file lock "3par" at /var/lib/cinder/tmp/cinder-3par for method "terminate_connection"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:239
2014-05-02 17:31:23.891 22459 ERROR cinder.openstack.common.rpc.amqp [req-2118f4aa-4468-41cd-bfba-2c055d2d3275 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Exception during message handling
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp **args)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 819, in wrapper
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 658, in terminate_connection
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp self.driver.terminate_connection(volume_ref, connector, force=force)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py", line 233, in inner
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_fc.py", line 233, in terminate_connection
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp wwn=connector['wwpns'])
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py", line 1099, in terminate_connection
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp self.delete_vlun(volume, hostname)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py", line 546, in delete_vlun
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp self._delete_3par_host(hostname)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py", line 408, in _delete_3par_host
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp self.client.deleteHost(hostname)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/hp3parclient/client.py", line 388, in deleteHost
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp reponse, body = self.http.delete('/hosts/%s' % name)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/hp3parclient/http.py", line 327, in delete
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp return self._cs_request(url, 'DELETE', **kwargs)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/hp3parclient/http.py", line 231, in _cs_request
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp **kwargs)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/hp3parclient/http.py", line 205, in _time_request
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp resp, body = self.request(url, method, **kwargs)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/hp3parclient/http.py", line 199, in request
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp raise exceptions.from_response(resp, body)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp HTTPConflict: Conflict (HTTP 409) 77 - host is a member of a set
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp
2014-05-02 17:31:23.897 22459 ERROR cinder.openstack.common.rpc.common [req-2118f4aa-4468-41cd-bfba-2c055d2d3275 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Returning exception Conflict (HTTP 409) 77 - host is a member of a set to caller |
When an HP 3par host in host set, the cinder volume detach will hang. Looking at the log files, the exception shows the following exception after deleting all the vluns, the host entry is a pre-existing one since the host storage is using the same 3par system:
cinder.openstack.common.rpc.amqp HTTPConflict: Conflict (HTTP 409) 77 - host is a member of a set
in master/cinder/volume/drivers/san/hp/hp_3par_common.py
def delete_vlun(self, volume, hostname):
volume_name = self._get_3par_vol_name(volume['id'])
vlun = self._get_vlun(volume_name, hostname)
if vlun is not None:
# VLUN Type of MATCHED_SET 4 requires the port to be provided
if self.VLUN_TYPE_MATCHED_SET == vlun['type']:
self.client.deleteVLUN(volume_name, vlun['lun'], hostname,
vlun['portPos'])
else:
self.client.deleteVLUN(volume_name, vlun['lun'], hostname)
try:
self._delete_3par_host(hostname)
self._remove_hosts_naming_dict_host(hostname)
>> except hpexceptions.HTTPConflict as ex:
>> # host will only be removed after all vluns
>> # have been removed
>> if 'has exported VLUN' in ex.get_description():
>> pass
>> else:
>> raise
Yes, the problem occurred in Havana. I believe the problem is in Icehouse as well, but I have not tested it yet.
Log info:
ccontrol': 'no-cache',
'connection': 'close',
'date': 'Fri, 02 May 2014 17:32:17 GMT',
'pragma': 'no-cache',
'server': 'hp3par-wsapi',
'status': '200'}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:166
2014-05-02 17:31:23.698 22459 DEBUG hp3parclient.http [-] RESP BODY:
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:167
2014-05-02 17:31:23.699 22459 DEBUG hp3parclient.http [-]
REQ: curl -i https://192.168.3.20:8080/api/v1/hosts/kvm1-4 DELETE -H "X-Hp3Par-Wsapi-Sessionkey: 272d1-c810c89243580a0047fec3b7c713b650-a1d66353" -H "Accept: application/json" -H "User-Agent: python-3parclient"
_http_log_req /usr/lib/python2.6/site-packages/hp3parclient/http.py:159
2014-05-02 17:31:23.866 22459 DEBUG hp3parclient.http [-] RESP:{'connection': 'close',
'content-type': 'application/json',
'date': 'Fri, 02 May 2014 17:32:17 GMT',
'server': 'hp3par-wsapi',
'status': '409'}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:166
2014-05-02 17:31:23.867 22459 DEBUG hp3parclient.http [-] RESP BODY:{"code":77,"desc":"host is a member of a set"}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:167
2014-05-02 17:31:23.868 22459 DEBUG hp3parclient.http [-]
REQ: curl -i https://192.168.3.20:8080/api/v1/credentials/272d1-c810c89243580a0047fec3b7c713b650-a1d66353 DELETE -H "X-Hp3Par-Wsapi-Sessionkey: 272d1-c810c89243580a0047fec3b7c713b650-a1d66353" -H "Accept: application/json" -H "User-Agent: python-3parclient"
_http_log_req /usr/lib/python2.6/site-packages/hp3parclient/http.py:159
2014-05-02 17:31:23.887 22459 DEBUG hp3parclient.http [-] RESP:{'cache-control': 'no-cache',
'connection': 'close',
'date': 'Fri, 02 May 2014 17:32:17 GMT',
'pragma': 'no-cache',
'server': 'hp3par-wsapi',
'status': '200'}
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:166
2014-05-02 17:31:23.888 22459 DEBUG hp3parclient.http [-] RESP BODY:
_http_log_resp /usr/lib/python2.6/site-packages/hp3parclient/http.py:167
2014-05-02 17:31:23.889 22459 DEBUG cinder.volume.drivers.san.hp.hp_3par_common [req-2118f4aa-4468-41cd-bfba-2c055d2d3275 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Disconnect from 3PAR client_logout /usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py:201
2014-05-02 17:31:23.890 22459 DEBUG cinder.openstack.common.lockutils [req-2118f4aa-4468-41cd-bfba-2c055d2d3275 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Released file lock "3par" at /var/lib/cinder/tmp/cinder-3par for method "terminate_connection"... inner /usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py:239
2014-05-02 17:31:23.891 22459 ERROR cinder.openstack.common.rpc.amqp [req-2118f4aa-4468-41cd-bfba-2c055d2d3275 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Exception during message handling
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp **args)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 819, in wrapper
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 658, in terminate_connection
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp self.driver.terminate_connection(volume_ref, connector, force=force)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/lockutils.py", line 233, in inner
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_fc.py", line 233, in terminate_connection
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp wwn=connector['wwpns'])
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py", line 1099, in terminate_connection
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp self.delete_vlun(volume, hostname)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py", line 546, in delete_vlun
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp self._delete_3par_host(hostname)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/san/hp/hp_3par_common.py", line 408, in _delete_3par_host
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp self.client.deleteHost(hostname)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/hp3parclient/client.py", line 388, in deleteHost
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp reponse, body = self.http.delete('/hosts/%s' % name)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/hp3parclient/http.py", line 327, in delete
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp return self._cs_request(url, 'DELETE', **kwargs)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/hp3parclient/http.py", line 231, in _cs_request
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp **kwargs)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/hp3parclient/http.py", line 205, in _time_request
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp resp, body = self.request(url, method, **kwargs)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/hp3parclient/http.py", line 199, in request
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp raise exceptions.from_response(resp, body)
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp HTTPConflict: Conflict (HTTP 409) 77 - host is a member of a set
2014-05-02 17:31:23.891 22459 TRACE cinder.openstack.common.rpc.amqp
2014-05-02 17:31:23.897 22459 ERROR cinder.openstack.common.rpc.common [req-2118f4aa-4468-41cd-bfba-2c055d2d3275 e08faaac660b4cefa8371d37e2717113 66c49057ea4d478e975c576a926a1415] Returning exception Conflict (HTTP 409) 77 - host is a member of a set to caller |
|
