3PAR: HTTPConflict: Conflict (HTTP 409) 16 - host exists

[Gorka Eguileor]

With the 3PAR driver creating volume from images fail with:

2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc [req-6fe0a3e5-3957-4e6d-a861-1ba55acdf4e9 8fff3b9af5364b1893702c71c4786dfa 59bfce3d280d4f839756ae97ca6f8e08 - default default] Create FC host caught HTTP conflict code: 16: HTTPConflict: Conflict (HTTP 409) 16 - host exists
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc Traceback (most recent call last):
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hpe/hpe_3par_fc.py", line 336, in _create_3par_fibrechan_host
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc 'persona': persona_id})
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc File "/usr/lib/python2.7/site-packages/hpe3parclient/client.py", line 1684, in createHost
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc response, body = self.http.post('/hosts', body=info)
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc File "/usr/lib/python2.7/site-packages/hpe3parclient/http.py", line 376, in post
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc return self._cs_request(url, 'POST', **kwargs)
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc File "/usr/lib/python2.7/site-packages/hpe3parclient/http.py", line 321, in _cs_request
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc **kwargs)
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc File "/usr/lib/python2.7/site-packages/hpe3parclient/http.py", line 297, in _time_request
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc resp, body = self.request(url, method, **kwargs)
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc File "/usr/lib/python2.7/site-packages/hpe3parclient/http.py", line 262, in request
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc raise exceptions.from_response(resp, body)
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc HTTPConflict: Conflict (HTTP 409) 16 - host exists
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc

This is bug in the 3PAR driver, that I believe affects both iSCSI and FC drivers, and I think there are 2 possible causes:

Cause #1:
---------
A race conditions when creating a host on the 3PAR Array to use it in initialize_connection.

Assuming we have 2 calls to initialize_connection for different volumes to connect to the same host (be it controller or compute), then the following timeline will raise the exception:

    - Call #1: Cinder driver checks if the host with the WWNs exists ==> It doesn't
    - Call #1: Cinder driver calls createHost for the host & WWNs

    - Call #2: Cinder driver checks if the host with the WWNs exists ==> It doesn't
    - Call #2: Cinder driver calls createHost for the host & WWNs

    - Call #1: Storage Array creates the host

    - Call #2: Storage Array fails because the host exists, returning HTTP status code 409 and Operation-specific API error code EXISTENT_HOST (16 - The host exists).

A possible fix would be on the `hpeexceptions.HTTPConflict` handling in both the `_create_3par_fibrechan_host` and `_create_3par_iscsi_host` methods, where the driver needs to check if `path_conflict.get_code()` is `EXISTENT_HOST` and then see if the host exists with the WWNs or not (as the race could be between iSCSI and FC drivers for the same backend). If it doesn't have the WWNs, then it needs to add them.

Cause #2:
---------

It may also happen if we use a mix of iSCSI and FC 3PAR drivers on the same host.

FC first then iSCSI:
- FC initialize_connection creates the host with the WWNs
- iSCSI initialize_connection cannot find the host on the 3PAR array with the iSCSI initiator, so i tries to create the host, which fails because the host already exists (it should add the iSCSI initiator name to the existing one instead)

If the iSCSI happens first it's the same thing, but reversed, where the FC won't find the host with those WWNs and will fail to create it because it already exist with the iSCSI initiator info.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/756711