Concurrent iSCSI Target Portal Login fails with Cinder HPE 3PAR iSCSI driver

[Hariprasad S Venkatasubramanian]

When multiople cinder volumes are created concurrently from the same source image for the first time (image-volume-cache-miss flow), some of the volumes fail getting created. The number of failures is found to vary across runs. They are ultimately set to "error" state and the underlying exception noted (in c-vol logs) is "Could not login to any iSCSI portal."

Affected Version: stable/mitaka

Steps to Reproduce:
1. Configure cinder.conf to use HPE3PARISCSIDriver as given in config-reference.
2. chap auth should be enabled and image volume cache should be enabled.
2. Complete all other cinder.conf settings.

   [HPE-3PAR-backend]
   san_password=<your_san_password>
   hp3par_username=<3par_username>
   volume_backend_name=<your_volume_backend_name>
   san_login=<your_san_login_name>
   image_volume_cache_enabled=True
   hp3par_api_url=<your_3par_management_wsapi_endpoint_url>
   volume_driver=cinder.volume.drivers.san.hp.hp_3par_iscsi.HP3PARISCSIDriver
   hp3par_password=<3par_password>
   hp3par_iscsi_ips=<iscsi_target_portal_ips(comma separated list)>
   hp3par_cpg=<3par_cpg_name>
   hp3par_iscsi_chap_enabled=True
   san_ip=<san_ip>

3. Start all cinder services and ensure the driver gets initialized successfully.
4. Execute the following command from cinder node via CLI:

   $ export TYP=<your_vol_type>
   $ export IMG=<glance_image_id>
   $ for i in `seq 15`; do cinder create --image-id $IMG --name $TYP$i --volume-type $TYP 100 & done

5. Execute cinder list and observe the status of the created volumes.
6. Rinse and repeat step 4 and 5 a few times, each time with a different image (to exercise vol-cache-miss flow)

NOTE:
 * This issue is NOT seen in HPE3PARFCDriver
 * This issue is not reproducible with vol-cache-enabled set to FALSE
 * This issue is not seen when exercising vol-cache-hit code path (when image is available in vol backed cache)