Comment 12 for bug 1837686

Hi Tao,

I modified the code that the test suite uses in order to expand the time, from 30 seconds to 3 minutes to wait for the shell. Now, I got this error with ISO `BUILD_DATE="2019-08-25 23:30:00 +0000" `

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
20190827 09:15:00.486 - INFO - +---- START KW: OpenStack.Create Volume [ ${cirros_volume_size} | ${cirros_image_name} | --bootable | ${cirros_volume_name_2} ]
20190827 09:15:00.486 - INFO - +----- START KW: BuiltIn.Set Variable [ openstack volume create ]
20190827 09:15:00.486 - INFO - ${openstack_cmd} = openstack volume create
20190827 09:15:00.486 - INFO - +----- END KW: BuiltIn.Set Variable (0)
20190827 09:15:00.487 - INFO - +----- START KW: BuiltIn.Catenate [ ${openstack_cmd} | --size ${size} | --image ${image} | ${bootable} | ${name} ]
20190827 09:15:00.487 - INFO - ${cmd} = openstack volume create --size 20 --image cirros --bootable vol-cirros-2
20190827 09:15:00.487 - INFO - +----- END KW: BuiltIn.Catenate (1)
20190827 09:15:00.487 - INFO - +----- START KW: OpenStack.Run OS Command [ ${cmd} | True | 3 min ]
20190827 09:15:00.487 - INFO - +------ START KW: BuiltIn.Set Variable [ export OS_CLOUD=openstack_helm ]
20190827 09:15:00.487 - INFO - ${load_os_token} = export OS_CLOUD=openstack_helm
20190827 09:15:00.487 - INFO - +------ END KW: BuiltIn.Set Variable (0)
20190827 09:15:00.488 - INFO - +------ START KW: SSHLibrary.Execute Command [ ${load_os_token} && ${cmd} | return_stdout=True | return_stderr=True | return_rc=True | timeout=${timeout} ]
20190827 09:15:00.488 - INFO - Executing command 'export OS_CLOUD=openstack_helm && openstack volume create --size 20 --image cirros --bootable vol-cirros-2'.
20190827 09:17:09.079 - INFO - Command exited with return code 1.
20190827 09:17:09.079 - INFO - ${stdout} =
20190827 09:17:09.079 - INFO - ${stderr} = Unable to establish connection to http://cinder-api.openstack.svc.cluster.local:8776/v2/7a413f8546d246059fd3e95f2bf1977f/volumes: HTTPConnectionPool(host='cinder-api.openstack.svc.cluster.local', port...
20190827 09:17:09.079 - INFO - ${rc} = 1
20190827 09:17:09.080 - INFO - +------ END KW: SSHLibrary.Execute Command (128592)
20190827 09:17:09.080 - INFO - +------ START KW: BuiltIn.Create Dictionary [ stdout=${stdout} | stderr=${stderr} | rc=${rc} ]
20190827 09:17:09.082 - INFO - ${res} = {u'stdout': u'', u'stderr': u"Unable to establish connection to http://cinder-api.openstack.svc.cluster.local:8776/v2/7a413f8546d246059fd3e95f2bf1977f/volumes: HTTPConnectionPool(host='cinder-api.open...
20190827 09:17:09.082 - INFO - +------ END KW: BuiltIn.Create Dictionary (2)
20190827 09:17:09.082 - INFO - +------ START KW: BuiltIn.Run Keyword If [ ${rc} != 0 and ${fail_if_error} == True | FAIL | ${stderr} ]
20190827 09:17:09.084 - INFO - +------- START KW: BuiltIn.Fail [ ${stderr} ]
20190827 09:17:09.084 - FAIL - Unable to establish connection to http://cinder-api.openstack.svc.cluster.local:8776/v2/7a413f8546d246059fd3e95f2bf1977f/volumes: HTTPConnectionPool(host='cinder-api.openstack.svc.cluster.local', port=8776): Max retries exceeded with url: /v2/7a413f8546d246059fd3e95f2bf1977f/volumes (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f805824c1d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
20190827 09:17:09.084 - DEBUG - Traceback (most recent call last):
  None
20190827 09:17:09.084 - INFO - +------- END KW: BuiltIn.Fail (1)
20190827 09:17:09.084 - INFO - +------ END KW: BuiltIn.Run Keyword If (2)
20190827 09:17:09.084 - INFO - +----- END KW: OpenStack.Run OS Command (128597)
20190827 09:17:09.085 - INFO - +---- END KW: OpenStack.Create Volume (128599)

Note that it took around 2 minutes to return this error. I tried later, several times, and the command worked, so, it's indeed intermittent.

I attached a full collect for further debug. Please note that the timestamps on collect logs will be 2 hours earlier (different time zones), so, the logs will be at ~7am.