OSTF hangs on selecting HTTP proxy for cloud

[Artem Panchenko]

Fuel version info (7.0 build #301): http://paste.openstack.org/show/473891/

If some of HTTP proxies on controllers work incorrectly (e.g. Apache accepts connections, but doesn't reply on them) then OSTF hangs forever waiting for keystone client initialization:

https://github.com/stackforge/fuel-ostf/blob/master/fuel_health/config.py#L762-L767

2015-09-24 14:39:21 INFO (config) Online controllers ips is [u'10.109.0.5', u'10.109.0.6', u'10.109.0.3']
2015-09-24 14:39:21 INFO (config) Online compute ips is [u'10.109.0.7', u'10.109.0.4']
2015-09-24 14:39:21 INFO (config) COMPUTES IPS [u'10.109.0.7', u'10.109.0.4']
2015-09-24 14:39:21 INFO (config) parse node cluster successful
2015-09-24 14:39:21 INFO (config) parse network configuration successful
2015-09-24 14:39:21 INFO (config) set endpoints successful
2015-09-24 14:39:21 WARNING (config) Try to check proxy on 10.109.0.5
2015-09-24 14:39:21 INFO (connectionpool) Starting new HTTP connection (1): 10.109.0.5
2015-09-24 14:39:21 WARNING (config) Try to check proxy on 10.109.0.6
2015-09-24 14:39:21 INFO (connectionpool) Starting new HTTP connection (1): 10.109.0.6
2015-09-24 14:39:21 INFO (config) parse network configuration successful
2015-09-24 14:39:21 INFO (config) set endpoints successful
2015-09-24 14:39:21 WARNING (config) Try to check proxy on 10.109.0.5
2015-09-24 14:39:21 INFO (connectionpool) Starting new HTTP connection (1): 10.109.0.5
2015-09-24 14:39:22 WARNING (config) Try to check proxy on 10.109.0.6
2015-09-24 14:39:22 INFO (connectionpool) Starting new HTTP connection (1): 10.109.0.6
2015-09-24 14:39:22 WARNING (config) Try to check proxy on 10.109.0.3
2015-09-24 14:39:22 INFO (connectionpool) Starting new HTTP connection (1): 10.109.0.3
2015-09-24 14:39:22 WARNING (config) Try to check proxy on 10.109.0.3
2015-09-24 14:39:22 INFO (connectionpool) Starting new HTTP connection (1): 10.109.0.3
2015-09-24 14:39:22 WARNING (config) Try to check proxy on 10.109.0.6
2015-09-24 14:39:22 INFO (connectionpool) Starting new HTTP connection (1): 10.109.0.6
2015-09-24 14:39:22 WARNING (config) Try to check proxy on 10.109.0.6
2015-09-24 14:39:22 INFO (connectionpool) Starting new HTTP connection (1): 10.109.0.6
2015-09-24 14:39:22 WARNING (config) Try to check proxy on 10.109.0.3
2015-09-24 14:39:22 INFO (connectionpool) Starting new HTTP connection (1): 10.109.0.3
2015-09-24 14:39:22 WARNING (config) Try to check proxy on 10.109.0.3
2015-09-24 14:39:22 INFO (connectionpool) Starting new HTTP connection (1): 10.109.0.3
...
2015-09-24 14:39:24 DEBUG (auth_token) Authenticating user token
2015-09-24 14:39:27 DEBUG (auth_token) Authenticating user token
2015-09-24 14:39:30 DEBUG (auth_token) Authenticating user token
2015-09-24 14:39:33 DEBUG (auth_token) Authenticating user token
2015-09-24 14:39:36 DEBUG (auth_token) Authenticating user token
2015-09-24 14:39:39 DEBUG (auth_token) Authenticating user token
2015-09-24 14:39:42 DEBUG (auth_token) Authenticating user token
2015-09-24 14:39:45 DEBUG (auth_token) Authenticating user token
2015-09-24 14:39:48 DEBUG (auth_token) Authenticating user token
2015-09-24 14:39:51 DEBUG (auth_token) Authenticating user token
2015-09-24 14:39:54 DEBUG (auth_token) Authenticating user token
2015-09-24 14:39:57 DEBUG (auth_token) Authenticating user token
2015-09-24 14:40:00 DEBUG (auth_token) Authenticating user token
2015-09-24 14:40:03 DEBUG (auth_token) Authenticating user token
2015-09-24 14:40:06 DEBUG (auth_token) Authenticating user token

We need to add connection timeout in order to avoid health check hang.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/227355

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/227355
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=0e7bc3d9e08f9b83f49b532613748790e4b4d84b
Submitter: Jenkins
Branch: master

commit 0e7bc3d9e08f9b83f49b532613748790e4b4d84b
Author: Artem Panchenko <email address hidden>
Date: Thu Sep 24 17:59:45 2015 +0300

    Add timeout for keystone client in proxy checker

    Add 10 seconds timeout for Keystone client initialization
    during HTTP proxy check, because otherwise it could hang
    forever.
    Also check that find_proxy returns IP address before
    using it.

    Change-Id: I4fabe0f5fee3d2acc3e7e2387ac858b2c56493c7
    Closes-bug: #1499415

[Nastya Urlapova (aurlapova)]

I vote to fix it in 7.o-updates also.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/237925

[Tatyanka (tatyana-leontovich)]

verified on 8.0-167 kilo for 8.0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237925
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=1d0fe68fd84608d9a7821d199d12c2bd157fdcf3
Submitter: Jenkins
Branch: stable/7.0

commit 1d0fe68fd84608d9a7821d199d12c2bd157fdcf3
Author: Artem Panchenko <email address hidden>
Date: Thu Sep 24 17:59:45 2015 +0300

    [stable/7.0]Add timeout for keystone client in proxy checker

    Add 10 seconds timeout for Keystone client initialization
    during HTTP proxy check, because otherwise it could hang
    forever.
    Also check that find_proxy returns IP address before
    using it.

    Change-Id: I4fabe0f5fee3d2acc3e7e2387ac858b2c56493c7
    Closes-bug: #1499415
    (cherry picked from commit 0e7bc3d9e08f9b83f49b532613748790e4b4d84b)

[Denis Puchkin (dpuchkin)]

Verified on 7.0

with fuel-ostf-7.0.0-825.2.git1d0fe68