Fail of grenade Heat job due to ssh connection issues

[Sergey Kraynev]

Grenade job sporadically fails with ssh Timeout error:

example: http://logs.openstack.org/32/209732/2/gate/gate-grenade-dsvm-heat/8ce2be5/console.html#_2015-08-06_07_54_57_178

There are couple tracebacks from this fail:

2015-08-06 07:54:57.153 | 2015-08-06 07:54:47,031 19348 ERROR [tempest_lib.common.ssh] Failed to establish authenticated ssh connection to cirros@172.24.5.85 after 16 attempts
2015-08-06 07:54:57.153 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh Traceback (most recent call last):
2015-08-06 07:54:57.154 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh File "/opt/stack/new/tempest/.tox/smoke/local/lib/python2.7/site-packages/tempest_lib/common/ssh.py", line 75, in _get_ssh_connection
2015-08-06 07:54:57.154 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh timeout=self.channel_timeout, pkey=self.pkey)
2015-08-06 07:54:57.154 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh File "/opt/stack/new/tempest/.tox/smoke/local/lib/python2.7/site-packages/paramiko/client.py", line 251, in connect
2015-08-06 07:54:57.154 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh retry_on_signal(lambda: sock.connect(addr))
2015-08-06 07:54:57.154 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh File "/opt/stack/new/tempest/.tox/smoke/local/lib/python2.7/site-packages/paramiko/util.py", line 270, in retry_on_signal
2015-08-06 07:54:57.154 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh return function()
2015-08-06 07:54:57.154 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh File "/opt/stack/new/tempest/.tox/smoke/local/lib/python2.7/site-packages/paramiko/client.py", line 251, in <lambda>
2015-08-06 07:54:57.154 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh retry_on_signal(lambda: sock.connect(addr))
2015-08-06 07:54:57.176 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh File "/usr/lib/python2.7/socket.py", line 224, in meth
2015-08-06 07:54:57.176 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh return getattr(self._sock,name)(*args)
2015-08-06 07:54:57.176 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh error: [Errno 113] No route to host
2015-08-06 07:54:57.177 | 2015-08-06 07:54:47.031 19348 ERROR tempest_lib.common.ssh
2015-08-06 07:54:57.177 | 2015-08-06 07:54:47,034 19348 ERROR [tempest.scenario.manager]

(TestVolumeBootPatternV2:test_volume_boot_pattern) Initializing SSH connection to 172.24.5.85 failed. Error: Connection to the 172.24.5.85 via SSH timed out.
2015-08-06 07:54:57.177 | User: cirros, Password: None
2015-08-06 07:54:57.177 | 2015-08-06 07:54:47.034 19348 ERROR tempest.scenario.manager Traceback (most recent call last):
2015-08-06 07:54:57.177 | 2015-08-06 07:54:47.034 19348 ERROR tempest.scenario.manager File "tempest/scenario/manager.py", line 315, in get_remote_client
2015-08-06 07:54:57.177 | 2015-08-06 07:54:47.034 19348 ERROR tempest.scenario.manager linux_client.validate_authentication()
2015-08-06 07:54:57.177 | 2015-08-06 07:54:47.034 19348 ERROR tempest.scenario.manager File "tempest/common/utils/linux/remote_client.py", line 62, in validate_authentication
2015-08-06 07:54:57.177 | 2015-08-06 07:54:47.034 19348 ERROR tempest.scenario.manager self.ssh_client.test_connection_auth()
2015-08-06 07:54:57.177 | 2015-08-06 07:54:47.034 19348 ERROR tempest.scenario.manager File "/opt/stack/new/tempest/.tox/smoke/local/lib/python2.7/site-packages/tempest_lib/common/ssh.py", line 151, in test_connection_auth
2015-08-06 07:54:57.178 | 2015-08-06 07:54:47.034 19348 ERROR tempest.scenario.manager connection = self._get_ssh_connection()
2015-08-06 07:54:57.178 | 2015-08-06 07:54:47.034 19348 ERROR tempest.scenario.manager File "/opt/stack/new/tempest/.tox/smoke/local/lib/python2.7/site-packages/tempest_lib/common/ssh.py", line 87, in _get_ssh_connection
2015-08-06 07:54:57.178 | 2015-08-06 07:54:47.034 19348 ERROR tempest.scenario.manager password=self.password)
2015-08-06 07:54:57.178 | 2015-08-06 07:54:47.034 19348 ERROR tempest.scenario.manager SSHTimeout: Connection to the 172.24.5.85 via SSH timed out.
2015-08-06 07:54:57.178 | 2015-08-06 07:54:47.034 19348 ERROR tempest.scenario.manager User: cirros, Password: None

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/209890

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/209890
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=ba4a7026c7c5e9558770354c6d58d2f60ff46e75
Submitter: Jenkins
Branch: master

commit ba4a7026c7c5e9558770354c6d58d2f60ff46e75
Author: Sirushti Murugesan <email address hidden>
Date: Thu Aug 6 16:56:09 2015 +0530

    Grenade: Run select heat_integrationtests instead of tempest smoke

    Only a few tests are run because these tests only verify the
    sanity of the build rather than doing the upgrade tests itself.

    Closes-Bug: #1482152
    partial blueprint upgrade-tests

    Change-Id: I8fed19bd25cc3a62c03c35afafe2b662551dbd4e

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/heat 6.0.0.0b1 development milestone.