integration tests: socket.error: Connection refused frequently happens in selenium remote driver

[Akihiro Motoki]

socket.error: [Errno 111] Connection refused frequently occurs recently in selenium remote_connection.
This is seen in various tests. It seems selenium fails to connect a browser.

For example, http://logs.openstack.org/14/303514/55/check/gate-horizon-dsvm-integration-current-ubuntu-xenial/af1bef2/console.html#_2016-09-21_10_56_45_019822

66 hits in last 7 days in 311 integration test run (dsvm-integration-current and dsvm-integration-deprecated)
The failure rate is 21%. The rate is high enough.

logstash query:
message:"socket.error: [Errno 111] Connection refused"
AND filename:"console.html"
AND (build_name:"gate-horizon-dsvm-integration-deprecated-ubuntu-xenial" OR build_name:"gate-horizon-dsvm-integration-current-ubuntu-xenial")

======================================================================
ERROR: openstack_dashboard.test.integration_tests.tests.test_user_settings.TestDashboardHelp.test_dashboard_help_redirection
----------------------------------------------------------------------
_StringException: Traceback (most recent call last):
  File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 299, in setUp
    super(TestCase, self).setUp()
  File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 162, in setUp
    desired_capabilities=desired_capabilities
  File "/opt/stack/new/horizon/horizon/test/firefox_binary.py", line 74, in __init__
    desired_capabilities, proxy)
  File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 85, in __init__
    keep_alive=True)
  File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 90, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 177, in start_session
    response = self.execute(Command.NEW_SESSION, capabilities)
  File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 234, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 401, in execute
    return self._request(command_info[0], url, body=data)
  File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 432, in _request
    self._conn.request(method, parsed_url.path, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1057, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1097, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 897, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 859, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 575, in create_connection
    raise err
socket.error: [Errno 111] Connection refused

[Ying Zuo (yingzuo)]

Looks like the same issue reported in bug 1626292. https://bugs.launchpad.net/horizon/+bug/1626292

[Akihiro Motoki (amotoki)]

Query for elastic-recheck https://review.openstack.org/374980

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/375042

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/375258

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akihiro Motoki (<email address hidden>) on branch: master
Review: https://review.openstack.org/375042

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/375258
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=b89c1abfa5c3cc0b2bf942a46cc868341cff3b31
Submitter: Jenkins
Branch: master

commit b89c1abfa5c3cc0b2bf942a46cc868341cff3b31
Author: Akihiro Motoki <email address hidden>
Date: Fri Sep 23 03:03:46 2016 +0900

    Multiple integration test fixes

    This patch represents the following patches, combined:

    ---

    integration tests: retry WebDriver instantiation when conn refused
      https://review.openstack.org/#/c/375042/

    We see connection refused which leads to integration test failure randomly.
    This seems to come from high load of test infra. This commit retries
    WebDriver instantiation when connection refused is returned.

    There is no confident this workaround works, but it would be great
    if this reduces the number of failures. Let's try.

    ---

    integration test: ensure IPv4 address is extracted
      https://review.openstack.org/#/c/374826/

    In test_floatingip_associate_disassociate, IPv6 address is passed
    to Floating IP associate API. This commit ensures to pick up IPv4
    address of an instance.

    Change-Id: Iba71d1e17e252527a82cd53c34dca35723367b28
    Co-Authored-By: Akihiro Motoki <email address hidden>
    Partial-Bug: #1626643
    Closes-Bug: #1626536

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/375406

[Timur Sufiev (tsufiev-x)]

There are more patterns that seem to be closely related to start up problems.

First one is

2016-09-23 12:11:39.698828 | 2016-09-23 12:11:39.698 | _StringException: Traceback (most recent call last):
2016-09-23 12:11:39.700537 | 2016-09-23 12:11:39.700 | File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 299, in setUp
2016-09-23 12:11:39.703591 | 2016-09-23 12:11:39.702 | super(TestCase, self).setUp()
2016-09-23 12:11:39.705580 | 2016-09-23 12:11:39.704 | File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 162, in setUp
2016-09-23 12:11:39.707337 | 2016-09-23 12:11:39.706 | desired_capabilities=desired_capabilities
2016-09-23 12:11:39.709230 | 2016-09-23 12:11:39.708 | File "/opt/stack/new/horizon/horizon/test/firefox_binary.py", line 87, in __init__
2016-09-23 12:11:39.712891 | 2016-09-23 12:11:39.712 | desired_capabilities, proxy)
2016-09-23 12:11:39.715117 | 2016-09-23 12:11:39.714 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 60, in __init__
2016-09-23 12:11:39.717565 | 2016-09-23 12:11:39.717 | capabilities.update(self.options.to_capabilities())
2016-09-23 12:11:39.719683 | 2016-09-23 12:11:39.719 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/firefox/options.py", line 93, in to_capabilities
2016-09-23 12:11:39.722995 | 2016-09-23 12:11:39.721 | desired["firefox_profile"] = self._profile.encoded
2016-09-23 12:11:39.725622 | 2016-09-23 12:11:39.724 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 175, in encoded
2016-09-23 12:11:39.729062 | 2016-09-23 12:11:39.728 | zipped.write(filename, filename[path_root:])
2016-09-23 12:11:39.731219 | 2016-09-23 12:11:39.730 | File "/usr/lib/python2.7/zipfile.py", line 1123, in write
2016-09-23 12:11:39.733183 | 2016-09-23 12:11:39.732 | st = os.stat(filename)
2016-09-23 12:11:39.735987 | 2016-09-23 12:11:39.735 | OSError: [Errno 2] No such file or directory: '/tmp/tmpUJU0SW/lock'

[Timur Sufiev (tsufiev-x)]

On the second thought, I decided to file a separate bug 1627010

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/375406
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=a33ac87de1015ee3756813688224c2886ef14eeb
Submitter: Jenkins
Branch: stable/newton

commit a33ac87de1015ee3756813688224c2886ef14eeb
Author: Akihiro Motoki <email address hidden>
Date: Fri Sep 23 03:03:46 2016 +0900

    Multiple integration test fixes

    This patch represents the following patches, combined:

    ---

    integration tests: retry WebDriver instantiation when conn refused
      https://review.openstack.org/#/c/375042/

    We see connection refused which leads to integration test failure randomly.
    This seems to come from high load of test infra. This commit retries
    WebDriver instantiation when connection refused is returned.

    There is no confident this workaround works, but it would be great
    if this reduces the number of failures. Let's try.

    ---

    integration test: ensure IPv4 address is extracted
      https://review.openstack.org/#/c/374826/

    In test_floatingip_associate_disassociate, IPv6 address is passed
    to Floating IP associate API. This commit ensures to pick up IPv4
    address of an instance.

    Change-Id: Iba71d1e17e252527a82cd53c34dca35723367b28
    Co-Authored-By: Akihiro Motoki <email address hidden>
    Partial-Bug: #1626643
    Closes-Bug: #1626536
    (cherry picked from commit b89c1abfa5c3cc0b2bf942a46cc868341cff3b31)

[Akihiro Motoki (amotoki)]

we now disable the integration tests in our gate, so it is not high priority at all.

[Akihiro Motoki (amotoki)]

The situation looks unknown.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Dashboard (Horizon) because there has been no activity for 60 days.]