Integration tests sometime fail with "WebDriverException: Message: The browser appears to have exited before we could connect."

[Julie Pichon]

For example on test here ( http://logs.openstack.org/04/148204/4/check/gate-horizon-dsvm-integration/5b55890/console.html )

2015-01-19 10:04:02.134 | 2015-01-19 10:04:02.118 | ERROR: openstack_dashboard.test.integration_tests.tests.test_flavors.TestFlavors.test_flavor_create
2015-01-19 10:04:02.136 | 2015-01-19 10:04:02.119 | ----------------------------------------------------------------------
2015-01-19 10:04:02.168 | 2015-01-19 10:04:02.121 | _StringException: Traceback (most recent call last):
2015-01-19 10:04:02.168 | 2015-01-19 10:04:02.122 | File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 72, in setUp
2015-01-19 10:04:02.169 | 2015-01-19 10:04:02.124 | super(AdminTestCase, self).setUp()
2015-01-19 10:04:02.169 | 2015-01-19 10:04:02.125 | File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 33, in setUp
2015-01-19 10:04:02.169 | 2015-01-19 10:04:02.127 | self.driver = webdriver.WebDriverWrapper()
2015-01-19 10:04:02.169 | 2015-01-19 10:04:02.128 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 59, in __init__
2015-01-19 10:04:02.170 | 2015-01-19 10:04:02.130 | self.binary, timeout),
2015-01-19 10:04:02.170 | 2015-01-19 10:04:02.131 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
2015-01-19 10:04:02.170 | 2015-01-19 10:04:02.132 | self.binary.launch_browser(self.profile)
2015-01-19 10:04:02.171 | 2015-01-19 10:04:02.134 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 66, in launch_browser
2015-01-19 10:04:02.171 | 2015-01-19 10:04:02.135 | self._wait_until_connectable()
2015-01-19 10:04:02.171 | 2015-01-19 10:04:02.137 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 100, in _wait_until_connectable
2015-01-19 10:04:02.171 | 2015-01-19 10:04:02.138 | raise WebDriverException("The browser appears to have exited "
2015-01-19 10:04:02.172 | 2015-01-19 10:04:02.140 | WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.

3 tests with that error in http://logs.openstack.org/47/147047/5/gate/gate-horizon-dsvm-integration/2e044be/console.html :

2015-01-19 14:09:49.343 | 2015-01-19 14:09:49.309 | ======================================================================
2015-01-19 14:09:49.343 | 2015-01-19 14:09:49.310 | ERROR: openstack_dashboard.test.integration_tests.tests.test_keypair.TestKeypair.test_keypair
2015-01-19 14:09:49.343 | 2015-01-19 14:09:49.311 | ----------------------------------------------------------------------
2015-01-19 14:09:49.343 | 2015-01-19 14:09:49.313 | _StringException: Traceback (most recent call last):
2015-01-19 14:09:49.343 | 2015-01-19 14:09:49.314 | File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 59, in setUp
2015-01-19 14:09:49.344 | 2015-01-19 14:09:49.315 | super(TestCase, self).setUp()
2015-01-19 14:09:49.344 | 2015-01-19 14:09:49.316 | File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 33, in setUp
2015-01-19 14:09:49.344 | 2015-01-19 14:09:49.318 | self.driver = webdriver.WebDriverWrapper()
2015-01-19 14:09:49.344 | 2015-01-19 14:09:49.319 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 59, in __init__
2015-01-19 14:09:49.344 | 2015-01-19 14:09:49.320 | self.binary, timeout),
2015-01-19 14:09:49.345 | 2015-01-19 14:09:49.321 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
2015-01-19 14:09:49.345 | 2015-01-19 14:09:49.323 | self.binary.launch_browser(self.profile)
2015-01-19 14:09:49.345 | 2015-01-19 14:09:49.324 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 66, in launch_browser
2015-01-19 14:09:49.345 | 2015-01-19 14:09:49.325 | self._wait_until_connectable()
2015-01-19 14:09:49.345 | 2015-01-19 14:09:49.327 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 100, in _wait_until_connectable
2015-01-19 14:09:49.345 | 2015-01-19 14:09:49.329 | raise WebDriverException("The browser appears to have exited "
2015-01-19 14:09:49.346 | 2015-01-19 14:09:49.330 | WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.

Maybe we should also set that log_file attribute and add it to the list of collected logs to help debug gate job failures.

[Julie Pichon (jpichon)]

15 hits in the last 7 days: http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9uYW1lOlwiZ2F0ZS1ob3Jpem9uLWRzdm0taW50ZWdyYXRpb25cIiBBTkQgdGFnczpcImNvbnNvbGVcIiBBTkQgbWVzc2FnZTpcIldlYkRyaXZlckV4Y2VwdGlvbjogTWVzc2FnZTogVGhlIGJyb3dzZXIgYXBwZWFycyB0byBoYXZlIGV4aXRlZCBiZWZvcmUgd2UgY291bGQgY29ubmVjdC5cIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyMTY4MTE3NzgzMiwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

[Tomáš Nováčik (tnovacik-6)]

I believe this is caused by xvfb memory bug http://blog.jeffterrace.com/2012/07/xvfb-memory-leak-workaround.html. Supplying -noreset parameter should fix this issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/148300

[Julie Pichon (jpichon)]

Thanks for the link and taking this one on Tomas! Seems worth doing if there are no other side-effects.

[Lin Hua Cheng (lin-hua-cheng)]

Interesting, good find! Thanks!

[Julie Pichon (jpichon)]

Raising to medium since jobs failing because of this cause patches to fail the gate, and there's been 52 more hits over the last 24h.

[Julie Pichon (jpichon)]

ER query was submitted at https://review.openstack.org/#/c/148293/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/148300
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=2ebb804c05b776b31e65f814df5226287cd939b2
Submitter: Jenkins
Branch: master

commit 2ebb804c05b776b31e65f814df5226287cd939b2
Author: Tomáš Nováčik <email address hidden>
Date: Mon Jan 19 16:51:50 2015 +0100

    Adding -ac and -noreset arguments to Xvfb

    There is outstanding memory leak issue with Xvfb, that should be fixed
    by -noreset argument.

    -ac disables X access control - which i believe is not necessary for
                                    testing

    Partially implements blueprint: selenium-integration-testing
    Closes-Bug: #1412469

    Change-Id: I5f603266f891728bb8e83331f5a4e57799f07f79

[Julie Pichon (jpichon)]

0 hits so far since this merged, looks like that was the correct fix! Well debugged Tomas, thanks :)