selenium tests are not running in gate and many are broken

[David Lyle]

running selenium tests locally, I see

Running Horizon application tests
[snip]
Ran 160 tests in 106.527s
FAILED (SKIP=153, errors=3, failures=2)

Running openstack_dashboard tests
[snip]
Ran 1385 tests in 168.336s
FAILED (SKIP=1374, errors=4, failures=5)

Sample output from the gate jobs:

Running Horizon application tests
2015-07-16 12:21:50.680 |
2015-07-16 12:21:57.048 |
2015-07-16 12:21:57.049 | ----------------------------------------------------------------------
2015-07-16 12:21:57.049 | Ran 153 tests in 6.368s
2015-07-16 12:21:57.049 |
2015-07-16 12:21:57.077 | OK (SKIP=153)
2015-07-16 12:21:57.131 | nosetests horizon --nocapture --nologcapture --exclude-dir=horizon/conf/ --exclude-dir=horizon/test/customization --cover-package=horizon --cover-inclusive --all-modules --exclude-dir=openstack_dashboard/test/integration_tests --verbosity=1
2015-07-16 12:21:57.131 | Creating test database for alias 'default'...
2015-07-16 12:21:57.131 | Destroying test database for alias 'default'...
2015-07-16 12:21:57.138 | Running openstack_dashboard tests
2015-07-16 12:22:00.595 | WARNING:root:No local_settings file found.
2015-07-16 12:22:00.993 |
2015-07-16 12:22:09.731 | 
2015-07-16 12:22:09.731 | ----------------------------------------------------------------------
2015-07-16 12:22:09.731 | Ran 1372 tests in 8.737s
2015-07-16 12:22:09.732 |
2015-07-16 12:22:09.759 | OK (SKIP=1371)

[Richard Jones (r1chardj0n3s)]

I've just started to look into this, and there's a bunch of separate reasons why the tests are failing:

horizon.test.tests.selenium_tests:LazyLoadedTabsTests
Three tests here fail because the "select all" user interface element for the checkbox list does not appear in the interface any longer. I have no idea whether: a) it should be appearing, and a bug is causing it not to, b) it shouldn't be appearing and the test is incorrect or c) it should be appearing if the interface is updated somehow to be configured correctly.

horizon.test.tests.selenium_tests.LazyLoadedTabsTests.test_delete_button_is_disabled_on_empty_selection
This test fails because the action button ("delete", but for some reason called "sell puppies" in the test) isn't disabled when there is no selection. It is not clear whether a) this is an actual regression (bug) of the feature, b) the test itself is buggy and not configuring itself correctly to allow the disabling feature to work or c) the feature has been removed and should no longer be tested.

horizon.test.tests.selenium_tests:BrowserTests.test_qunit
These tests all fail because the global "horizon" no longer exists. It is up for question whether we shouldn't just migrate the 14 qunit tests over to the new javascript test runner, or even just delete the tests altogether.

openstack_dashboard.dashboards.identity.projects.tests:SeleniumTests.test_inline_editing_cancel
The browser window displays a 500 error attempting to run this test. Could be an actual error in inline editing.
Also, a mox "error" (ie assertion failure) in tearDown, probably just a knock-on effect of the 500 error. Why the are the mox assertions in *tearDown* :(

openstack_dashboard.dashboards.identity.projects.tests:SeleniumTests.test_inline_editing_update
The browser window displays a 500 error attempting to run this test. Could be an actual error in inline editing.
(also see previous comment about bloody mox)

openstack_dashboard.dashboards.identity.projects.tests:SeleniumTests.test_membership_list_loads_correctly
I believe the test has fallen out of sync with the structure of the page it is testing here.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/214011

[Richard Jones (r1chardj0n3s)]

Recording some progress:

I have identified that the table checkbox bugs are caused by the absence of Javascript in the pages: ironically caused by one of my reorganisation patches (https://review.openstack.org/#/c/186295/) which went under the assumption that no Javascript was needed on the Horizon unit test side (that the Horizon test suite relied solely on HTML).

I've spent some time trying to reconstruct the Javascript loading required to make the page work but there's something I'm missing; just adding horizon.tables.js and its dependencies, and invoking horizon.init() doesn't cause the checkbox to reappear. I'll need to poke into the horizon.tables.js source more tomorrow.

Additionally, we might need to re-think the move of all the Horizon JS loading over to openstack_dashboard - I suspect we'll need to have a split where horizon has a template that knows how to load its JS components, and openstack_dashboard includes that, and adds its own JS components. I recall arguing for this back in Vancouver, but I was assured no JS was needed in the Horizon tests - which technically was incorrect but in a practical sense was correct since the Selenium tests weren't being exercised :)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/214011
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=654b1a61b0874d1fb91b13c2bb697f956e28f756
Submitter: Jenkins
Branch: master

commit 654b1a61b0874d1fb91b13c2bb697f956e28f756
Author: David Lyle <email address hidden>
Date: Mon Aug 17 23:54:29 2015 -0600

    Fixes Selenium tests for openstack_dashboard

    When the javascript autodiscover code was added, it was not added to the
    test settings. This caused javascript files to be missed and numerous
    errors in the Selenium tests.

    This patch fixes the openstack_dashboard part of the Selenium tests
    only.

    Co-Authored-By: Richard Jones<email address hidden>

    Partial-Bug: #1475387
    Change-Id: Ib8d4d6ee8199e3d62aeeda904b2d67a2d029cce8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/215435

[Richard Jones (r1chardj0n3s)]

The tests are now running in the gate (non-voting) as of https://review.openstack.org/#/c/218068/2

[Richard Jones (r1chardj0n3s)]

So, we still have an issue with instability in the selenium tests in the gate. Specifically, these tests can "fail" in the gate at random:

test_create_image_from_file
test_create_image_from_url
test_modal_create_image_from_file
test_modal_create_image_from_url

There's probably some timing issue, since these are checking for an element to be updated in response to an input being set. I am going to investigate further.

[Richard Jones (r1chardj0n3s)]

Actually, I get reliable failures in those tests locally now. I guess something broke between David's fix and now.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/215408
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=7a3d5be9bc82b5c8907b293c6b318d4ae2469960
Submitter: Jenkins
Branch: master

commit 7a3d5be9bc82b5c8907b293c6b318d4ae2469960
Author: Richard Jones <Richard Jones>
Date: Fri Aug 21 12:20:29 2015 +1000

    Fix the selenium LazyLoadedTabsTests

    These tests were broken when we removed Javascript loading
    from the test base.html during the ngReorg. A minimal set
    of Javascript and configuration has been re-added.

    Also note that the loadAngular function from the tech-debt
    utils service has been moved into the only place it's
    used to reduce configuration overhead.

    Partial-Bug: #1475387
    Change-Id: If5472e93d7941608479b430512fc79c1fdb3407c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/156572
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=274921e8b480fe718c3e359e24aa716fc0bdf2f3
Submitter: Jenkins
Branch: master

commit 274921e8b480fe718c3e359e24aa716fc0bdf2f3
Author: Vlad Okhrimenko <email address hidden>
Date: Tue Jul 7 11:16:14 2015 +0300

    Drop QUnit JS tests in favor of Jasmine

    Add `Instance`, `Tables`, `Templates`, `Messages`,
    `Modals` Jasmine tests. Removed corresponding `Qunit`
    javascript tests and dependencies on qunit library.

    Partial-Bug: #1475387
    Implements: blueprint replace-qunit-tests-with-jasmine
    Change-Id: Ia415b65fb7b6299934c99e62a8c1f7225b93b03b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/218722
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=46e396d7c0a70bbb955737c063969a7d80203fc5
Submitter: Jenkins
Branch: master

commit 46e396d7c0a70bbb955737c063969a7d80203fc5
Author: Richard Jones <email address hidden>
Date: Mon Aug 31 14:20:17 2015 +1000

    Correct usage of angular.element

    The $element usage here was incorrect ($element is not callable),
    the code actually wanted to use angular.element()

    This resulted in the image format element not updating correctly,
    and the selenium test failing (correctly).

    Change-Id: I2ffbcc831b3ff42c4f5d1caeaed56e0c2048fa61
    Partial-Bug: 1475387

[OpenStack Infra (hudson-openstack)]

Change abandoned by Richard Jones (<email address hidden>) on branch: master
Review: https://review.openstack.org/215435
Reason: The other patch merged.

[Richard Jones (r1chardj0n3s)]

I think this bug should be closed; there are two specific bugs open on Horizon that address the remaining issues in the Selenium suite.

https://bugs.launchpad.net/horizon/+bug/1494149
https://bugs.launchpad.net/horizon/+bug/1501163