PhantomJS crashing in npm-run-test

[Rob Cresswell]

There is an intermittent (but common) failure on the npm-run-test check. However, when it does fail, the reason is always the same:

2015-11-13 10:12:30.403 | PhantomJS 1.9.8 (Linux): Executed 177 of 291 SUCCESS (0 secs / 3.004 secs)
2015-11-13 10:12:30.404 | PhantomJS 1.9.8 (Linux): Executed 178 of 291 SUCCESS (0 secs / 3.014 secs)
2015-11-13 10:12:30.404 | PhantomJS 1.9.8 (Linux): Executed 179 of 291 SUCCESS (0 secs / 3.014 secs)
2015-11-13 10:12:30.720 | ERROR [launcher]: PhantomJS crashed.
2015-11-13 10:12:30.720 |
2015-11-13 10:12:30.722 | INFO [launcher]: Trying to start PhantomJS again (1/2).
2015-11-13 10:12:40.987 | WARN [PhantomJS 1.9.8 (Linux)]: Disconnected (1 times), because no message in 10000 ms.
2015-11-13 10:12:40.987 | PhantomJS 1.9.8 (Linux): Executed 179 of 291 DISCONNECTED (13.51 secs / 3.014 secs)
2015-11-13 10:12:40.988 |
2015-11-13 10:12:40.988 | ==================== Coverage / Threshold summary =============================
2015-11-13 10:12:40.989 | Statements : 100% ( 0/0 ) Threshold : 86%
2015-11-13 10:12:40.989 | Branches : 100% ( 0/0 ) Threshold : 68%
2015-11-13 10:12:40.989 | Functions : 100% ( 0/0 ) Threshold : 87%
2015-11-13 10:12:40.989 | Lines : 100% ( 0/0 ) Threshold : 87%
2015-11-13 10:12:40.989 | ================================================================================
2015-11-13 10:12:41.131 | Build step 'Execute shell' marked build as failure
2015-11-13 10:12:55.591 | [SCP] Copying console log.
2015-11-13 10:12:56.008 | [SCP] Trying to create /srv/static/logs/61/244461/2/gate/gate-horizon-npm-run-test
2015-11-13 10:12:56.047 | [SCP] Trying to create /srv/static/logs/61/244461/2/gate/gate-horizon-npm-run-test/cf2dade
2015-11-13 10:12:56.104 | [SCP] Connecting to static.openstack.org
2015-11-13 10:12:58.606 | [SCP] No file(s) found: cover/**
2015-11-13 10:12:59.846 | [SCP] ‘cover/**’ doesn’t match anything, but ‘**’ does. Perhaps that’s what you mean?
2015-11-13 10:12:59.888 | [SCP] Connecting to static.openstack.org
2015-11-13 10:13:00.379 | [SCP] No file(s) found: reports/**
2015-11-13 10:13:00.895 | [SCP] ‘reports/**’ doesn’t match anything, but ‘**’ does. Perhaps that’s what you mean?
2015-11-13 10:13:00.897 | Finished: FAILURE

The key part is PhantomJS crashing at test 179 and then failing to restart.

[Rob Cresswell (robcresswell-deactivatedaccount)]

Marking Critical because this is slowing down our gate checks

[Matt Borland (palecrow)]

I suspect that this is a result of the patch:

 https://review.openstack.org/#/c/220585/

The reason for thinking this is because the tests seem to fail around test 180, which is in:

 horizon/static/framework/widgets/metadata/display/display.spec.js

Which was merged on Nov 9. The nature of the change is to specify the "top level" module rather than more explicit, lower-level modules, which could arguably be bad for resource management (esp. if there are memory leaks ).

Steve McLellan researched the logs and this only started occuring soon after that patch merged, later on Nov 9.

I've asked David Lyle and Rob Cresswell as to next steps.

[Steve McLellan (sjmc7)]

Added https://review.openstack.org/245279 to track in elastic-recheck

[Matt Riedemann (mriedem)]

The change in comment 2 sounds like a good guess, according to logstash the races started on 11/9:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%2239mPhantomJS%20crashed%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20build_name:%5C%22gate-horizon-npm-run-test%5C%22

[Rob Cresswell (robcresswell-deactivatedaccount)]

Marking invalid as this no longer seems to be an issue in Horizon.