Improve reliability of gate's npm-run-test

[Matt Borland]

The npm-run-test job in the gate has shown itself to be less reliable lately. In particular it tends to hang until the timeout, after successfully completing the tests.

There have been similar failures in the past due to a variety of reasons. Most likely there's a memory problem with the reloading of modules over and over again in Chrome. One factor that had affected this was the loading of modules that were too 'high' in the hierarchy. For example, instead of using 'horizon.app.core.images' a test would just load 'horizon.app.core' which would load ALL dependent modules, then have to destroy them.

Ideally we localize the tests loading only the modules needed.

Other options for assisting with the tests would be to reduce the number of dependencies within app.core, such as moving resource registrations out since they are not really needed as part of the core registrations (where common features are placed, esp. APIs).

This bug will remain in effect until npm-run-test appears to be fully stabilized.

[Richard Jones (r1chardj0n3s)]

Do we have a citation for the memory / module loading assertion?

[Matt Borland (palecrow)]

I'd noticed it in a patch Cindy introduced awhile ago, where she introduced
the exact opposite...generalizing the modules in tests to use higher level
modules...and that patch sustained a noticeably high rate of failure
compared with its contemporaries.

It's an anecdotal theory at best...I'm just sad this is a problem to begin
with...it's not as though the failures are intermittent in any other
environment. :P

Up for any suggestions, it seems fated that my patch approved last Monday
will never go thru...it makes me sad.

Thanks!

Matt

On Tuesday, May 31, 2016, Richard Jones <email address hidden> wrote:

> Do we have a citation for the memory / module loading assertion?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1587698
>
> Title:
> Improve reliability of gate's npm-run-test
>
> Status in OpenStack Dashboard (Horizon):
> In Progress
>
> Bug description:
> The npm-run-test job in the gate has shown itself to be less reliable
> lately. In particular it tends to hang until the timeout, after
> successfully completing the tests.
>
> There have been similar failures in the past due to a variety of
> reasons. Most likely there's a memory problem with the reloading of
> modules over and over again in Chrome. One factor that had affected
> this was the loading of modules that were too 'high' in the hierarchy.
> For example, instead of using 'horizon.app.core.images' a test would
> just load 'horizon.app.core' which would load ALL dependent modules,
> then have to destroy them.
>
> Ideally we localize the tests loading only the modules needed.
>
> Other options for assisting with the tests would be to reduce the
> number of dependencies within app.core, such as moving resource
> registrations out since they are not really needed as part of the core
> registrations (where common features are placed, esp. APIs).
>
> This bug will remain in effect until npm-run-test appears to be fully
> stabilized.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/horizon/+bug/1587698/+subscriptions
>

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Borland (<email address hidden>) on branch: master
Review: https://review.openstack.org/323570

[Matt Borland (palecrow)]

Krotscheck has found the problem is due to low open file ulimit settings. He has a patch here:

https://review.openstack.org/#/c/324735/

[Matt Borland (palecrow)]

The above patch merged. With luck this will close this ticket.

[Matt Borland (palecrow)]

So far, the prognosis is that the above change to up the ulimit on open files *helped* somewhat with our failure rate, but hasn't reduced it to zero by any stretch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/346944

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Borland (<email address hidden>) on branch: master
Review: https://review.openstack.org/346944

[Rob Cresswell (robcresswell-deactivatedaccount)]

I think we can close this now. It doesn't appear to be a current issue.