It appears the job is timing out. We will investigate.


On 16 September 2013 07:47, Andrew Johnson <email address hidden> wrote:

> I can't find anywhere in the message which particular target is in effect
> when this happens.  Any clue?
>
>
> On 16 September 2013 08:20, Andrew McDermott
> <email address hidden>wrote:
>
> > ** Description changed:
> >
> >   The following job:
> >
> > -   https://validation.linaro.org/scheduler/job/72876
> > +   https://validation.linaro.org/scheduler/job/72876
> >
> >   reports back with:
> >
> > -   Dispatcher error or test failure
> > +   Dispatcher error or test failure
> >
> > -   ErrorMessage: None
> > +   ErrorMessage: None
> >
> >   The job itself eventually runs the `jtreg' tests which can be seen in
> >   the output starting at:
> >
> > -   11:50:45 AM INFO: [ACTION-B] lava_test_shell is started with
> > +   11:50:45 AM INFO: [ACTION-B] lava_test_shell is started with
> >   {u'timeout': 180000, u'testdef_repos': [{u'git-repo': u'git...
> >
> >   and the timeout for that test is 50 hours which, as an aside, I'm
> >   suspecting is not long enough for this particular test/job.  I would
> have
> >   expected LAVA to fail with a `timeout error' but I don't see that.
> >
> >   And looking through the test runs I see that test 'lava_test_shell #13'
> >   fails with:
> >
> > +
> https://validation.linaro.org/dashboard/streams/public/team/linaro/pre-
> > + built-vexpress64/bundles/0a75f2ca9c78f77fbadf327f8371ea7e5f7287c5
> > + /344362fe-1dd9-11e3-8b3b-001e671608cd/result/13/
> > +
> >   Lava failed at action lava_test_shell with error:None
> >   Traceback (most recent call last):
> > -   File
> >
> "/srv/lava<email address hidden>
> /lava_dispatcher/job.py",
> > line 254, in run
> > -     action.run(**params)
> > -   File
> >
> "/srv/lava<email address hidden>
> /lava_dispatcher/actions/lava_test_shell.py",
> > line 559, in run
> > -     while self._keep_running(runner, timeout, signal_director):
> > -   File
> >
> "/srv/lava<email address hidden>
> /lava_dispatcher/actions/lava_test_shell.py",
> > line 574, in _keep_running
> > -     idx = runner._connection.expect(patterns, timeout=timeout)
> > -   File
> >
> "/srv/lava<email address hidden>
> /lava_dispatcher/utils.py",
> > line 206, in expect
> > -     return super(logging_spawn, self).expect(*args, **kw)
> > -   File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 1316, in expect
> > -     return self.expect_list(compiled_pattern_list, timeout,
> > searchwindowsize)
> > -   File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 1330, in expect_list
> > -     return self.expect_loop(searcher_re(pattern_list), timeout,
> > searchwindowsize)
> > -   File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 1383, in expect_loop
> > -     c = self.read_nonblocking (self.maxread, timeout)
> > -   File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 820, in read_nonblocking
> > -     r,w,e = self.__select([self.child_fd], [], [], timeout)
> > -   File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 1554, in __select
> > -     return select.select (iwtd, owtd, ewtd, timeout)
> > +   File
> >
> "/srv/lava<email address hidden>
> /lava_dispatcher/job.py",
> > line 254, in run
> > +     action.run(**params)
> > +   File
> >
> "/srv/lava<email address hidden>
> /lava_dispatcher/actions/lava_test_shell.py",
> > line 559, in run
> > +     while self._keep_running(runner, timeout, signal_director):
> > +   File
> >
> "/srv/lava<email address hidden>
> /lava_dispatcher/actions/lava_test_shell.py",
> > line 574, in _keep_running
> > +     idx = runner._connection.expect(patterns, timeout=timeout)
> > +   File
> >
> "/srv/lava<email address hidden>
> /lava_dispatcher/utils.py",
> > line 206, in expect
> > +     return super(logging_spawn, self).expect(*args, **kw)
> > +   File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 1316, in expect
> > +     return self.expect_list(compiled_pattern_list, timeout,
> > searchwindowsize)
> > +   File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 1330, in expect_list
> > +     return self.expect_loop(searcher_re(pattern_list), timeout,
> > searchwindowsize)
> > +   File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 1383, in expect_loop
> > +     c = self.read_nonblocking (self.maxread, timeout)
> > +   File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 820, in read_nonblocking
> > +     r,w,e = self.__select([self.child_fd], [], [], timeout)
> > +   File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 1554, in __select
> > +     return select.select (iwtd, owtd, ewtd, timeout)
> >   KeyboardInterrupt
> >
> >   Where is KeyboardInterrupt coming from?
> >
> >   The other thing I notice is that the job itself seems to run for
> exactly
> >   24 hours (and not 50 hours as specified for the jtreg test case).
> >
> >   From:
> >
> > -
> https://validation.linaro.org/dashboard/streams/public/team/linaro/pre-
> > +
> https://validation.linaro.org/dashboard/streams/public/team/linaro/pre-
> >   built-vexpress64/bundles/0a75f2ca9c78f77fbadf327f8371ea7e5f7287c5/
> >
> > -  Started  at: Sept. 15, 2013, 7:34 a.m.
> > -  Finished at: Sept. 16, 2013, 7:34 a.m.
> > +  Started  at: Sept. 15, 2013, 7:34 a.m.
> > +  Finished at: Sept. 16, 2013, 7:34 a.m.
> >
> > --
> > You received this bug notification because you are a member of Linaro
> > Validation Team, which is subscribed to LAVA Dispatcher.
> > https://bugs.launchpad.net/bugs/1226017
> >
> > Title:
> >   Job errors with "ErrorMessage: None", possibly due to job timeout
> >
> > Status in LAVA Dispatcher:
> >   New
> >
> > Bug description:
> >   The following job:
> >
> >     https://validation.linaro.org/scheduler/job/72876
> >
> >   reports back with:
> >
> >     Dispatcher error or test failure
> >
> >     ErrorMessage: None
> >
> >   The job itself eventually runs the `jtreg' tests which can be seen in
> >   the output starting at:
> >
> >     11:50:45 AM INFO: [ACTION-B] lava_test_shell is started with
> >   {u'timeout': 180000, u'testdef_repos': [{u'git-repo': u'git...
> >
> >   and the timeout for that test is 50 hours which, as an aside, I'm
> >   suspecting is not long enough for this particular test/job.  I would
> have
> >   expected LAVA to fail with a `timeout error' but I don't see that.
> >
> >   And looking through the test runs I see that test 'lava_test_shell #13'
> >   fails with:
> >
> >   https://validation.linaro.org/dashboard/streams/public/team/linaro
> >   /pre-built-vexpress64/bundles/0a75f2ca9c78f77fbadf327f8371ea7e5f7287c5
> >   /344362fe-1dd9-11e3-8b3b-001e671608cd/result/13/
> >
> >   Lava failed at action lava_test_shell with error:None
> >   Traceback (most recent call last):
> >     File
> >
> "/srv/lava<email address hidden>
> /lava_dispatcher/job.py",
> > line 254, in run
> >       action.run(**params)
> >     File
> >
> "/srv/lava<email address hidden>
> /lava_dispatcher/actions/lava_test_shell.py",
> > line 559, in run
> >       while self._keep_running(runner, timeout, signal_director):
> >     File
> >
> "/srv/lava<email address hidden>
> /lava_dispatcher/actions/lava_test_shell.py",
> > line 574, in _keep_running
> >       idx = runner._connection.expect(patterns, timeout=timeout)
> >     File
> >
> "/srv/lava<email address hidden>
> /lava_dispatcher/utils.py",
> > line 206, in expect
> >       return super(logging_spawn, self).expect(*args, **kw)
> >     File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 1316, in expect
> >       return self.expect_list(compiled_pattern_list, timeout,
> > searchwindowsize)
> >     File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 1330, in expect_list
> >       return self.expect_loop(searcher_re(pattern_list), timeout,
> > searchwindowsize)
> >     File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 1383, in expect_loop
> >       c = self.read_nonblocking (self.maxread, timeout)
> >     File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 820, in read_nonblocking
> >       r,w,e = self.__select([self.child_fd], [], [], timeout)
> >     File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> > 1554, in __select
> >       return select.select (iwtd, owtd, ewtd, timeout)
> >   KeyboardInterrupt
> >
> >   Where is KeyboardInterrupt coming from?
> >
> >   The other thing I notice is that the job itself seems to run for
> exactly
> >   24 hours (and not 50 hours as specified for the jtreg test case).
> >
> >   From:
> >
> >    https://validation.linaro.org/dashboard/streams/public/team/linaro
> >   /pre-built-
> >   vexpress64/bundles/0a75f2ca9c78f77fbadf327f8371ea7e5f7287c5/
> >
> >    Started  at: Sept. 15, 2013, 7:34 a.m.
> >    Finished at: Sept. 16, 2013, 7:34 a.m.
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/lava-dispatcher/+bug/1226017/+subscriptions
> >
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Dispatcher.
> https://bugs.launchpad.net/bugs/1226017
>
> Title:
>   Job errors with "ErrorMessage: None", possibly due to job timeout
>
> Status in LAVA Dispatcher:
>   New
>
> Bug description:
>   The following job:
>
>     https://validation.linaro.org/scheduler/job/72876
>
>   reports back with:
>
>     Dispatcher error or test failure
>
>     ErrorMessage: None
>
>   The job itself eventually runs the `jtreg' tests which can be seen in
>   the output starting at:
>
>     11:50:45 AM INFO: [ACTION-B] lava_test_shell is started with
>   {u'timeout': 180000, u'testdef_repos': [{u'git-repo': u'git...
>
>   and the timeout for that test is 50 hours which, as an aside, I'm
>   suspecting is not long enough for this particular test/job.  I would have
>   expected LAVA to fail with a `timeout error' but I don't see that.
>
>   And looking through the test runs I see that test 'lava_test_shell #13'
>   fails with:
>
>   https://validation.linaro.org/dashboard/streams/public/team/linaro
>   /pre-built-vexpress64/bundles/0a75f2ca9c78f77fbadf327f8371ea7e5f7287c5
>   /344362fe-1dd9-11e3-8b3b-001e671608cd/result/13/
>
>   Lava failed at action lava_test_shell with error:None
>   Traceback (most recent call last):
>     File
> "/srv/lava<email address hidden>/lava_dispatcher/job.py",
> line 254, in run
>       action.run(**params)
>     File
> "/srv/lava<email address hidden>/lava_dispatcher/actions/lava_test_shell.py",
> line 559, in run
>       while self._keep_running(runner, timeout, signal_director):
>     File
> "/srv/lava<email address hidden>/lava_dispatcher/actions/lava_test_shell.py",
> line 574, in _keep_running
>       idx = runner._connection.expect(patterns, timeout=timeout)
>     File
> "/srv/lava<email address hidden>/lava_dispatcher/utils.py",
> line 206, in expect
>       return super(logging_spawn, self).expect(*args, **kw)
>     File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> 1316, in expect
>       return self.expect_list(compiled_pattern_list, timeout,
> searchwindowsize)
>     File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> 1330, in expect_list
>       return self.expect_loop(searcher_re(pattern_list), timeout,
> searchwindowsize)
>     File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> 1383, in expect_loop
>       c = self.read_nonblocking (self.maxread, timeout)
>     File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> 820, in read_nonblocking
>       r,w,e = self.__select([self.child_fd], [], [], timeout)
>     File "/srv/lava/.cache/eggs/pexpect-2.4-py2.7.egg/pexpect.py", line
> 1554, in __select
>       return select.select (iwtd, owtd, ewtd, timeout)
>   KeyboardInterrupt
>
>   Where is KeyboardInterrupt coming from?
>
>   The other thing I notice is that the job itself seems to run for exactly
>   24 hours (and not 50 hours as specified for the jtreg test case).
>
>   From:
>
>    https://validation.linaro.org/dashboard/streams/public/team/linaro
>   /pre-built-
>   vexpress64/bundles/0a75f2ca9c78f77fbadf327f8371ea7e5f7287c5/
>
>    Started  at: Sept. 15, 2013, 7:34 a.m.
>    Finished at: Sept. 16, 2013, 7:34 a.m.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-dispatcher/+bug/1226017/+subscriptions
>



-- 
Tyler Baker
Technical Architect, LAVA
Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog