lava_test_shell connection dropped on highbank and midway when performing network test

Please refer to attachment to find the log for manually run in LAVA hacking session.

Same issue is observed on "midway" test result started from build 600:

https://validation.linaro.org/dashboard/image-reports/linux-linaro-midway

https://validation.linaro.org/scheduler/job/108292/log_file#L_78_0

<LAVA_DISPATCHER>2014-02-01 03:37:26 AM WARNING: lava_test_shell connection dropped

Full log can be found here:

https://validation.linaro.org/scheduler/job/108292/log_file

Hi Dave,

Any progress made on this bug?

Haven't had a chance to look, but this is a known problem with Calxeda nodes. They get into a weird power state and can't be controlled. It usually takes a while and then the node starts responding. In the past I've fixed it with a power status query, so maybe I'll look at adding that into the high bank dispatcher code and trying it out.

https://review.linaro.org/1186

Put up for review

A slight mistake in my original fix - now fixed and tested in staging

Hi Dave and Tyler, looks like change 1186 hasn't fixed this issue:

https://validation.linaro.org/dashboard/image-reports/linux-linaro-highbank

https://validation.linaro.org/dashboard/image-reports/linux-linaro-midway

Test still be dropped on 23 March and 25 March, but that change had been merged on 21 March. Has it been published to the production environment? I remember there was a case that the change could only be deployed in a LAVA upgrade event.

@Tyler, @Dave, would you please change the status of this bug to "Fix Committed"? I ask because this issue still exists on the latest test result in LAVA, for both Highbank and Midway (#626):

https://validation.linaro.org/dashboard/image-reports/linux-linaro-highbank

https://validation.linaro.org/dashboard/image-reports/linux-linaro-midway

So it actually should not be considered as "Fix Released", thanks.

For the test in LAVA happened on 24 April 2014 with the build #630 (same as the build ran on 21 April 2014), the issue has gone:

https://validation.linaro.org/dashboard/streams/private/team/linaro/pre-built-midway/bundles/78303c66ecf2382bf04375505ac7f407042f841b/

But let's wait for 1 more build to see what will happen.

Now Highbank and Midway have been removed from Linux Linaro ubuntu daily test services.