cpuhotplug_07 fails on highbank node

[Milosz Wasilewski]

cpuhotpllug_07 test fails on calxeda highbank node:
http://validation.linaro.org/scheduler/job/74558/log_file#L_7_677

The test is run using linux-linaro kernel and Ubuntu rootfs

Increasing timeout to sync to the userspace doesn't solve the issue.

There was at least one run in the past that had some tests passed:
http://validation.linaro.org/scheduler/job/67850/log_file#L_59_528

[Milosz Wasilewski (mwasilew)]

Sanjay,
Any update here?

[sanjay singh rawat (sanjay-rawat)]

Milosz, sorry no updates yet. Looking into it.

[sanjay singh rawat (sanjay-rawat)]

Tried to test the hotplug functionality with a simple basic script(what test suite does), i can see the udev manager receiving the events. Looking in the issue.

------------------------------------------------------------------------------------------------
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

plug-in cpu1 after 5 sec
KERNEL[7045.361787] offline /devices/system/cpu/cpu1 (cpu)
UDEV [7045.362159] offline /devices/system/cpu/cpu1 (cpu)
plug-out cpu2 after 5 sec
KERNEL[7050.367627] online /devices/system/cpu/cpu1 (cpu)
UDEV [7050.367953] online /devices/system/cpu/cpu1 (cpu)
plug-in cpu2 after 5 sec
KERNEL[7055.371167] offline /devices/system/cpu/cpu2 (cpu)
UDEV [7055.371642] offline /devices/system/cpu/cpu2 (cpu)
plug-out cpu3 after 5 sec
KERNEL[7060.376670] online /devices/system/cpu/cpu2 (cpu)
UDEV [7060.376993] online /devices/system/cpu/cpu2 (cpu)
KERNEL[7065.380108] offline /devices/system/cpu/cpu3 (cpu)
plug-in cpu3 after 5 sec
UDEV [7065.380431] offline /devices/system/cpu/cpu3 (cpu)
KERNEL[7070.385898] online /devices/system/cpu/cpu3 (cpu)
UDEV [7070.386162] online /devices/system/cpu/cpu3 (cpu)

[sanjay singh rawat (sanjay-rawat)]

- It seems repeated hot plug-in/out is failing in Calxeda. I see from cpuhotplug case cpuhotplug_02.sh to 08.sh plug in/out is done in each case once. In the 6th attempt(cpuhotplug_07.sh) the hotplug feature is failing.

- even after having 5sec gap after EACH plug-out and EACH plug-in still it fails. I think it needs to be checked in highbank kernel code.

[sanjay singh rawat (sanjay-rawat)]

The calxeda test board is running on old kernel version, possibly will not be an issue on recent version. Got the information from concerned calxeda enginner.

[Milosz Wasilewski (mwasilew)]

On 9 October 2013 15:24, sanjay singh rawat <email address hidden> wrote:
> The calxeda test board is running on old kernel version, possibly will
> not be an issue on recent version. Got the information from concerned
> calxeda enginner.

So 3.11.0-1 is too old for pm-qa features?
http://validation.linaro.org/scheduler/job/74558/log_file#L_7_91

[sanjay singh rawat (sanjay-rawat)]

Shouldn't be, but 3.5 was i think. Will look for a test-board with 3.11 kernel.

[Milosz Wasilewski (mwasilew)]

Only master image runs 3.5 kernel in LAVA. Master image is not used for testing. hotplug still fails on 3.12 kernel:
http://validation.linaro.org/scheduler/job/78219/log_file#L_59_671

[Milosz Wasilewski (mwasilew)]

This test case fails also on the new FVP model with cpu hotplug feature enabled:
http://validation.linaro.org/dashboard/attachment/508698/view#L236

[sanjay singh rawat (sanjay-rawat)]

The test machine is upgraded to 3.12, looks like the hotplug issue is resolved in the kernel. In the test-suite failing case udevadm is missing some event, beacuse of which failure is returned. Added a change to resolve this issue, sent for review.

[Milosz Wasilewski (mwasilew)]

cpuhotplug_07 still fails on highbank. This time the test hangs and doesn't produce any results. This happens using kernel:

Linux highbank03 3.12.0-1-linaro-highbank #1ubuntu1~ci+131122122634-Ubuntu SMP Fri Nov 22 12:50:18 UTC 201 armv7l armv7l armv7l GNU/Linux

The test case produces results using kernel:
Linux ubuntu 3.12.0-1-linaro-highbank #1ubuntu1~ci+131017101439-Ubuntu SMP Thu Oct 17 10:37:15 UTC 201 armv7l armv7l armv7l GNU/Linux

[Botao (botao-sun)]

This test passed in LAVA since build 540, 29 November 2013:

https://validation.linaro.org/dashboard/streams/private/team/linaro/pre-built-highbank/bundles/46432ab05dd9de5ca2858cf05eac4e07d234d42a/78560d88-a5f2-48a7-8fda-d328aae2e658/

https://validation.linaro.org/dashboard/attachment/577682/view#L170

However, we still have many other failures as show up in above test result page.

[Milosz Wasilewski (mwasilew)]

Things are back to 'normal' now. The cpuhotplug_07 doesn't hang any more on highbank, but fails again:
https://validation.linaro.org/dashboard/attachment/598402/view#L173

[Botao (botao-sun)]

Since 9 January 2013, build 558, the test case number in total has been reduced from 178 to 65. Test "cpufreq_07.0" has been skipped, though we still have other failures there:

https://validation.linaro.org/dashboard/streams/private/team/linaro/pre-built-highbank/bundles/2818e5cead5cddcf2716a93f94c16004a1f5d051/44c0bb77-c544-47de-86a0-38e8e80b0f54/

[Botao (botao-sun)]

All 173 power management test cases have passed on build #630, second ran on 24 April 2014:

https://validation.linaro.org/dashboard/streams/private/team/linaro/pre-built-highbank/bundles/23d2cd7c767aec4a7663d7130db2346b7a2dc35d/

But let's wait one more build to see the status.

[Milosz Wasilewski (mwasilew)]

The problem was a bit different there. First we had hang on cpuhotplug_07. Than we had runaway 'skip' results. Now the results look sane:
https://validation.linaro.org/dashboard/streams/private/team/linaro/pre-built-highbank/bundles/23d2cd7c767aec4a7663d7130db2346b7a2dc35d/b86297fb-6567-417b-afdd-890844d9235f/?search=cpuhotplug_07&length=25

Checking the definition used: https://validation.linaro.org/dashboard/attachment/769791/view
It seems that the 0.4.9 release was used. So the fix was successful. I propose to close this bug and open a new one if something else goes wrong.

[Botao (botao-sun)]

Now Highbank and Midway have been removed from Linux Linaro ubuntu daily test services.