iTCO_wdt: Unexpected close, not stopping watchdog

[johnnyliao]

Running stress-ng test and system will auomatically reboot after 60s. From the dmesg, there is one error shown “iTCO_wdt: Unexpected close, not stopping watchdog”.
However, I already disabled IPMI and the problem still persists. From system BIOS saying, BIOS does not fully control iTCO_wdt, but OS does. By running other stress program, there is no such problem found during same H/W, enviornment or operation.

I suspect stress-ng triggered TCO_EN bit and force the system automatically reboot with specific time (always 60s). Is there any other suggestion or further advice will be highly appreciated.

[Colin Ian King (colin-king)]

Is there any specific stress test that triggers this issue?

[johnnyliao (johnnyliao)]

HI Colin,
Only run stress-ng. Once while it's executed, the system will reboot after done for few cycles.
I can easily duplicate the problem in most of Haswell platform boards.
If I disable "iTCO_wdt" or rmmod iTCO_wdt driver from kernel, the problem goes away.

Sinc TCO is not controlled by BIOS, but OS, our engineering team has no idea on how's going t happen.
The reboot problem is only happening while executing stress-ng test program. I verified other system stress test or burn-in utilities, they did not have such problem at all.

[Colin Ian King (colin-king)]

My guess is that the "stress-ng --sysfs" test is causing the problem. Can you run:

stress-ng --sysfs 1

and see if that triggers it

[johnnyliao (johnnyliao)]

HI Colin,
Yes. From trying the parameter "--sysfs 1", iTCO_wdt is not been triggered and the server does not reboot self anymore. Appreciate the support. Any question is on "sysfs 1", what does this mean for? Any reason for having such problem if I did not add "sysfs"?

[Colin Ian King (colin-king)]

OK, are you running this as a normal user or as root?

[johnnyliao (johnnyliao)]

Hi Colin,
I was operating the command with root account.

[Colin Ian King (colin-king)]

..and my guess is that it does not occur when running it with normal user privileges. Do you mind just checking for that too? Thanks

[johnnyliao (johnnyliao)]

Yes. I create a testing account and did not get server reboot with 30 minutes operation. If I use root account, it will reboot within 2~3 minutes. But, I saw lots of permission denied since I did not grant full permission for the test account.

[Colin Ian King (colin-king)]

Thanks, I'm uploading a fix right now, should land in stress-ng 0.04.06 in a day or so

[Colin Ian King (colin-king)]

Fix: http://kernel.ubuntu.com/git/cking/stress-ng.git/commit/?id=54f3fdc9ba20cce76da0cc9366b891323446165a

[johnnyliao (johnnyliao)]

Thanks for help, Colin. You really give us a big help here. :)