WIN32: IOC scan loops hang

[Freddie Akeroyd]

The IOC scan loops hung on all IOCs on several computers, on investigation this looks to be caused by the monotonic clock wrapping. The values of next and now in dbScan.c / periodicTask() are consistent with GetTickCount() wrapping (which happens every 49.7 days), which I believe lead to epicsEventWaitWithTimeout() being called with a large delay. I was initially confused by the debugger saying a delay of 0 had been passed to epicsEventWaitWithTimeout(), but I cannot see how this is possible (the minimum delay should be penalty) so I assume the debugger is reporting incorrectly due to the code being in release mode.

The epics Monotonic time functions should not normally be using GetTickCount(), however it looks like the logic in the test for whether to use GetPerformanceCounter() is inverted and so GetTickCount() is always used. I will push a patch to fix this, and also to use GetTickCount64() as a fallback instead (which wraps on a much longer timescale).

[rivers (rivers)]

I observed an a Windows areaDetector IOC not updating the ArrayRate_RBV records when it should. It appeared that the 1 second scan was not running. Restarting the IOC fixed the problem. This is consistent with this bug

[mdavidsaver (mdavidsaver)]

Use of GetTickCount64() seems reasonable to me, it does close the door on XP compatibility though.

I guess this is an preview of 2038. So it may be worthwhile to investigate if epicsTimeDiffInSeconds() can be made to handle rollover reasonably.

[Andrew Johnson (anj)]

Hi Freddie, which version of Base are you using here? I thought the code that used the monotonic clock in the internal timer queues never got into a released version, but I just found that dbScan.c is calling epicsTimeGetMonotonic() in its periodic scan code (from commit 8b9ad212c4eb43d) so it seems I was wrong, that code has been in all releases since 7.0.3.1.

[Freddie Akeroyd (freddie-akeroyd)]

I have added GetTickCount64() wrapped in a _WIN32_WINNT check, so it is still possible to compile for Windows XP by defining _WIN32_WINNT as 0x0501 externally

[Freddie Akeroyd (freddie-akeroyd)]

We are running base 7.0.3.1

[mdavidsaver (mdavidsaver)]

> ... I thought the code that used the monotonic clock in the internal timer queues never got into a released version ...

dbScan.c was the one usage I still had confidence in, so I left it. sigh...

epicsTimeDiffInSeconds() only tries to handle rollover >= `ULONG_MAX/2`. Using it with a time source which rolls over sooner is wrong, and likely unfixable.

[mdavidsaver (mdavidsaver)]

And for that matter, use of `ULONG_MAX` seems wrong on targets where `sizeof(long)==8`. eg. if:

> epicsTimeStamp A = {0xffffffff, 0};
> epicsTimeStamp B = {0x00000001, 0};

Then epicsTimeDiffInSeconds(&B, &A) should be 2.0 right?

On Linux, I get -4294967294.0

Down the rabbit hole we go...

[Freddie Akeroyd (freddie-akeroyd)]

QueryPerformanceCounter() is likely to work on Windows 2000 and above, and guaranteed to work on Windows XP and above so I have removed use of either GetTickCount() or GetTickCount64() in the PR

[mdavidsaver (mdavidsaver)]

Cross-linking: epicsTime: fix overflow handling

https://github.com/epics-base/epics-base/pull/92

[Andrew Johnson (anj)]

The problem of scan threads hanging was fixed by reversion of the use of monotonic time for timers, but the underlying issue with the monotonic time support on Windows has now been fixed by Freddie's branch.

[Andrew Johnson (anj)]

My previous comment was incorrect, this bug has been present since 7.0.3.1 and should finally be fixed in the up-coming 7.0.5 release.