IOC stops delivering events

[Jeff Hill]

> On Behalf Of John Sinclair
> Sent: Tuesday, April 28, 2009 8:27 AM
> To: Tech-talk
> Subject: linux ioc problem
>
> I have been having a problem with EPICS IOCs (3.14.9) on linux (fedora
> 2, fedora 8, rhel 4). Every once in awhile ( perhaps one to six months
> ), the IOC stops delivering events. Both caput and caget work in the
> sense that caget retrieves the new value set by caput, but no events are
> delivered.
>
> The IOC never prints any error messages and epicsThreadShowAll indicates
> all OK.
>
> Does anyone else see this behavior?

Original Mantis Bug: mantis-338
    http://www.aps.anl.gov/epics/mantis/view_bug_page.php?f_id=338

[Jeff Hill (johill-lanl)]

I created Mantis 338. I made the assumption that its CA because that’s where the symptoms are, but as this is tracked down we may find that it needs re-assignment elsewhere. For example, device support's not delivering IO interrupt processing events.

When this happens, does the time stamp in the record update?

Run also netstat on the IOCs host and look for a Send-Q which chronically have bytes backed up. This indicates IP delivery issues upstream of the ca server.

Also, make certain that a runaway thread isn’t using all of the CPU and starving out the CA server.

Also, if the above two diagnostics are ok, then send a stack trace for the offending event thread in the ca server and I will have a look. Use the "casr" diagnostic at higher interest levels to find out which thread it is (thread <=> client IP address correlation).

[Jeff Hill (johill-lanl)]

Jeff,

The time stamp does not update.

I'll get back to you with the other diagnostic information whenever I see it happen again.

John

[Jeff Hill (johill-lanl)]

If the time stamp doesn't update in the record then this is probably not an issue with ca.