EPICS Base

Bug #541371 (mantis-342)
Comment #2

Comment 2 for bug 541371

Revision history for this message

Jeff Hill (johill-lanl) wrote on 2009-06-04:

> This caused only one thread to stop so I doubt that the CA server is
> consuming an abnormal number of MBUFs. Each TCP circuit has a hard quota.
> Since NTP time sync is failing also it sounds like a network stack
> issue is the precipitating event.

The thread was in state SUSPEND+I which means its priority was inherited due to it owning a semaphore that some higher priority task wanted. The output from inetstatShow some time later (see below) demonstrates that the network buffers were probably all being used up by requests from CA clients not being handled.

iocsrvid2> inetstatShow
Active Internet PCB Proto Recv-Q Send-Q -------- ----- ------ ------ 1d4c958 TCP 8056 1d4c8d4 TCP 8760 1d4c850 TCP 8760 1d4c7cc TCP 8760 1d4c748 TCP 8760 1d4c6c4 TCP 8760 1d4c640 TCP 8760 1d4c094 TCP 8760 1d4c118 TCP 2484 1d4bf8c TCP 1d4c430 TCP 1d4c010 TCP 1d4c4b4 TCP 1d4c3ac TCP 2560 1d4c19c TCP 1d4ba64 TCP 1d4bd7c UDP 1d4c328 UDP 1d4c2a4 UDP 1d4c220 UDP 1d4bf08 UDP 1d4be00 UDP 1d4bcf8 UDP 1d4bc74 UDP 1d4bbf0 UDP 1d4bb6c UDP 1d4bae8 UDP 1d4b9e0 UDP 1d4b95c UDP 1d4b8d8 UDP value = 1 = 0x1 connections (including servers)
Local Address Foreign Address (state)
----------------- ------------------ -------
0 164.54.3.153.5064 164.54.3.75.45183 ESTABLISHED
0 164.54.3.153.5064 164.54.3.75.45174 ESTABLISHED
0 164.54.3.153.5064 164.54.3.75.45164 ESTABLISHED
0 164.54.3.153.5064 164.54.3.75.45155 ESTABLISHED
0 164.54.3.153.5064 164.54.3.75.45146 ESTABLISHED
0 164.54.3.153.5064 164.54.3.75.45136 ESTABLISHED
0 164.54.3.153.5064 164.54.3.75.45127 ESTABLISHED
0 164.54.3.153.5064 164.54.3.75.45118 ESTABLISHED
0 164.54.3.153.5064 164.54.3.75.45102 CLOSE_WAIT
0 0 164.54.3.153.5064 164.54.1.62.53928 ESTABLISHED
0 0 164.54.3.153.5064 164.54.3.62.1067 ESTABLISHED
0 0 164.54.3.153.5064 164.54.2.123.1034 ESTABLISHED
0 0 164.54.3.153.5064 164.54.2.217.41003 ESTABLISHED
0 164.54.3.153.5064 164.54.1.62.36421 ESTABLISHED
0 0 164.54.3.153.5064 164.54.2.253.52591 ESTABLISHED
0 0 0.0.0.0.5064 0.0.0.0.0 LISTEN
0 0 0.0.0.0.673 0.0.0.0.0
0 0 0.0.0.0.674 0.0.0.0.0
0 0 164.54.3.153.1032 0.0.0.0.0
0 0 127.0.0.1.1031 127.0.0.1.1026
0 0 0.0.0.0.894 0.0.0.0.0
0 0 0.0.0.0.1030 0.0.0.0.0
0 0 0.0.0.0.5065 0.0.0.0.0
0 0 0.0.0.0.1027 0.0.0.0.0
0 0 0.0.0.0.1026 0.0.0.0.0
0 0 0.0.0.0.5064 0.0.0.0.0
0 0 164.54.3.153.1028 164.54.3.255.5065
0 0 0.0.0.0.1025 0.0.0.0.0
0 0 0.0.0.0.898 0.0.0.0.0
0 0 0.0.0.0.18322 0.0.0.0.0

> This caused only one thread to stop so I doubt that the CA server is 
> consuming an abnormal number of MBUFs. Each TCP circuit has a hard quota.
> Since NTP time sync is failing also it sounds like a network stack 
> issue is the precipitating event.

The thread was in state SUSPEND+I which means its priority was inherited due to it owning a semaphore that some higher priority task wanted.  The output from inetstatShow some time later (see below) demonstrates that the network buffers were probably all being used up by requests from CA clients not being handled.

iocsrvid2> inetstatShow
Active Internet connections (including servers)
PCB      Proto Recv-Q Send-Q  Local Address     Foreign Address    (state)
-------- ----- ------ ------  ----------------- ------------------ -------
1d4c958  TCP     8056      0  164.54.3.153.5064 164.54.3.75.45183  ESTABLISHED
1d4c8d4  TCP     8760      0  164.54.3.153.5064 164.54.3.75.45174  ESTABLISHED
1d4c850  TCP     8760      0  164.54.3.153.5064 164.54.3.75.45164  ESTABLISHED
1d4c7cc  TCP     8760      0  164.54.3.153.5064 164.54.3.75.45155  ESTABLISHED
1d4c748  TCP     8760      0  164.54.3.153.5064 164.54.3.75.45146  ESTABLISHED
1d4c6c4  TCP     8760      0  164.54.3.153.5064 164.54.3.75.45136  ESTABLISHED
1d4c640  TCP     8760      0  164.54.3.153.5064 164.54.3.75.45127  ESTABLISHED
1d4c094  TCP     8760      0  164.54.3.153.5064 164.54.3.75.45118  ESTABLISHED
1d4c118  TCP     2484      0  164.54.3.153.5064 164.54.3.75.45102  CLOSE_WAIT
1d4bf8c  TCP        0      0  164.54.3.153.5064 164.54.1.62.53928  ESTABLISHED
1d4c430  TCP        0      0  164.54.3.153.5064 164.54.3.62.1067   ESTABLISHED
1d4c010  TCP        0      0  164.54.3.153.5064 164.54.2.123.1034  ESTABLISHED
1d4c4b4  TCP        0      0  164.54.3.153.5064 164.54.2.217.41003 ESTABLISHED
1d4c3ac  TCP     2560      0  164.54.3.153.5064 164.54.1.62.36421  ESTABLISHED
1d4c19c  TCP        0      0  164.54.3.153.5064 164.54.2.253.52591 ESTABLISHED
1d4ba64  TCP        0      0  0.0.0.0.5064      0.0.0.0.0          LISTEN
1d4bd7c  UDP        0      0  0.0.0.0.673       0.0.0.0.0
1d4c328  UDP        0      0  0.0.0.0.674       0.0.0.0.0
1d4c2a4  UDP        0      0  164.54.3.153.1032 0.0.0.0.0
1d4c220  UDP        0      0  127.0.0.1.1031    127.0.0.1.1026
1d4bf08  UDP        0      0  0.0.0.0.894       0.0.0.0.0
1d4be00  UDP        0      0  0.0.0.0.1030      0.0.0.0.0
1d4bcf8  UDP        0      0  0.0.0.0.5065      0.0.0.0.0
1d4bc74  UDP        0      0  0.0.0.0.1027      0.0.0.0.0
1d4bbf0  UDP        0      0  0.0.0.0.1026      0.0.0.0.0
1d4bb6c  UDP        0      0  0.0.0.0.5064      0.0.0.0.0
1d4bae8  UDP        0      0  164.54.3.153.1028 164.54.3.255.5065
1d4b9e0  UDP        0      0  0.0.0.0.1025      0.0.0.0.0
1d4b95c  UDP        0      0  0.0.0.0.898       0.0.0.0.0
1d4b8d8  UDP        0      0  0.0.0.0.18322     0.0.0.0.0
value = 1 = 0x1