Sometimes, if something goes wrong (unexpected RabbitMQ crash or something similar), the TCP connections are invalidated. However, Nailgun Receiver may not receive this information and may stuck on 'recv' execution.
[root@nailgun ~]# strace -p 44
Process 44 attached
recvfrom(3,
where
3 is a socket connection to RabbitMQ
44 is a PID of receiver process
However, on RabbitMQ side we can see there's no consumers of Nailgun queue (Nailgun queue is used to retrieve messages from Astute), and there's some messages in the queue:
[root@nailgun /]# rabbitmqctl list_queues name consumers messages
Listing queues ...
nailgun 0 7
naily 7 0
naily_service_1196f899-d676-4115-9b84-88df0dca8e48 1 0
naily_service_4c4d3844-2721-4184-ac0d-ee381cceb230 1 0
naily_service_72950288-2750-419d-bc01-0c5af39cb5cb 1 0
naily_service_bd284d98-c028-48ee-a487-e1f14fa0b91a 1 0
naily_service_c94ecd5e-0501-41be-8ce7-f799427fbf7d 1 0
naily_service_daf069d2-bc0a-4781-adb6-467ec559d1f9 1 0
naily_service_ed4a53ce-d198-4e8d-a8db-2648f606da26 1 0
Moreover, the socket on Receiver side has only "FREAD" flag, while it should be at least "O_NONBLOCK" (since it's initially opened this way). This is super strange, and leads to the fact that we hang waiting for the input from socket and will figure out that it's dead only when TCP keepalive will check connection (7200 seconds by default).
Apparently, we must have some mechanism to check it sooner. So we need either enable RMQ heartbits for Receiver, or decrease TCP keepalive timeout to, let's say, 1 minute.
P.S: the issue with hanged receiver has been occurred few times on QA envs.
(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:
actual result
steps to reproduce
For more detailed information on the contents of each of the listed sections see https:/