Rabbitmq crashes when tokens are rapidly requested

[Eugene Nikanorov]

Fuel 8.x

When radosgw works with several clients uploading small files, rabbitmq may crash.

The following crash is available in /var/lib/rabbitmq/erl_crash.dmp

Slogan: Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{k
System version: Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:24:24] [async-threads:10] [kernel-poll:false]
Compiled: Tue Jan 26 04:34:23 2016
Taints:
Atoms: 4809

[Denis Meltsaykin (dmeltsaykin)]

Unfortunately this crash is not informative by itself, could you please provide more logs? A diagnostic snapshot would be even better.

[Dmitry Sutyagin (dsutyagin)]

Asked Anatolii to fetch and provide data to you

[Anatolii Neliubin (aneliubin)]

Folks, Diag snapshot is useless. @Denis, let me know what exact logs is necessary

Just in short. RabbitMQ does not work well under load:
root@<censored>:~# grep -c 7-Feb-2018::19:47:22 /var/log/rabbitmq/rabbit@messaging-<censored>.log
0
root@<censored>:~# grep -c 7-Feb-2018::19:47:23 /var/log/rabbitmq/rabbit@messaging-<censored>.log
297
root@<censored>:~# grep -c 7-Feb-2018::19:47:24 /var/log/rabbitmq/rabbit@messaging-<censored>.log
2210
root@<censored>:~# grep -c 7-Feb-2018::19:47:25 /var/log/rabbitmq/rabbit@messaging-<censored>.log
0
root@<censored>:~# grep -c 7-Feb-2018::19:47:26 /var/log/rabbitmq/rabbit@messaging-<censored>.log
0
root@<censored>:~#

[Denis Meltsaykin (dmeltsaykin)]

If you say it's under load then you should provide at least rabbimqctl report. Even better would also be screenshots from management plugin with messages rate graphics.