Pacemaker kills RabbitMQ

[Andrey Grebennikov]

Fuel 6.1, Ubuntu,
RabbitMQ OCF script updated to the latest from the MU3.

We noticed the instance of RabbitMQ was re-created some time ago, it turned out Pacemaker restarted it.
This is a part of a log from Pacemaker:

Oct 29 20:45:15 [10696] controller3 pacemaker_remoted: notice: operation_finished: p_rabbitmq-server_monitor_30000:29874:stderr [ ^M ]
Oct 29 20:45:15 [10696] controller3 pacemaker_remoted: notice: operation_finished: p_rabbitmq-server_monitor_30000:29874:stderr [ Crash dump was written to: erl_crash.dump^M ]
Oct 29 20:45:15 [10696] controller3 pacemaker_remoted: notice: operation_finished: p_rabbitmq-server_monitor_30000:29874:stderr [ Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,auth,{"Too short cookie string",[{auth,in^M ]
Oct 29 20:45:15 [10699] controller3 crmd: notice: process_lrm_event: Operation p_rabbitmq-server_monitor_30000: unknown error (node=controller3, call=40805, rc=1, cib-update=275893, confirmed=false)
Oct 29 20:45:15 [10699] controller3 crmd: notice: process_lrm_event: GGUTTPLDI003.ebiz.verizon.com-p_rabbitmq-server_monitor_30000:40805 [ \r\nCrash dump was written to: erl_crash.dump\r\nKernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,auth,{"Too short cookie string",[{auth,in\r\n ]

[Andrey Grebennikov (agrebennikov)]

Seems I hit the same bug with erlang cookies once again, but the output is a bit different:

Nov 13 17:10:52 [10699] controller3 crmd: notice: process_lrm_event: controller3-p_rabbitmq-server_monitor_30000:40837 [ \r\nCrash dump was written to: erl_crash.dump\r\nKernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,auth,{{case_clause,eof},[{auth,read_cooki\r\n ]

[Polina Petriuk (ppetriuk)]

Related code change - https://github.com/openstack/fuel-library/commit/8e1bfe6f5c8994800200b0670ab8d4820d9f8c2e

[Alexey Lebedeff (alebedev-a)]

Yes, attached crashdump is also about this bug.

[Polina Petriuk (ppetriuk)]

Backport for MOS6.1 - https://review.openstack.org/#/c/246231/

[Sergii Rizvan (srizvan)]

Backport for MOS 7.0 - https://review.openstack.org/#/c/246232/

[Timur Nurlygayanov (tnurlygayanov)]

Not reproduced on the latest MOS 8.0 ISO on my environment.

[Sergii Rizvan (srizvan)]

Verified on 6.1 on Ubuntu.
Packages:
fuel-ha-util
Version:
6.1.0-6777.1

[Sergii Rizvan (srizvan)]

Verified on 7.0 on Ubuntu.
Packages:
fuel-ha-utils
Version:
7.0.0-7257.1.gita808cd9

[Roman Rufanov (rrufanov)]

please check on 6.0

[Alexey Stupnikov (astupnikov)]

We no longer support MOS5.1, MOS6.0, MOS6.1
We deliver only Critical/Security fixes to MOS7.0, MOS8.0.
We deliver only High/Critical/Security fixes to MOS9.2.