Bug #1487581 “Pacemaker thinks RabbitMQ is up, but no beam proce...” : Bugs : Fuel for OpenStack

Revision history for this message

Ilya Shakhat (shakhat) wrote on 2015-08-21:

#1

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "189"
  build_id: "2015-08-18_17-24-26"
  nailgun_sha: "d0b727cdd0d8e7ce5e17e6ea1306d835bfdfb5e7"
  python-fuelclient_sha: "6ad5e0eb4dbbca6cade3444554606927ecd3f16f"
  fuel-agent_sha: "57145b1d8804389304cd04322ba0fb3dc9d30327"
  fuel-nailgun-agent_sha: "e01693992d7a0304d926b922b43f3b747c35964c"
  astute_sha: "e24ca066bf6160bc1e419aaa5d486cad1aaa937d"
  fuel-library_sha: "7c80eed2119260cc15a700068b9eb20ccc773926"
  fuel-ostf_sha: "235f21b230fea15724d625b2dc44ade0464527e2"
  fuelmain_sha: "c9dad194e82a60bf33060eae635fff867116a9ce"

Revision history for this message

Ilya Shakhat (shakhat) wrote on 2015-08-21:

#2

Download full text (21.1 KiB)

From lrmd.log @ node-4:
2015-08-21T15:33:56.928132+00:00 info: INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:33:57.473084+00:00 info: INFO: p_rabbitmq-server: get_monitor(): get_status() returns 0.
2015-08-21T15:33:57.477217+00:00 info: INFO: p_rabbitmq-server: get_monitor(): also checking if we are master.
2015-08-21T15:33:57.981325+00:00 info: INFO: p_rabbitmq-server: get_monitor(): master attribute is 1
2015-08-21T15:33:58.528780+00:00 info: INFO: p_rabbitmq-server: get_monitor(): checking if rabbit app is running
2015-08-21T15:33:58.540762+00:00 info: INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. checking if we are the part of healthy cluster
2015-08-21T15:33:58.588117+00:00 info: INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. looking for master on node-4.domain.tld
2015-08-21T15:33:58.636903+00:00 info: INFO: p_rabbitmq-server: get_monitor(): fetched master attribute for node-4.domain.tld. attr value is 1
2015-08-21T15:33:58.650643+00:00 info: INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. looking for master on node-5.domain.tld
2015-08-21T15:33:58.702779+00:00 info: INFO: p_rabbitmq-server: get_monitor(): fetched master attribute for node-5.domain.tld. attr value is 0
2015-08-21T15:33:58.715842+00:00 info: INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. master is node-5.domain.tld
2015-08-21T15:33:59.302150+00:00 info: INFO: p_rabbitmq-server: get_monitor(): rabbit app is running and is member of healthy cluster
2015-08-21T15:33:59.308134+00:00 info: INFO: p_rabbitmq-server: get_monitor(): preparing to update master score for node
2015-08-21T15:33:59.392991+00:00 info: INFO: p_rabbitmq-server: get_monitor(): comparing our uptime (21226) with node-5.domain.tld (23354)
2015-08-21T15:34:09.342443+00:00 info: INFO: p_rabbitmq-server: notify: pre-promote begin.
2015-08-21T15:34:09.355550+00:00 info: INFO: p_rabbitmq-server: my_host(): hostlist is: node-2.domain.tld
2015-08-21T15:34:11.976741+00:00 info: INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 0: /usr/sbin/rabbitmqctl list_channels 2>&1 > /dev/null
2015-08-21T15:34:12.677780+00:00 info: INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 0: /usr/sbin/rabbitmqctl -q eval 'rabbit_alarm:get_alarms().'
2015-08-21T15:34:19.960448+00:00 info: INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 0: /usr/sbin/rabbitmqctl -q list_queues memory messages consumer_utilisa
tion
2015-08-21T15:34:21.973623+00:00 info: INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 0: /usr/sbin/rabbitmqctl -q status
2015-08-21T15:34:22.045735+00:00 info: INFO: p_rabbitmq-server: get_monitor(): RabbitMQ is running 231 queues consuming 6m of 24245m total, with 14 queued messages, average consume
r utilization 0.793103
2015-08-21T15:34:22.100066+00:00 info: INFO: p_rabbitmq-server: get_monitor(): RabbitMQ status: [{pid,3691}, {running_applications, [{rabbitmq_management,"RabbitMQ Management Conso
le","3.5.4"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.5.4"}, {webmachine,"webmachine","1.10.3-rmq3.5.4-gite9359c7"}, {mochiweb,"M...

From lrmd.log @ node-4:
2015-08-21T15:33:56.928132+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:33:57.473084+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 0.
2015-08-21T15:33:57.477217+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): also checking if we are master.
2015-08-21T15:33:57.981325+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): master attribute is 1
2015-08-21T15:33:58.528780+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): checking if rabbit app is running
2015-08-21T15:33:58.540762+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. checking if we are the part of healthy cluster
2015-08-21T15:33:58.588117+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. looking for master on node-4.domain.tld
2015-08-21T15:33:58.636903+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): fetched master attribute for node-4.domain.tld. attr value is 1
2015-08-21T15:33:58.650643+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. looking for master on node-5.domain.tld
2015-08-21T15:33:58.702779+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): fetched master attribute for node-5.domain.tld. attr value is 0
2015-08-21T15:33:58.715842+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. master is node-5.domain.tld
2015-08-21T15:33:59.302150+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): rabbit app is running and is member of healthy cluster
2015-08-21T15:33:59.308134+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): preparing to update master score for node
2015-08-21T15:33:59.392991+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): comparing our uptime (21226) with node-5.domain.tld (23354)
2015-08-21T15:34:09.342443+00:00 info:  INFO: p_rabbitmq-server: notify: pre-promote begin.
2015-08-21T15:34:09.355550+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-2.domain.tld
2015-08-21T15:34:11.976741+00:00 info:  INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 0: /usr/sbin/rabbitmqctl list_channels 2>&1 > /dev/null
2015-08-21T15:34:12.677780+00:00 info:  INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 0: /usr/sbin/rabbitmqctl -q eval 'rabbit_alarm:get_alarms().'
2015-08-21T15:34:19.960448+00:00 info:  INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 0: /usr/sbin/rabbitmqctl -q list_queues memory messages consumer_utilisa
tion
2015-08-21T15:34:21.973623+00:00 info:  INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 0: /usr/sbin/rabbitmqctl -q status
2015-08-21T15:34:22.045735+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): RabbitMQ is running 231 queues consuming 6m of 24245m total, with 14 queued messages, average consume
r utilization 0.793103
2015-08-21T15:34:22.100066+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): RabbitMQ status: [{pid,3691}, {running_applications, [{rabbitmq_management,"RabbitMQ Management Conso
le","3.5.4"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.5.4"}, {webmachine,"webmachine","1.10.3-rmq3.5.4-gite9359c7"}, {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.5.4-gi
t680dba8"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.5.4"}, {rabbit,"RabbitMQ","3.5.4"}, {os_mon,"CPO CXC 138 46","2.2.14"}, {amqp_client,"RabbitMQ AMQP Client","3.
5.4"}, {mnesia,"MNESIA CXC 138 12","4.11"}, {inets,"INETS CXC 138 49","5.9.7"}, {xmerl,"XML parser","1.3.5"}, {sasl,"SASL CXC 138 11","2.3.4"}, {stdlib,"ERTS CXC 138 10","1.19.4"}, 
{kernel,"ERTS CXC 138 10","2.16.4"}]}, {os,{unix,linux}}, {erlang_version, "Erlang R16B03 (erts-5.10.4) [source] [64-bit] [async-threads:30] [kernel-poll:true]\n"}, {memory, [{total
,156889264}, {connection_readers,640720}, {connection_writers,0}, {connection_channels,0}, {connection_other,695864}, {queue_procs,14323624}, {queue_slave_procs,245160}, {plugins,18
6776}, {other_proc,14362816}, {mnesia,1019136}, {mgmt_db,2632}, {msg_index,197040}, {other_ets,1497064}, {binary,90176456}, {code,20153630}, {atom,924601}, {other_system,12463745}]}
, {alarms,[]}, {listeners,[{clustering,41055,"::"},{amqp,5673,"::"}]}, {vm_memory_high_watermark,0.4}, {vm_memory_limit,6731640012}, {disk_free_limit,50000000}, {disk_free,859182284
8}, {file_descriptors, [{total_limit,102300}, {total_used,39}, {sockets_limit,92068}, {sockets_used,37}]}, {processes,[{limit,1048576},{used,1380}]}, {run_queue,3}, {uptime,21276}]
2015-08-21T15:34:22.131511+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_monitor function ready to return 0
2015-08-21T15:34:32.415842+00:00 info:  INFO: p_rabbitmq-server: notify: post-promote begin.
2015-08-21T15:34:32.437045+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-2.domain.tld
2015-08-21T15:34:32.461492+00:00 info:  INFO: p_rabbitmq-server: jjj_join: Joining to cluster by node 'node-2.domain.tld'
2015-08-21T15:34:32.484339+00:00 info:  INFO: p_rabbitmq-server: join_to_cluster(): start.
2015-08-21T15:34:32.497369+00:00 info:  INFO: p_rabbitmq-server: join_to_cluster(): Joining to cluster by node 'rabbit@node-2'.
2015-08-21T15:34:33.801753+00:00 info:  INFO: p_rabbitmq-server: join_to_cluster(): rabbitmq app will be stopped.
2015-08-21T15:34:35.246573+00:00 info:  INFO: p_rabbitmq-server: join_to_cluster(): Execute stop_app with timeout: 60
2015-08-21T15:34:36.681173+00:00 info:  INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 0: /usr/sbin/rabbitmqctl stop_app 2>&1 >> "/var/log/rabbitmq/shutdown_lo
g"
2015-08-21T15:34:38.410592+00:00 info:  INFO: p_rabbitmq-server: join_to_cluster(): RMQ-server app stopped succesfully.
2015-08-21T15:34:38.429475+00:00 info:  INFO: p_rabbitmq-server: join_to_cluster(): Execute join_cluster with timeout: 60
2015-08-21T15:34:39.687786+00:00 info:  INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 2: /usr/sbin/rabbitmqctl join_cluster rabbit@node-2
2015-08-21T15:34:39.698147+00:00 err:  ERROR: p_rabbitmq-server: join_to_cluster(): Can't join to cluster by node 'rabbit@node-2'. Stopping.
2015-08-21T15:34:39.721723+00:00 info:  INFO: p_rabbitmq-server: stop: action begin.
2015-08-21T15:34:39.949405+00:00 info:  INFO: p_rabbitmq-server: stop: RMQ-runtime (beam) going to down.
2015-08-21T15:34:39.982881+00:00 info:  INFO: p_rabbitmq-server: stop_server_process(): Execute stop with timeout: 60
2015-08-21T15:34:44.338433+00:00 info:  INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 0: /usr/sbin/rabbitmqctl stop /var/run/rabbitmq/p_pid 2>&1 >> "/var/log/rabbitmq/shutdown_log"
2015-08-21T15:34:44.347005+00:00 info:  INFO: p_rabbitmq-server: stop_server_process(): RMQ-server process (PID=3691) stopped succesfully.
2015-08-21T15:34:52.309854+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:34:53.325479+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:34:53.389076+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:34:54.408703+00:00 info:  INFO: p_rabbitmq-server: stop: action end.
2015-08-21T15:34:55.395476+00:00 info:  INFO: p_rabbitmq-server: stop: RMQ-runtime (beam) not running.
2015-08-21T15:34:55.405769+00:00 err:  ERROR: p_rabbitmq-server: jjj_join: Failed to join the cluster. The mnesia will be reset.
2015-08-21T15:34:56.146797+00:00 warning:  WARNING: p_rabbitmq-server: reset_mnesia(): There is no Beam process running.
2015-08-21T15:34:56.161666+00:00 warning:  WARNING: p_rabbitmq-server: reset_mnesia(): Beam have been killed. Mnesia files appear corrupted and have been removed.
2015-08-21T15:34:56.169836+00:00 info:  INFO: p_rabbitmq-server: notify: post-promote end.
2015-08-21T15:34:56.175305+00:00 err:  ERROR: p_rabbitmq-server: notify: Failed to join the cluster on post-promote. The resource will be restarted.
2015-08-21T15:35:14.361815+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 30
2015-08-21T15:35:15.468019+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:35:15.473575+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:35:24.188684+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:35:25.306435+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:35:25.319716+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:35:55.800016+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:35:56.825209+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:35:56.847662+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:36:14.738701+00:00 info:  INFO: p_rabbitmq-server: notify: post-demote begin.
2015-08-21T15:36:14.747239+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-2.domain.tld
2015-08-21T15:36:15.720879+00:00 info:  INFO: p_rabbitmq-server: notify: post-demote end.
2015-08-21T15:36:27.286993+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:36:28.007972+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:36:28.015545+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:36:33.843955+00:00 info:  INFO: p_rabbitmq-server: notify: post-stop begin.
2015-08-21T15:36:33.849684+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-2.domain.tld
2015-08-21T15:36:34.495705+00:00 info:  INFO: p_rabbitmq-server: notify: post-stop end.
2015-08-21T15:36:58.404925+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:36:58.979669+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 30
2015-08-21T15:36:59.761632+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:36:59.773509+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:37:00.151815+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:37:00.159986+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:37:13.692656+00:00 info:  INFO: p_rabbitmq-server: notify: post-start begin.
2015-08-21T15:37:13.699548+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-2.domain.tld 
2015-08-21T15:37:30.082869+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:37:30.825147+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:37:30.830355+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:38:01.162095+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:38:02.767621+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:38:02.795196+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:38:06.093990+00:00 info:  INFO: p_rabbitmq-server: notify: post-stop begin.
2015-08-21T15:38:06.100243+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-2.domain.tld
2015-08-21T15:38:06.778330+00:00 info:  INFO: p_rabbitmq-server: notify: post-stop end.
2015-08-21T15:38:33.856814+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:38:34.423486+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:38:34.430419+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:38:43.514689+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 30
2015-08-21T15:38:44.094166+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:38:44.100579+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:39:04.726874+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:39:05.194268+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:39:05.196107+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:39:35.783686+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:39:36.505008+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:39:36.510546+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:39:37.457571+00:00 info:  INFO: p_rabbitmq-server: notify: post-start begin.
2015-08-21T15:39:37.472010+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-2.domain.tld 
2015-08-21T15:39:59.918001+00:00 info:  INFO: p_rabbitmq-server: notify: post-demote begin.
2015-08-21T15:39:59.926935+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-5.domain.tld
2015-08-21T15:40:00.656058+00:00 info:  INFO: p_rabbitmq-server: notify: post-demote end.
2015-08-21T15:40:06.830505+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:40:07.565204+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:40:07.581656+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:40:20.514622+00:00 info:  INFO: p_rabbitmq-server: notify: post-stop begin.
2015-08-21T15:40:20.537703+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-5.domain.tld
2015-08-21T15:40:21.332706+00:00 info:  INFO: p_rabbitmq-server: notify: post-stop end.
2015-08-21T15:40:27.581562+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 30
2015-08-21T15:40:28.910757+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:40:28.915661+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:40:36.750045+00:00 info:  INFO: p_rabbitmq-server: notify: post-start begin.
2015-08-21T15:40:36.755697+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-5.domain.tld 
2015-08-21T15:40:36.763579+00:00 warning:  WARNING: p_rabbitmq-server: notify: there are no nodes to join to reported on post-start. The resource will be restarted.
2015-08-21T15:40:36.767948+00:00 info:  INFO: p_rabbitmq-server: notify: post-start end.
2015-08-21T15:40:38.100822+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:40:38.305140+00:00 info:  INFO: p_rabbitmq-server: notify: pre-promote begin.
2015-08-21T15:40:38.318551+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-5.domain.tld
2015-08-21T15:40:39.625341+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:40:39.652294+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:40:40.592508+00:00 info:  INFO: p_rabbitmq-server: notify: pre-promote begin.
2015-08-21T15:40:40.598486+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-5.domain.tld
2015-08-21T15:41:10.031223+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:41:10.649932+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:41:10.656888+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:41:36.769488+00:00 info:  INFO: p_rabbitmq-server: notify: post-promote begin.
2015-08-21T15:41:36.779154+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-5.domain.tld
2015-08-21T15:41:36.789483+00:00 info:  INFO: p_rabbitmq-server: jjj_join: Joining to cluster by node 'node-5.domain.tld'
2015-08-21T15:41:36.801822+00:00 info:  INFO: p_rabbitmq-server: join_to_cluster(): start.
2015-08-21T15:41:36.806179+00:00 info:  INFO: p_rabbitmq-server: join_to_cluster(): Joining to cluster by node 'rabbit@node-5'.
2015-08-21T15:41:37.456339+00:00 info:  INFO: p_rabbitmq-server: join_to_cluster(): Execute join_cluster with timeout: 60
2015-08-21T15:41:38.207654+00:00 info:  INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 2: /usr/sbin/rabbitmqctl join_cluster rabbit@node-5
2015-08-21T15:41:38.216516+00:00 err:  ERROR: p_rabbitmq-server: join_to_cluster(): Can't join to cluster by node 'rabbit@node-5'. Stopping.
2015-08-21T15:41:38.226719+00:00 info:  INFO: p_rabbitmq-server: stop: action begin.
2015-08-21T15:41:38.320775+00:00 info:  INFO: p_rabbitmq-server: stop: RMQ-runtime (beam) going to down.
2015-08-21T15:41:38.333976+00:00 err:  ERROR: p_rabbitmq-server: stop_server_process(): RMQ-server process PIDFILE was not found!
2015-08-21T15:41:39.242466+00:00 info:  INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 2: /usr/sbin/rabbitmqctl stop 2>&1 >> "/var/log/rabbitmq/shutdown_log"
2015-08-21T15:41:39.251926+00:00 err:  ERROR: p_rabbitmq-server: stop_server_process(): Cannot stop RMQ-server process, and cannot kill it by unknown PID! Try to stop it manually!
2015-08-21T15:41:40.925807+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:41:41.636757+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:41:41.643656+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:41:49.298724+00:00 info:  INFO: p_rabbitmq-server: stop: action end.
2015-08-21T15:41:50.012236+00:00 info:  INFO: p_rabbitmq-server: stop: RMQ-runtime (beam) not running.
2015-08-21T15:41:50.021574+00:00 err:  ERROR: p_rabbitmq-server: jjj_join: Failed to join the cluster. The mnesia will be reset.
2015-08-21T15:41:50.778850+00:00 warning:  WARNING: p_rabbitmq-server: reset_mnesia(): There is no Beam process running.
2015-08-21T15:41:50.792642+00:00 warning:  WARNING: p_rabbitmq-server: reset_mnesia(): Beam have been killed. Mnesia files appear corrupted and have been removed.
2015-08-21T15:41:50.801554+00:00 info:  INFO: p_rabbitmq-server: notify: post-promote end.
2015-08-21T15:41:50.810072+00:00 err:  ERROR: p_rabbitmq-server: notify: Failed to join the cluster on post-promote. The resource will be restarted.
2015-08-21T15:42:11.900356+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:42:12.435930+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 30
2015-08-21T15:42:13.394650+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:42:13.407505+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:42:14.212950+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:42:14.223736+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:42:43.877702+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:42:44.759083+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:42:44.808198+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:43:12.903691+00:00 info:  INFO: p_rabbitmq-server: notify: post-stop begin.
2015-08-21T15:43:12.914535+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-2.domain.tld
2015-08-21T15:43:13.574966+00:00 info:  INFO: p_rabbitmq-server: notify: post-stop end.
2015-08-21T15:43:15.497770+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:43:16.573349+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:43:16.586106+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:43:47.047745+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:43:47.742703+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:43:47.778544+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:43:53.321387+00:00 info:  INFO: p_rabbitmq-server: notify: post-start begin.
2015-08-21T15:43:53.330470+00:00 info:  INFO: p_rabbitmq-server: my_host(): hostlist is: node-2.domain.tld 
2015-08-21T15:43:57.648428+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 30
2015-08-21T15:43:58.342829+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:43:58.352307+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:44:18.128634+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:44:18.825573+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.
2015-08-21T15:44:18.842920+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): ensuring this slave does not get promoted.
2015-08-21T15:44:49.132957+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-08-21T15:44:49.809838+00:00 info:  INFO: p_rabbitmq-server: get_monitor(): get_status() returns 7.

the rest 3 lines appear till the end of the log

Revision history for this message

Ilya Shakhat (shakhat) wrote on 2015-08-21:

#3

pacemaker, lrmd and rabbit logs Edit (10.2 MiB, application/octet-stream)

tags:	added: rabbitmq
Changed in fuel:
milestone:	none → 7.0
assignee:	nobody → MOS Oslo (mos-oslo)

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2015-08-24:

#4

Ilya, could you please also add diagnostic snapshot?

Changed in fuel:
importance:	Undecided → High
status:	New → Confirmed

Revision history for this message

Ilya Shakhat (shakhat) wrote on 2015-08-24:

#5

Timur, the snapshot is ~ 700Mb, that's why I picked only necessary files.

Revision history for this message

Dmitry Mescheryakov (dmitrymex) wrote on 2015-08-26:

#6

Ilya, please provide us an env with repro. It is really hard to tell what went wrong just from the logs.

Changed in fuel:
status:	Confirmed → Incomplete
assignee:	MOS Oslo (mos-oslo) → Ilya Shakhat (shakhat)

Revision history for this message

Vitaly Sedelnik (vsedelnik) wrote on 2015-09-02:

#7

Targeted to 7.0-updates and 8.0 as we have not enough time to fix it for 7.0

Changed in fuel:
milestone:	7.0 → 7.0-updates

Revision history for this message

Dmitry Mescheryakov (dmitrymex) wrote on 2015-09-07:

#8

ilya_logs.tgz Edit (6.6 MiB, application/x-tar)

Ilya provided access to the env, but it self-healed itself at that moment. One observation from me is that it seems like log rotation triggered pacemaker's self-healing: 11 minutes after that it can be seen in lrmd.log that RabbitMQ is starting. (The rotation happened at 2015-08-23T00:00:00). See lrmd and pacemaker logs in the attached archive. Unfortunately we don't have pacemaker logs for earlier time - they are already rotated.

The issue seems to be the same as https://bugs.launchpad.net/fuel/+bug/1472230

We need access to an env with the issue being active. Until then I am leaving the issue in incomplete state and reassigning it back to MOS Oslo team.

Changed in fuel:
assignee:	Ilya Shakhat (shakhat) → MOS Oslo (mos-oslo)

Revision history for this message

Dmitry Mescheryakov (dmitrymex) wrote on 2015-09-21:

#9

We have an automated test which reproduces this bug with probability around 70%. It is ha_neutron_tun_scale test.

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2015-09-28:

#10

Folks, can we get logs for these tests failing? I do not see any update here and we merged a set of fixes since the latest report.

Dmitry Pyzhov (dpyzhov) on 2015-10-22

tags:

added: area-mos

Dmitry Pyzhov (dpyzhov) on 2015-11-05

Changed in fuel:
milestone:	7.0-updates → 8.0
no longer affects:	fuel/8.0.x

Revision history for this message

Davanum Srinivas (DIMS) (dims-v) wrote on 2015-11-26:

#11

Need logs per vova's request in #10

Changed in fuel:
status:	Confirmed → Incomplete

Revision history for this message

Dmitry Mescheryakov (dmitrymex) wrote on 2015-12-22:

#12

QA team, could you please reproduce the issue and collect the snapshot after that? Also, please attach /var/log/pacemaker.log* files from each controller as well, as they are not put into the snapshot.

Changed in fuel:
assignee:	MOS Oslo (mos-oslo) → MOS QA Team (mos-qa)

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2015-12-24:

#13

Ok, will do.

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2015-12-24:

#14

Reproduced on my environment, MOS 8.0 ISO #343, 3 controllers, 2 computes, Neutron VLAN.

Steps To Reproduce
1. Login to one contrroller
2. Get PID of beam process:
ps ax | grep beam
3. Kill this process by PID:
kill -9 <ID>
4, Check status of rabbitMQ cluster in pacemaker:

root@node-3:~# ps ax | grep beam
19876 pts/8 S+ 0:00 grep --color=auto beam
root@node-3:~# pcs resource
Clone Set: clone_p_vrouter [p_vrouter]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
vip__management (ocf::fuel:ns_IPaddr2): Started
vip__vrouter_pub (ocf::fuel:ns_IPaddr2): Started
vip__vrouter (ocf::fuel:ns_IPaddr2): Started
vip__public (ocf::fuel:ns_IPaddr2): Started
Clone Set: clone_p_haproxy [p_haproxy]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     p_rabbitmq-server (ocf::fuel:rabbitmq-server): FAILED
     Masters: [ node-2.test.domain.local ]
     Slaves: [ node-5.test.domain.local ]
Clone Set: clone_p_mysql [p_mysql]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_dns [p_dns]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_neutron-l3-agent [p_neutron-l3-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_neutron-dhcp-agent [p_neutron-dhcp-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_heat-engine [p_heat-engine]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Master/Slave Set: master_p_conntrackd [p_conntrackd]
     Masters: [ node-2.test.domain.local ]
     Slaves: [ node-3.test.domain.local node-5.test.domain.local ]
sysinfo_node-2.test.domain.local (ocf::pacemaker:SysInfo): Started
sysinfo_node-5.test.domain.local (ocf::pacemaker:SysInfo): Started
sysinfo_node-3.test.domain.local (ocf::pacemaker:SysInfo): Started
Clone Set: clone_p_ntp [p_ntp]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_ping_vip__public [ping_vip__public]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]

Reproduced on my environment, MOS 8.0 ISO #343, 3 controllers, 2 computes, Neutron VLAN.

Steps To Reproduce
1. Login to one contrroller
2. Get PID of beam process:
ps ax | grep beam
3. Kill this process by PID:
kill -9 <ID>
4, Check status of rabbitMQ cluster in pacemaker:

root@node-3:~# ps ax | grep beam
19876 pts/8    S+     0:00 grep --color=auto beam
root@node-3:~# pcs resource
 Clone Set: clone_p_vrouter [p_vrouter]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 vip__management        (ocf::fuel:ns_IPaddr2): Started 
 vip__vrouter_pub       (ocf::fuel:ns_IPaddr2): Started 
 vip__vrouter   (ocf::fuel:ns_IPaddr2): Started 
 vip__public    (ocf::fuel:ns_IPaddr2): Started 
 Clone Set: clone_p_haproxy [p_haproxy]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     p_rabbitmq-server  (ocf::fuel:rabbitmq-server):    FAILED 
     Masters: [ node-2.test.domain.local ]
     Slaves: [ node-5.test.domain.local ]
 Clone Set: clone_p_mysql [p_mysql]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_p_dns [p_dns]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_p_neutron-l3-agent [p_neutron-l3-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_p_neutron-dhcp-agent [p_neutron-dhcp-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_p_heat-engine [p_heat-engine]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Master/Slave Set: master_p_conntrackd [p_conntrackd]
     Masters: [ node-2.test.domain.local ]
     Slaves: [ node-3.test.domain.local node-5.test.domain.local ]
 sysinfo_node-2.test.domain.local       (ocf::pacemaker:SysInfo):       Started 
 sysinfo_node-5.test.domain.local       (ocf::pacemaker:SysInfo):       Started 
 sysinfo_node-3.test.domain.local       (ocf::pacemaker:SysInfo):       Started 
 Clone Set: clone_p_ntp [p_ntp]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_ping_vip__public [ping_vip__public]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2015-12-24:

#15

after 5 minutes pcs will detect that RammitMQ was closed:

root@node-3:~# pcs resource
Clone Set: clone_p_vrouter [p_vrouter]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
vip__management (ocf::fuel:ns_IPaddr2): Started
vip__vrouter_pub (ocf::fuel:ns_IPaddr2): Started
vip__vrouter (ocf::fuel:ns_IPaddr2): Started
vip__public (ocf::fuel:ns_IPaddr2): Started
Clone Set: clone_p_haproxy [p_haproxy]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     Masters: [ node-2.test.domain.local ]
     Slaves: [ node-5.test.domain.local ]
     Stopped: [ node-3.test.domain.local ]
Clone Set: clone_p_mysql [p_mysql]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_dns [p_dns]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_neutron-l3-agent [p_neutron-l3-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_neutron-dhcp-agent [p_neutron-dhcp-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_p_heat-engine [p_heat-engine]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Master/Slave Set: master_p_conntrackd [p_conntrackd]
     Masters: [ node-2.test.domain.local ]
     Slaves: [ node-3.test.domain.local node-5.test.domain.local ]
sysinfo_node-2.test.domain.local (ocf::pacemaker:SysInfo): Started
sysinfo_node-5.test.domain.local (ocf::pacemaker:SysInfo): Started
sysinfo_node-3.test.domain.local (ocf::pacemaker:SysInfo): Started
Clone Set: clone_p_ntp [p_ntp]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
Clone Set: clone_ping_vip__public [ping_vip__public]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
root@node-3:~# ps ax | grep beam
1210 pts/8 S+ 0:00 grep --color=auto beam

after 5 minutes pcs will detect that RammitMQ was closed:

root@node-3:~# pcs resource     
 Clone Set: clone_p_vrouter [p_vrouter]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 vip__management        (ocf::fuel:ns_IPaddr2): Started 
 vip__vrouter_pub       (ocf::fuel:ns_IPaddr2): Started 
 vip__vrouter   (ocf::fuel:ns_IPaddr2): Started 
 vip__public    (ocf::fuel:ns_IPaddr2): Started 
 Clone Set: clone_p_haproxy [p_haproxy]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     Masters: [ node-2.test.domain.local ]
     Slaves: [ node-5.test.domain.local ]
     Stopped: [ node-3.test.domain.local ]
 Clone Set: clone_p_mysql [p_mysql]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_p_dns [p_dns]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_p_neutron-l3-agent [p_neutron-l3-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_p_neutron-dhcp-agent [p_neutron-dhcp-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_p_heat-engine [p_heat-engine]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Master/Slave Set: master_p_conntrackd [p_conntrackd]
     Masters: [ node-2.test.domain.local ]
     Slaves: [ node-3.test.domain.local node-5.test.domain.local ]
 sysinfo_node-2.test.domain.local       (ocf::pacemaker:SysInfo):       Started 
 sysinfo_node-5.test.domain.local       (ocf::pacemaker:SysInfo):       Started 
 sysinfo_node-3.test.domain.local       (ocf::pacemaker:SysInfo):       Started 
 Clone Set: clone_p_ntp [p_ntp]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
 Clone Set: clone_ping_vip__public [ping_vip__public]
     Started: [ node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local ]
root@node-3:~# ps ax | grep beam
 1210 pts/8    S+     0:00 grep --color=auto beam

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2015-12-24:

#16

So, we can reproduce the bug and in case if beam process will be killed pacemaker detect it only after several minutes, but then pacemaker is able to properly start this service again:

(5-10 minutes after kill -9 <ID of beam process>)

root@node-3:~# ps ax | grep beam
2246 ? Sl 0:06 /usr/lib/erlang/erts-7.1/bin/beam.smp -W w -A 64 -K true -A30 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.6/sbin/../ebin -noshell -noinput -sname rabbit@node-3 -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -rabbit tcp_listeners [{"10.109.1.7",5673}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.6/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@node-3-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@node-3"
5469 pts/8 S+ 0:00 grep --color=auto beam

Revision history for this message

Ivan Ponomarev (ivanzipfer) wrote on 2015-12-30:

#17

Please provide additional information and reopen if required

Changed in fuel:
status:	Incomplete → Invalid

Fuel for OpenStack

Pacemaker thinks RabbitMQ is up, but no beam process running

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches