Guys, looks like we have some unexpected data in server.cfg about plugin.activemq. We do not this data at all in config.
Also this data affect mcollective behavior. We try to use activemq. And after it no logs record about heartbeat.
I think we have good chance to solve this problem if we remove this section from server.cfg in provisioned nodes.
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.586827 #7605] INFO -- : mcollectived:31:in `<main>' The Marionette Collective @DEVELOPMENT_VERSION@ started logging at info level
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.616589 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 0 to stomp://mcollective@stomp1:6163
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.641641 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp://mcollective@stomp1:6163 failed on attempt 0
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.651914 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 1 to stomp://mcollective@stomp1:6163
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.658442 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp://mcollective@stomp1:6163 failed on attempt 1
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.678674 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 2 to stomp://mcollective@stomp1:6163
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.680422 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp://mcollective@stomp1:6163 failed on attempt 2
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.720662 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 3 to stomp://mcollective@stomp1:6163
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.723436 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp://mcollective@stomp1:6163 failed on attempt 3
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.803709 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 4 to stomp://mcollective@stomp1:6163
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.805674 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp://mcollective@stomp1:6163 failed on attempt 4
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.965977 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 5 to stomp://mcollective@stomp1:6163
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:55:59.968060 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp://mcollective@stomp1:6163 failed on attempt 5
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:56:00.288352 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 6 to stomp://mcollective@stomp1:6163
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:56:00.290455 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp://mcollective@stomp1:6163 failed on attempt 6
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:56:00.930753 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 7 to stomp://mcollective@stomp1:6163
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:56:00.932861 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp://mcollective@stomp1:6163 failed on attempt 7
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:56:02.213156 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 8 to stomp://mcollective@stomp1:6163
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:56:02.215271 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp://mcollective@stomp1:6163 failed on attempt 8
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:56:05.127300 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 9 to stomp://mcollective@stomp1:6163
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:56:05.130081 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp://mcollective@stomp1:6163 failed on attempt 9
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:56:10.250539 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 10 to stomp://mcollective@stomp1:6163
2015-05-25T00:57:33.579989+00:00 debug: I, [2015-05-25T00:56:10.254084 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp://mcollective@stomp1:6163 failed on attempt 10
Guys, looks like we have some unexpected data in server.cfg about plugin.activemq. We do not this data at all in config.
Also this data affect mcollective behavior. We try to use activemq. And after it no logs record about heartbeat.
I think we have good chance to solve this problem if we remove this section from server.cfg in provisioned nodes.
In bootstrap we use this section:
connector = rabbitmq rabbitmq. vhost = mcollective rabbitmq. pool.size = 1 rabbitmq. pool.1. host = 10.109.0.2 rabbitmq. pool.1. port = 61613 rabbitmq. pool.1. user = mcollective rabbitmq. pool.1. password= HPkTOIK3 rabbitmq. heartbeat_ interval = 30
plugin.
plugin.
plugin.
plugin.
plugin.
plugin.
plugin.
But in provisioned env:
connector = rabbitmq activemq. pool.size = 1 activemq. pool.1. host = stomp1 activemq. pool.1. port = 6163 activemq. pool.1. user = mcollective activemq. pool.1. password = marionette
plugin.
plugin.
plugin.
plugin.
plugin.
# Facts e/facts. yaml rabbitmq. pool.1. password = HPkTOIK3 rabbitmq. heartbeat_ interval = 30 rabbitmq. vhost = mcollective rabbitmq. pool.1. host = 10.109.0.2 rabbitmq. pool.1. port = 61613 rabbitmq. pool.size = 1 rabbitmq. pool.1. user = mcollective
factsource = yaml
plugin.yaml = /etc/mcollectiv
plugin.
plugin.
plugin.
direct_addressing = 1
plugin.
plugin.
ttl = 4294957
plugin.
plugin.
Connection log:
2015-05- 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.586827 #7605] INFO -- : mcollectived:31:in `<main>' The Marionette Collective @DEVELOPMENT_ VERSION@ started logging at info level 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.616589 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 0 to stomp:/ /mcollective@ stomp1: 6163 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.641641 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp:/ /mcollective@ stomp1: 6163 failed on attempt 0 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.651914 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 1 to stomp:/ /mcollective@ stomp1: 6163 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.658442 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp:/ /mcollective@ stomp1: 6163 failed on attempt 1 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.678674 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 2 to stomp:/ /mcollective@ stomp1: 6163 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.680422 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp:/ /mcollective@ stomp1: 6163 failed on attempt 2 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.720662 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 3 to stomp:/ /mcollective@ stomp1: 6163 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.723436 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp:/ /mcollective@ stomp1: 6163 failed on attempt 3 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.803709 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 4 to stomp:/ /mcollective@ stomp1: 6163 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.805674 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp:/ /mcollective@ stomp1: 6163 failed on attempt 4 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.965977 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 5 to stomp:/ /mcollective@ stomp1: 6163 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:55: 59.968060 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp:/ /mcollective@ stomp1: 6163 failed on attempt 5 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:56: 00.288352 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 6 to stomp:/ /mcollective@ stomp1: 6163 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:56: 00.290455 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp:/ /mcollective@ stomp1: 6163 failed on attempt 6 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:56: 00.930753 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 7 to stomp:/ /mcollective@ stomp1: 6163 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:56: 00.932861 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp:/ /mcollective@ stomp1: 6163 failed on attempt 7 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:56: 02.213156 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 8 to stomp:/ /mcollective@ stomp1: 6163 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:56: 02.215271 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp:/ /mcollective@ stomp1: 6163 failed on attempt 8 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:56: 05.127300 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 9 to stomp:/ /mcollective@ stomp1: 6163 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:56: 05.130081 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp:/ /mcollective@ stomp1: 6163 failed on attempt 9 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:56: 10.250539 #7616] INFO -- : activemq.rb:96:in `on_connecting' TCP Connection attempt 10 to stomp:/ /mcollective@ stomp1: 6163 25T00:57: 33.579989+ 00:00 debug: I, [2015-05- 25T00:56: 10.254084 #7616] INFO -- : activemq.rb:111:in `on_connectfail' TCP Connection to stomp:/ /mcollective@ stomp1: 6163 failed on attempt 10
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-
2015-05-