@Tyler, the suggested fix affects blocking rule logic and should be changed as appropriate:
By design, an iptables blocking rule should persist from the moment then start_rmq_server_app() starts and unless it exits.
Putting unblock in the middle of this function, as my initial patch suggests, is a poor workaround breaking the block/unblock implementation.
The proper one, perhaps, would be to put unblock right into the start, just before to apply the blocking rule as well.
I analyzed the logs attached and found the "moment of truth":
2014-09-24T01:42:30.328672 node-69 ./node-69.lax3.ubiquity.io/rabbitmq-server.log:2014-09-24T01:42:30.328672+01:00 info: INFO: p_rabbitmq-server: start_rmq_server_app(): begin.
(action failed by timeout, there is no "end." event, hence, iptables blocking rule was not removed!)
2014-09-24T01:45:48.730049 node-69 ./node-69.lax3.ubiquity.io/rabbitmq-server.log:2014-09-24T01:45:48.730049+01:00 info: INFO: p_rabbitmq-server: start_rmq_server_app(): begin.
(2nd blocking rule added...)
If we apply the suggested solution - put "safe" unblock prior to the blocking, that would resolve the idempotency issue.
There is an another way - make block and unblock calls idempotent by checking if the rule already exists, but I guess it would complicate the OCF script logic even more and make it more error prone... Anyway, the "safe unblock" solutions works as well.
@Tyler, the suggested fix affects blocking rule logic and should be changed as appropriate: server_ app() starts and unless it exits.
By design, an iptables blocking rule should persist from the moment then start_rmq_
Putting unblock in the middle of this function, as my initial patch suggests, is a poor workaround breaking the block/unblock implementation.
The proper one, perhaps, would be to put unblock right into the start, just before to apply the blocking rule as well.
I analyzed the logs attached and found the "moment of truth": 24T01:42: 30.328672 node-69 ./node- 69.lax3. ubiquity. io/rabbitmq- server. log:2014- 09-24T01: 42:30.328672+ 01:00 info: INFO: p_rabbitmq-server: start_rmq_ server_ app(): begin. 24T01:45: 48.730049 node-69 ./node- 69.lax3. ubiquity. io/rabbitmq- server. log:2014- 09-24T01: 45:48.730049+ 01:00 info: INFO: p_rabbitmq-server: start_rmq_ server_ app(): begin.
2014-09-
(action failed by timeout, there is no "end." event, hence, iptables blocking rule was not removed!)
2014-09-
(2nd blocking rule added...)
If we apply the suggested solution - put "safe" unblock prior to the blocking, that would resolve the idempotency issue.
There is an another way - make block and unblock calls idempotent by checking if the rule already exists, but I guess it would complicate the OCF script logic even more and make it more error prone... Anyway, the "safe unblock" solutions works as well.