Failed to reach elasticsearch server due to the firewall

[guillaume thouvenin]

After deploying StackLight 0.8.1 I see the following error in the log:

2016/03/31 12:46:41 Plugin 'elasticsearch_output' error: can't index: HTTP request failed: Post http://10.109.2.3:9200/_bulk: dial tcp 10.109.2.3:9200: connection timed out

The elasticsearch server was not reachable. On the Elasticsearch node I see:

# iptables-save
# Generated by iptables-save v1.4.21 on Thu Mar 31 12:47:53 2016
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [66530:32053741]
-A INPUT -p icmp -m comment --comment "000 accept all icmp requests" -j ACCEPT
-A INPUT -i lo -m comment --comment "001 accept all to lo interface" -j ACCEPT
-A INPUT -m comment --comment "002 accept related established rules" -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m multiport --ports 22 -m comment --comment "020 ssh" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 8083,8086 -m comment --comment "200 influxdb" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 8000 -m comment --comment "201 grafana" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 8001 -m comment --comment "300 nagios cgi" -j ACCEPT
-A INPUT -m comment --comment "999 drop all other requests" -j DROP
COMMIT
# Completed on Thu Mar 31 12:47:53 2016

To solve the problem I rerun the firewall.pp on Elasticsearch and it worked.

# puppet apply --debug /etc/fuel/plugins/elasticsearch_kibana-0.8/puppet/manifests/firewall.pp

This is not happening all the time

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/0.8
Review: https://review.openstack.org/304421

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/304421
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-elasticsearch-kibana/commit/?id=53a0cb22d8f43e15d52f2b3cb4bd8b0f2f6c0c32
Submitter: Jenkins
Branch: stable/0.8

commit 53a0cb22d8f43e15d52f2b3cb4bd8b0f2f6c0c32
Author: Simon Pasquier <email address hidden>
Date: Tue Apr 12 09:22:45 2016 +0200

    Re-execute the firewall task after reboot

    Change-Id: I7027170f3a6d3b80f8122872f099396904213309
    Closes-Bug: #1564394