Juju Charms Collection
elasticsearch package

Firewall rules fail on ec2

Bug #1386664 reported by Samuel Cozannet
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
elasticsearch (Juju Charms Collection)
Fix Released
Undecided
Michael Nelson

Bug Description

* Using CLI and GUI, deployment fails when I try to create relations between ES and Kibana or ES and Logstash Indexer.
* This was tested on EC2
* To reproduce, simply deploy the latest versions of ES and Kibana and try to create the relation

The issue is that on ec2, private-address is sent as a private dns entry, whereas the firewall rules require an IP. The attached branch fixes this.

The log looks like:
unit-kibana-0: 2014-10-28 10:56:27 INFO juju-log rest:68: elasticsearch/0 joined
unit-kibana-0: 2014-10-28 10:56:27 INFO rest-relation-changed ++ relation-get cluster-name
unit-kibana-0: 2014-10-28 10:56:28 INFO rest-relation-changed + '[' -z '' ']'
unit-kibana-0: 2014-10-28 10:56:28 INFO rest-relation-changed + exit 0
unit-kibana-0: 2014-10-28 10:56:28 INFO juju-log rest:68: elasticsearch/1 joined
unit-kibana-0: 2014-10-28 10:56:28 INFO rest-relation-changed ++ relation-get cluster-name
unit-kibana-0: 2014-10-28 10:56:28 INFO rest-relation-changed + '[' -z '' ']'
unit-kibana-0: 2014-10-28 10:56:28 INFO rest-relation-changed + exit 0
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined PLAY [localhost] **************************************************************
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined GATHERING FACTS ***************************************************************
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined ok: [localhost]
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined TASK: [Update ufw config to avoid error] **************************************
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined changed: [localhost]
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined TASK: [Reset firewall] ********************************************************
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined changed: [localhost]
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined TASK: [Turn on fire wall with logging.] ***************************************
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined changed: [localhost]
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined TASK: [Open the firewall for all clients] *************************************
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined skipping: [localhost] => (item={'value': {'private-address': 'ip-10-55-75-173.eu-west-1.compute.internal'}, 'key': 'elasticsearch/1'})
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined failed: [localhost] => (item={'value': {'private-address': 'ip-10-84-12-135.eu-west-1.compute.internal', '__unit__': 'kibana/0', '__relid__': 'client:68'}, 'key': 'kibana/0'}) => {"failed": true, "item": {"key": "kibana/0", "value": {"__relid__": "client:68", "__unit__": "kibana/0", "private-address": "ip-10-84-12-135.eu-west-1.compute.internal"}}}
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined msg: ERROR: Bad source address
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined FATAL: all hosts have already failed -- aborting
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined PLAY RECAP ********************************************************************
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined localhost : ok=4 changed=3 unreachable=0 failed=1
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined Traceback (most recent call last):
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined File "/var/lib/juju/agents/unit-elasticsearch-1/charm/hooks/client-relation-joined", line 42, in <module>
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined hooks.execute(sys.argv)
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined File "/var/lib/juju/agents/unit-elasticsearch-1/charm/hooks/charmhelpers/contrib/ansible/__init__.py", line 165, in execute
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined self.playbook_path, tags=[hook_name])
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined File "/var/lib/juju/agents/unit-elasticsearch-1/charm/hooks/charmhelpers/contrib/ansible/__init__.py", line 112, in apply_playbook
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined subprocess.check_call(call)
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined raise CalledProcessError(retcode, cmd)
unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined subprocess.CalledProcessError: Command '['ansible-playbook', '-c', 'local', 'playbook.yaml', '--tags', 'client-relation-joined']' returned non-zero exit status 2
unit-elasticsearch-1: 2014-10-28 10:56:34 ERROR juju.worker.uniter uniter.go:486 hook failed: exit status 1
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined PLAY [localhost] **************************************************************
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined GATHERING FACTS ***************************************************************
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined ok: [localhost]
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined TASK: [Update ufw config to avoid error] **************************************
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined changed: [localhost]
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined TASK: [Reset firewall] ********************************************************
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined changed: [localhost]
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined TASK: [Turn on fire wall with logging.] ***************************************
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined changed: [localhost]
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined TASK: [Open the firewall for all clients] *************************************
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined skipping: [localhost] => (item={'value': {'private-address': 'ip-10-89-140-57.eu-west-1.compute.internal'}, 'key': 'elasticsearch/0'})
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined failed: [localhost] => (item={'value': {'private-address': 'ip-10-84-12-135.eu-west-1.compute.internal', '__unit__': 'kibana/0', '__relid__': 'client:68'}, 'key': 'kibana/0'}) => {"failed": true, "item": {"key": "kibana/0", "value": {"__relid__": "client:68", "__unit__": "kibana/0", "private-address": "ip-10-84-12-135.eu-west-1.compute.internal"}}}
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined msg: ERROR: Bad source address
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined FATAL: all hosts have already failed -- aborting
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined PLAY RECAP ********************************************************************
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined localhost : ok=4 changed=3 unreachable=0 failed=1
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined Traceback (most recent call last):
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined File "/var/lib/juju/agents/unit-elasticsearch-0/charm/hooks/client-relation-joined", line 42, in <module>
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined hooks.execute(sys.argv)
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined File "/var/lib/juju/agents/unit-elasticsearch-0/charm/hooks/charmhelpers/contrib/ansible/__init__.py", line 165, in execute
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined self.playbook_path, tags=[hook_name])
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined File "/var/lib/juju/agents/unit-elasticsearch-0/charm/hooks/charmhelpers/contrib/ansible/__init__.py", line 112, in apply_playbook
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined subprocess.check_call(call)
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined raise CalledProcessError(retcode, cmd)
unit-elasticsearch-0: 2014-10-28 10:56:37 INFO client-relation-joined subprocess.CalledProcessError: Command '['ansible-playbook', '-c', 'local', 'playbook.yaml', '--tags', 'client-relation-joined']' returned non-zero exit status 2
unit-elasticsearch-0: 2014-10-28 10:56:37 ERROR juju.worker.uniter uniter.go:486 hook failed: exit status 1

See original description
Tags: ods

Related branches

lp:~s-matyukevich/charms/trusty/elasticsearch/elasticsearch-dns-bug-fix
Review Queue (community): Needs Fixing (automated testing)
charmers: Pending requested
Diff: 34 lines (+5/-2)
2 files modified
hooks/charmhelpers/core/hookenv.py (+4/-1)
tasks/setup-ufw.yml (+1/-1)
lp:~michael.nelson/charms/trusty/elasticsearch/use-new-charmhelpers
Simon Davy (community): Approve
charmers: Pending requested
Diff: 446 lines (+164/-87)
10 files modified
Makefile (+3/-1)
lookup_plugins/dns.py (+37/-0)
playbook.yaml (+0/-2)
tasks/setup-ufw.yml (+10/-8)
templates/elasticsearch.yml (+2/-6)
tests/00-create-index (+11/-12)
tests/01-config-changes (+3/-3)
tests/02-deploy-three-units (+3/-6)
tests/helpers/__init__.py (+33/-20)
unit_tests/test_hooks.py (+62/-29)
Revision history for this message
Samuel Cozannet (samuel-cozannet) wrote :

jcastro pointed out that this bug https://code.launchpad.net/~s-matyukevich/charms/trusty/elasticsearch/elasticsearch-dns-bug-fix/+merge/239547

refers to a similar situation. Applying the proposed patch fixes the issue.

Marking as dup.

Revision history for this message
Michael Nelson (michael.nelson) wrote :

Thanks Samuel. Yes, as per the related branch, ec2 is passing private dns rather than private IP for private-addresses. That branch will work and is fine for the moment, but I'm currently working on a more general fix that we can land.

Changed in elasticsearch (Juju Charms Collection):
status: New → In Progress
assignee: nobody → Michael Nelson (michael.nelson)
Revision history for this message
Samuel Cozannet (samuel-cozannet) wrote : Re: [Bug 1386664] Re: Relation ES to Kibana (and indexer) fail
Download full text (12.0 KiB)

Do you think this will be fixed by next week? I intend to use ES for a
lightning talk @ODS and I need to know which version I should work with.

Many thanks,
Samuel

On Tue, Oct 28, 2014 at 8:34 PM, Michael Nelson <email address hidden>
wrote:

> Thanks Samuel. Yes, as per the related branch, ec2 is passing private
> dns rather than private IP for private-addresses. That branch will work
> and is fine for the moment, but I'm currently working on a more general
> fix that we can land.
>
> ** Branch linked: lp:~s-matyukevich/charms/trusty/elasticsearch
> /elasticsearch-dns-bug-fix
>
> ** Changed in: elasticsearch (Juju Charms Collection)
> Status: New => In Progress
>
> ** Changed in: elasticsearch (Juju Charms Collection)
> Assignee: (unassigned) => Michael Nelson (michael.nelson)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1386664
>
> Title:
> Relation ES to Kibana (and indexer) fail
>
> Status in “elasticsearch” package in Juju Charms Collection:
> In Progress
>
> Bug description:
> * Using CLI and GUI, deployment fails when I try to create relations
> between ES and Kibana or ES and Logstash Indexer.
> * This was tested on EC2
> * To reproduce, simply deploy the latest versions of ES and Kibana and
> try to create the relation
>
> The log looks like:
> unit-kibana-0: 2014-10-28 10:56:27 INFO juju-log rest:68:
> elasticsearch/0 joined
> unit-kibana-0: 2014-10-28 10:56:27 INFO rest-relation-changed ++
> relation-get cluster-name
> unit-kibana-0: 2014-10-28 10:56:28 INFO rest-relation-changed + '[' -z
> '' ']'
> unit-kibana-0: 2014-10-28 10:56:28 INFO rest-relation-changed + exit 0
> unit-kibana-0: 2014-10-28 10:56:28 INFO juju-log rest:68:
> elasticsearch/1 joined
> unit-kibana-0: 2014-10-28 10:56:28 INFO rest-relation-changed ++
> relation-get cluster-name
> unit-kibana-0: 2014-10-28 10:56:28 INFO rest-relation-changed + '[' -z
> '' ']'
> unit-kibana-0: 2014-10-28 10:56:28 INFO rest-relation-changed + exit 0
> unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
> unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
> PLAY [localhost]
> **************************************************************
> unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
> unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
> GATHERING FACTS
> ***************************************************************
> unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
> ok: [localhost]
> unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
> unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
> TASK: [Update ufw config to avoid error]
> **************************************
> unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
> changed: [localhost]
> unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
> unit-elasticsearch-1: 2014-10-28 10:56:34 INFO client-relation-joined
> TASK: [Reset firewall]
> ********************************************************
> unit-ela...

Revision history for this message
Michael Nelson (michael.nelson) wrote :

Hi Samuel,

Yes, I've done a fix which also updates both the unit-tests and the functional tests so we can verify that it passes from now on on both HP and ec2. I'll try to get that tested and landed in the next day:

https://code.launchpad.net/~michael.nelson/charms/trusty/elasticsearch/use-new-charmhelpers/+merge/239935

Thanks.

summary: - Relation ES to Kibana (and indexer) fail
+ Firewall rules fail on ec2
description: updated
Revision history for this message
Michael Nelson (michael.nelson) wrote :

Verified the attached branch on ec2 as well:

dev-trusty# ~/charms/elasticsearch/trusty/elasticsearch
$ ./tests/02-deploy-three-units
2014-11-03 01:51:18 Starting deployment of amazon
2014-11-03 01:51:20 Deploying services...
2014-11-03 01:51:21 Deploying service elasticsearch using local:trusty/elasticsearch
2014-11-03 01:51:36 Deploying service elasticsearch-sentry using local:trusty/elasticsearch-sentry
2014-11-03 01:51:38 Deploying service relation-sentry using local:trusty/relation-sentry
2014-11-03 01:51:51 Config specifies num units for subordinate: elasticsearch-sentry
2014-11-03 01:58:47 Adding relations...
2014-11-03 01:58:48 Adding relation elasticsearch:juju-info <-> elasticsearch-sentry:juju-info
2014-11-03 02:00:36 Exposing service 'elasticsearch-sentry'
2014-11-03 02:00:37 Exposing service 'relation-sentry'
2014-11-03 02:00:37 Deployment complete in 558.99 seconds
Successfully deployed cluster of 3 units.

dev-trusty# ~/charms/elasticsearch/trusty/elasticsearch

That said, I'll hold off landing this until after ODS (when the auto-testing is hopefully also in place so we get local, hp, ec2 all tested before landing).

Michael Nelson (michael.nelson)
Changed in elasticsearch (Juju Charms Collection):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.