Elk and Kibana Version Incompatible

[Antonio Rosales]

Elasticsearch and Kibana deploy currently results in a Kibana dashboard that is not functional which looks like it may be due to incompatible versions. The deploy was done as follows:
juju deploy elasticsearch
juju deploy logstash-indexer
juju add-relation logstash-indexer elasticsearch:cluster
juju deploy kibana
juju add-relation kibana elasticsearch
juju expose kibana

> as per https://jujucharms.com/kibana/trusty/10

Opening this bug to look for root cause as well as possible suggestions on fixes.

-thanks,
Antonio

[Michael Nelson (michael.nelson)]

Hi Antonio.

TL;DR: you may find if you use the elasticsearch2 charm, things will work.

Long version: I hit a similar issue after the trusty kibana charm updated to kibana4 (which requires Elasticsearch 2 [1]) in r19 [2]. Generally, I don't think we should ever be updating application major versions in a stable charm like that, unless the charm takes explicit care (and testing) so that upgrade-charm manages the upgrade correctly - otherwise it'll be fine for demos where you're deploying anew every time, but will break peoples existing deployments.

Anyway, bottom line for me was I could no longer deploy ELK because the kibana charm was now using kibana4 and needed ES2. I specifically *didn't* update the trusty elasticsearch charm to use ES2 because that will break existing deployments if they upgrade-charm (as the kibana update did). Instead, I've got a separate trusty elasticsearch2 charm [2], which is working with the kibana charm for me.

[1] https://www.elastic.co/guide/en/kibana/current/setup.html
[2] https://code.launchpad.net/~chris.macnaughton/charms/trusty/kibana/version_bump/+merge/276709
[2] https://code.launchpad.net/~onlineservices-charmers/charms/trusty/elasticsearch/elasticsearch2

[Charles Butler (lazypower)]

Michael - so i actually updated the ElasticSearch charm to work for ES2 deployments.. there was just some config file twiddling that needed to happen

cs:~lazypower/trusty/elasticsearch (theres some ACL shenanigans going on here at the moment, sorry - i'll re-ping when its resolved and you can actually SEE the charm)

Are we abandoning the ElasticSearch charms and moving to this es2 charm?

[Antonio Rosales (arosales)]

I also think the issue here that we are seeing in the Kibana dashboard is the logstash-indexer isn't compatible with recent version of Kibana/Elk.

Thus, there are updates in Elk/Kibana we should consider as well as in logstash-indexer and perhaps logstash-agent.

-thanks
Antonio

[Michael Nelson (michael.nelson)]

On Fri, Mar 4, 2016 at 4:20 PM Charles Butler <email address hidden>
wrote:

> Michael - so i actually updated the ElasticSearch charm to work for ES2
> deployments.. there was just some config file twiddling that needed to
> happen
>

Hey Charles. I'm not sure if I'm missing something, or maybe it wasn't
clear what I wrote above, but you can't just update the existing trusty
elasticsearch charm to use elasticsearch 2 - at least I don't see how. I
mean, it's easy to update it and have it working for demos, but it's not
easy to update it and expect current elasticsearch deployments to survive
an upgrade-charm - unless you've done some pretty amazing work (which I'd
be very happy to see :) ).

>
> cs:~lazypower/trusty/elasticsearch (theres some ACL shenanigans going
> on here at the moment, sorry - i'll re-ping when its resolved and you
> can actually SEE the charm)
>

If that's just updating to use elasticsearch 2, I'd recommend the separate
elasticsearch2 charm I linked to above (I'm using it in production).

>
> Are we abandoning the ElasticSearch charms and moving to this es2 charm?
>

No - but it's stable. I'll keep fixing bugs etc on the elasticsearch charm,
but I don't personally see a way to upgrade-charm on an existing deployment
to run Elasticsearch 2. Let me know if you've solved that though...

Thanks!

[Mike McCracken (mikemc)]

logstash-indexer appears to have had only minor attention since about October 2013.

Paul Czarkowski, the original author, has more recently been contributing to a logstash chef recipe here: https://github.com/lusis/chef-logstash

[Michael Nelson (michael.nelson)]

On Fri, Mar 4, 2016 at 4:25 PM Antonio Rosales <email address hidden>
wrote:

> I also think the issue here that we are seeing in the Kibana dashboard
> is the logstash-indexer isn't compatible with recent version of
> Kibana/Elk.
>
> Thus, there are updates in Elk/Kibana we should consider as well as in
> logstash-indexer and perhaps logstash-agent.
>
>
Yes - you'll need logstash2 as well. In the ELK deployment I'm using, I'm
using the following charms to be able to work with the updated kibana charm
(which was updated to use kibana4 without considering related charms):

logstash
 lp:~canonical-is-sa/charms/trusty/logstash/logstash2;revno=61
elasticsearch
lp:~onlineservices-charmers/charms/trusty/elasticsearch/elasticsearch2;revno=45

Hope that helps. Again, I don't think we should just be updating the
existing charms to new major versions without considering the implications.
Yes, demo's will work fine, but you'll possibly break peoples' existing
deployments when they go to upgrade-charm.

[Charles Butler (lazypower)]

Before anyone goes off the rails with logstash, and writing a new charm - a quick note here about that:

http://github.com/juju-solutions/layer-logstash

We've been cycling on this charm for a couple days now and the layer is very minimal, but this layer is a great starting point for a few things:

- using a recent logstash
- Clearly defined interface-layers to boot
- You can build off this to make a mega-logstash-grokk-hub if you have crazy requirements for NGROK filters and services

its stand-alone and doesn't use any of the redis queue features the older logstash charm encapsulated. From what I can tell we're missing 2 things from this layer that existed in the prior logstash charm - which is agent support (now filebeat), and nrpe-external-monitor relationship(s)

-- Now with regards to updating the ES cluster

You're right, it does nothing for migrating the data to the 2.x series. The data buckets format has changed, and i'm not sure if its even something that Elastic has said is supported. With it being a major update, i doubt it, unless there were some kind of data-river plugin to migrate it.

With that being the case, we have an impass on what to do here :( As we cant upgrade existing deploys, and the only thing that *really* needed to happen (in terms of delivery) was just a single template file needed network_host: [__site__, __local__] added to it. The rest just worked.

[Charles Butler (lazypower)]

actually - now that i've looked at it closer, it seems like we can completely do a data migration

https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html

this is listed as supported from the 1.x -> 2.x series, and one we land on the 2.x series it supports rolling upgrades out of the box

[Michael Nelson (michael.nelson)]

That data migration is possible - but not automated in your proposed changes. If you can automate that data migration as part of your branch, test it on realistic deployments, that would be great!

We can't automatically upgrade the major version of our own ES units because of the related services (which talk with a version-dependent elasticsearch python library). We could possibly include both 1.x and 2.x libraries, detect the ES version and use them appropriately, but in our case, it's simpler to redeploy and switch to the new services.