Central logging not working, nothing happens when I enable it

[Martin Ananda Boeker]

Env: 2023.1

I am looking at the page https://docs.openstack.org/kolla-ansible/2023.1/reference/logging-and-monitoring/central-logging-guide.html trying to enable central logging.

I added 'enable_central_logging: true' to kolla/globals.yml and ran service deploy and it doesn't seem like it did anything. There's nothing listening on kolla_vip:5601 (or 9200), and no new containers were deployed.

From various searches I've now added this whole block to kolla/globals.yml:

enable_central_logging: true
enable_elasticsearch: true
enable_elasticsearch_curator: true
enable_osprofiler: true
osprofiler_backend: "elasticsearch"

And this block to inventory/groups:

[kibana]
# Empty group to provide declaration of kibana group.

[kibana:children]
controllers

[elasticsearch]
# Empty group to provide declaration of elasticsearch group.

[elasticsearch:children]
controllers

[opensearch]
# Empty group to provide declaration of opensearch group.

[opensearch:children]
controllers

And this block to overcloud-components.yml:

[kibana:children]
control

[elasticsearch:children]
control

[opensearch:children]
control

(Note that in overcloud-components the group is called control instead of controllers, not sure why, but it works for telegraf).

So far there's still no changes.. No new containers running, and nothing listening on the listed ports.

[Martin Ananda Boeker (mboeker)]

So once I added those three blocks to overcloud-components it did finally deploy the opensearch docker containers.

[Will Szumski (willjs)]

Don't forget that kayobe functions as a kolla config generator so groups in your kayobe inventory won't automaitcally be mapped through to kolla. This can be changed with kolla_overcloud_inventory_top_level_group_map:

https://github.com/openstack/kayobe/blob/e460099e1a3e2cc2899942d4fd48b72d8ed5616f/etc/kayobe/kolla.yml#L188

and the default:

https://github.com/openstack/kayobe/blob/master/ansible/inventory/group_vars/all/kolla#L359

Although, in this case it is a bit strange that you needed a customisation to overcloud-components... Just to note, Elasticsearch was removed in antelope:

https://github.com/openstack/kolla-ansible/commit/bc5e4621430c63577119275d96ad850a0ca38391

Which version are you are deploying? It would be recommended to deploy Opensearch rather than elasticsearch/kibana. There also could be a version mismatch between your kayobe config and kayobe.
Generally we recommend using the latest stable release and to make sure the versions of each component (kayobe, kolla-ansible, kayobe-config, kolla) all match.

It is also preferable to use the kolla flags in etc/kayobe/kolla.yml where available:

https://github.com/openstack/kayobe/blob/e460099e1a3e2cc2899942d4fd48b72d8ed5616f/etc/kayobe/kolla.yml#L275

[Martin Ananda Boeker (mboeker)]

I've managed to get OpenSearch working and it seems to be doing everything I needed from ELK (Elasticsearch/Kibana/Logstash). Unfortunately now I'm not really sure what changes exactly made it happen, but I'm guessing it was enable_central_logging and then either "[opensearch:children] controllers" in inventory/groups or "[opensearch:children] control" in overcloud-components.yml.

In general when I can't find something in docs, I start searching through venvs/kolla-ansible, that's where I found those variables and group definitions.

I'm quite sure everything in my deployment is set to 2023.1 across the board, I've been diligent about specifying the version everywhere.

[Will Szumski (willjs)]

I guess this could be another deficiency is the docs. I don't see how this would fail if you are using the builtin templates and variables under etc/kayobe/kolla.yml.