Kayobe inventory host and group definitions not being correctly reflected in kolla inventory

[Martin Ananda Boeker]

Kayobe 14.1, Kolla 16.2, Kolla-ansible 16.3

As I started getting the version info I realized that this may explain the problem, I didn't realize that my kayobe and kolla/kolla-ansible versions were so different..

Here's the issue I'm seeing. In my etc/kayobe/inventory/groups I have added this block:

[telegraf]
# Empty group to provide declaration of telegraf group.

[telegraf:children]
controllers

[ceph-mgr]
# Empty group to provide declaration of ceph-mgr group.

In my etc/kayobe/inventory/hosts I have this:

[ceph-mgr]
storage0
storage1

When I run kayobe commands, the inventory that is generated under etc/kolla/inventory/overcloud/hosts contains this:

[telegraf:children]
compute
control
monitoring
network
storage

and there is no sign of ceph-mgr.

[Martin Ananda Boeker (mboeker)]

Just noticed that https://docs.openstack.org/releasenotes/kayobe/2023.1.html, kolla/2023.1.html, and kolla-ansible/2023.1.html roughly match the versions I'm using, so they appear to be in harmony.

[Martin Ananda Boeker (mboeker)]

Another similar example: in kayobe/inventory/groups I have specified that prometheus-node-exporter and prometheus-cadvisor should *exclude* the storage group:

[prometheus-node-exporter:children]
monitoring
control
compute
network

[prometheus-cadvisor:children]
monitoring
control
compute
network

But when I run kayobe overcloud container image pull, the kolla/inventory/overcloud/hosts list puts storage back in there, and the prometheus docker images are deployed to the hosts in the storage group.

I am trying to get Kayobe to either a) deploy the proper prometheus configuration for ceph to the storage nodes (the group ceph-mgr did not work), or b) don't deploy prometheus to storage nodes at all and let me do that separately when I deploy ceph, and I will then figure out how to make them talk to each other.

Currently neither a) or b) are working for me.

[Will Szumski (willjs)]

Are you overriding the inventory templates with:

# Custom overcloud inventory containing a mapping from components to top level
# groups.
#kolla_overcloud_inventory_custom_components:
kolla_overcloud_inventory_custom_components: "{{ lookup('template', kayobe_config_path ~ '/kolla/overcloud-components.j2') }}"

# Custom overcloud inventory containing a mapping from services to components.
kolla_overcloud_inventory_custom_services: "{{ lookup('template', kayobe_config_path ~ '/kolla/overcloud-services.j2') }}"

These variables are in etc/kayobe/kolla.yml and you take the originals from:

https://github.com/openstack/kayobe/tree/master/ansible/roles/kolla-ansible/templates

making sure to use the correct branch for your release. That is the most powerful way of modifying the generated kolla inventory, although you have to remember to update these files when you upgrade. We are looking at ways to more easily customise the inventory (so you don't have to copy the whole file).

[Martin Ananda Boeker (mboeker)]

Interesting, have not even looked at that variable.

I am avoiding modifying anything outside of etc/kayobe. Are you saying that if I create etc/kayobe/kolla/overcloud-components with just these contents:

[telegraf]
controllers

and set kolla_overcloud_inventory_custom_components: "{{ lookup('template', kayobe_config_path ~ '/kolla/overcloud-components') }}" that that would overwrite the telegraf group that's being generated in etc/kolla/inventory/overcloud/hosts? Also, I don't see the [prometheus-node-exporter] or children in those files, how would I restrict which groups are in there?

Since posting my original message I've also discovered that the tasks that push the prometheus_ceph_mgr_exporter to all hosts in the ceph-mgr group no longer exist in the current versions of kayobe and kolla-ansible, does that mean that my goal "a)" above is no longer possible?

Thank you for your great input!

[Will Szumski (willjs)]

> Are you saying that if I create etc/kayobe/kolla/overcloud-components with just these contents

No, make sure you copy the full file and make sure it is the version from the release you are using.

Node exporter is here: https://github.com/openstack/kayobe/blob/595fe87e863f893e9e79b634843f5c85d21e14a3/ansible/roles/kolla-ansible/templates/overcloud-services.j2#L453

You will need to use: `kolla_overcloud_inventory_custom_services` for that one.

prometheus_ceph_mgr_exporter is just a group that contains your ceph mgrs. You enable node exporter using ceph and we just scrape it:

https://docs.ceph.com/en/quincy/mgr/prometheus/#enabling-prometheus-output

So its just used to define the scrape targets.

[Martin Ananda Boeker (mboeker)]

I've managed to sort out the ceph stuff using prometheus_ceph_mgr_exporter, it was very simple in the end, I think when I was testing I was doing something wrong.

However now I am wondering about setting prometheus_ceph_mgr_exporter_endpoints dynamically. I already have the port set to 9283, is there a way to include the whole [storage] group in that list? I can generate the list dynamically but it would be really nice if there was a built-in way to do that.

[Martin Ananda Boeker (mboeker)]

I'm still working on the issue of only deploying telegraf to controllers.

I've copied overcloud-components and modified the telegraf children, will test soon. Will the contents of etc/kayobe/kolla/overcloud-components take precedence over etc/kayobe/inventory?

[Martin Ananda Boeker (mboeker)]

Hmm.. Seems that implementing custom components has had some unexpected side-effects.

I copied overcloud-components.j2 from my local venvs/kayobe to etc/kayobe/kolla/overcloud-components, and removed all but controllers from [telegraf]
Then in kolla.yml I set kolla_overcloud_inventory_custom_components: "{{ lookup('template', kayobe_config_path ~ '/kolla/overcloud-components') }}"

That clearly took, but Kayobe wasn't very happy with it:

TASK [service-images-pull : neutron | Pull images] *************************************************************************************************************************************************************************************
Friday 09 February 2024 12:16:25 +0000 (0:00:00.122) 0:00:07.705 *******
fatal: [compute0]: FAILED! =>
  msg: '''dict object'' has no attribute ''neutron-server''. ''dict object'' has no attribute ''neutron-server''. ''dict object'' has no attribute ''neutron-server''. ''dict object'' has no attribute ''neutron-server'''
fatal: [compute1]: FAILED! =>
  msg: '''dict object'' has no attribute ''neutron-server''. ''dict object'' has no attribute ''neutron-server''. ''dict object'' has no attribute ''neutron-server''. ''dict object'' has no attribute ''neutron-server'''

[Martin Ananda Boeker (mboeker)]

Ignore my last comment, PEBKAC!

[Martin Ananda Boeker (mboeker)]

So this does work with this relatively complicated step, but it seems like it should be much simpler..

If a group like [telegraf] and its children is defined in etc/kayobe/inventory then that should be the configuration that's deployed, no? Changing group definitions in an obscure place like overcloud-components which needs to be imported first and then also enabled is cumbersome.

[Will Szumski (willjs)]

Agreed, this process needs simplifying. It makes more sense when you think of kayobe as a generator for kolla-ansible, but like you say, the current process to make changes to the inventory is cumbersome. Lets make this a ticket to make these customisations simpler.