Deploy failure during task ceilometer : Running Ceilometer bootstrap container

[Eric Miller]

Deploy of 7.0.0.0b3 on cleanly installed VMs (running on ESXi 5.5), which works fine with 7.0.0.0b2, 6.1.0, and 6.0.0, produces an error at this task.

I included a zip of the "kolla-ansible -vvv" output - see the bottom of the file for the error.

My guess is that the "image" attribute is defined using an array like this:

            "image": "['1', '9', '2', '.', '1', '6', '8', '.', '2', '5', '4', '.', '2', '3', '0', ':', '5', '0', '0', '0', '/', 'k', 'o', 'l', 'l', 'a', '/', 'c', 'e', 'n', 't', 'o', 's', '-', 's', 'o', 'u', 'r', 'c', 'e', '-', 'c', 'e', 'i', 'l', 'o', 'm', 'e', 't', 'e', 'r', '-', 'n', 'o', 't', 'i', 'f', 'i', 'c', 'a', 't', 'i', 'o', 'n', ':', '7', '.', '0', '.', '0']",

None of the other tasks use this format to describe the image, and I haven't found any reason why Ansible would format the image name in: /usr/share/kolla-ansible/ansible/roles/ceilometer/tasks/bootstrap_service.yml

as defined in: /usr/share/kolla-ansible/ansible/roles/ceilometer/defaults/main.yml

Ansible version info (using "ansible --version"):

ansible 2.6.1
  config file = /root/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

Any ideas?

Eric

[Eric Miller (erickmiller)]

Sorry, should have mentioned this is deployed on CentOS 7.5 with the latest patches on both the deployment machine and the OpenStack node VMs.

Eric

[Eric Miller (erickmiller)]

I guess I also didn't specify that docker images were built with the "source" option.

Docker version info:
Docker version 1.13.1, build dded712/1.13.1

Both kolla and kolla-ansible packages were installed after uninstalling version 7.0.0.0b2 on this deployment machine. Docker was also uninstalled prior, with all of /var/lib/docker deleted. Docker was installed with a new registry for docker source images to be pushed.

[Eric Miller (erickmiller)]

I think I found the problem, but am testing.

It looks like line 14 of /usr/share/kolla-ansible/ansible/roles/ceilometer/tasks/bootstrap_service.yml

has content:
    image: "{{ ceilometer_notification.image | reject('equalto', '')|list }}"

whereas the older version of this file has content:
    image: "{{ ceilometer_notification.image }}"

I removed the |list jinja2 filter and am testing the deployment.

Eric

[Eric Miller (erickmiller)]

Correction:
I removed the " | reject('equalto', '')|list" filter.

I was able to run a successfully completed deploy after doing this.

So, not sure what this was supposed to do, and why, but it caused an issue.

Eric

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/587327
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=ffe388ba063d3013b5f10b3d7b84bcd0171c89bc
Submitter: Zuul
Branch: master

commit ffe388ba063d3013b5f10b3d7b84bcd0171c89bc
Author: Eduardo Gonzalez <email address hidden>
Date: Tue Jul 31 09:55:28 2018 +0200

    Fix ceilometer bootstrap

    Ceilometer image filter was incorrectly used. Thr filter
    should be in the volume list instead.

    Change-Id: I3ad8a1d85607004b28f9a25b9a4e9f8b914ae9df
    Closes-Bug: #1784496

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.0.0.0rc1 release candidate.