Ceilometer deployment with kolla ansible fails on bootstrap

Bug #1884919 reported by Malware on 2020-06-24
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
kolla
High
Radosław Piliszek
Ussuri
High
Radosław Piliszek
Victoria
High
Radosław Piliszek

Bug Description

**Bug Report**

I installed the core services of Openstack(currently eval AIO) using master branch of kolla and kolla ansible.
After first fire up I activated AODH, gnocchi and ceilometer and rerun the deployment.

What happened:

TASK [ceilometer : Running Ceilometer bootstrap container] **************************************************************************************************************************************************************************************************
fatal: [10.253.90.10 -> 10.253.90.10]: FAILED! => {"changed": true, "msg": "Container exited with non-zero return code 2", "rc": 2, "stderr": "+ sudo -E kolla_set_configs\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO:__main__:Deleting /etc/ceilometer/ceilometer.conf\nINFO:__main__:Copying /var/lib/kolla/config_files/ceilometer.conf to /etc/ceilometer/ceilometer.conf\nINFO:__main__:Setting permission for /etc/ceilometer/ceilometer.conf\nINFO:__main__:Deleting /etc/ceilometer/event_definitions.yaml\nINFO:__main__:Copying /var/lib/kolla/config_files/event_definitions.yaml to /etc/ceilometer/event_definitions.yaml\nINFO:__main__:Setting permission for /etc/ceilometer/event_definitions.yaml\nINFO:__main__:Deleting /etc/ceilometer/event_pipeline.yaml\nINFO:__main__:Copying /var/lib/kolla/config_files/event_pipeline.yaml to /etc/ceilometer/event_pipeline.yaml\nINFO:__main__:Setting permission for /etc/ceilometer/event_pipeline.yaml\nINFO:__main__:Deleting /etc/ceilometer/pipeline.yaml\nINFO:__main__:Copying /var/lib/kolla/config_files/pipeline.yaml to /etc/ceilometer/pipeline.yaml\nINFO:__main__:Setting permission for /etc/ceilometer/pipeline.yaml\nINFO:__main__:Writing out command to execute\n++ cat /run_command\n+ CMD=ceilometer-agent-notification\n+ ARGS=\n+ sudo kolla_copy_cacerts\n+ [[ ! -n '' ]]\n+ . kolla_extend_start\n++ CEILOMETER_LOG_DIR=/var/log/kolla/ceilometer\n++ [[ ! -d /var/log/kolla/ceilometer ]]\n++ mkdir -p /var/log/kolla/ceilometer\n+++ stat -c %U:%G /var/log/kolla/ceilometer\n++ [[ ceilometer:kolla != \\c\\e\\i\\l\\o\\m\\e\\t\\e\\r\\:\\k\\o\\l\\l\\a ]]\n+++ stat -c %a /var/log/kolla/ceilometer\n++ [[ 2755 != \\7\\5\\5 ]]\n++ chmod 755 /var/log/kolla/ceilometer\n++ . /usr/local/bin/kolla_ceilometer_extend_start\n+++ [[ -n 0 ]]\n+++ [[ gnocchi == \\g\\n\\o\\c\\c\\h\\i ]]\n+++ ceilometer-upgrade ''\nusage: ceilometer-upgrade [-h] [--config-dir DIR] [--config-file PATH]\n [--debug] [--log-config-append PATH]\n [--log-date-format DATE_FORMAT] [--log-dir LOG_DIR]\n [--log-file PATH] [--nodebug]\n [--noskip-gnocchi-resource-types]\n [--noskip-metering-database] [--nouse-journal]\n [--nouse-json] [--nouse-syslog] [--nowatch-log-file]\n [--retry RETRY] [--skip-gnocchi-resource-types]\n [--skip-metering-database]\n [--syslog-log-facility SYSLOG_LOG_FACILITY]\n [--use-journal] [--use-json] [--use-syslog]\n [--version] [--watch-log-file]\nceilometer-upgrade: error: unrecognized arguments: \n", "stderr_lines": ["+ sudo -E kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Deleting /etc/ceilometer/ceilometer.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/ceilometer.conf to /etc/ceilometer/ceilometer.conf", "INFO:__main__:Setting permission for /etc/ceilometer/ceilometer.conf", "INFO:__main__:Deleting /etc/ceilometer/event_definitions.yaml", "INFO:__main__:Copying /var/lib/kolla/config_files/event_definitions.yaml to /etc/ceilometer/event_definitions.yaml", "INFO:__main__:Setting permission for /etc/ceilometer/event_definitions.yaml", "INFO:__main__:Deleting /etc/ceilometer/event_pipeline.yaml", "INFO:__main__:Copying /var/lib/kolla/config_files/event_pipeline.yaml to /etc/ceilometer/event_pipeline.yaml", "INFO:__main__:Setting permission for /etc/ceilometer/event_pipeline.yaml", "INFO:__main__:Deleting /etc/ceilometer/pipeline.yaml", "INFO:__main__:Copying /var/lib/kolla/config_files/pipeline.yaml to /etc/ceilometer/pipeline.yaml", "INFO:__main__:Setting permission for /etc/ceilometer/pipeline.yaml", "INFO:__main__:Writing out command to execute", "++ cat /run_command", "+ CMD=ceilometer-agent-notification", "+ ARGS=", "+ sudo kolla_copy_cacerts", "+ [[ ! -n '' ]]", "+ . kolla_extend_start", "++ CEILOMETER_LOG_DIR=/var/log/kolla/ceilometer", "++ [[ ! -d /var/log/kolla/ceilometer ]]", "++ mkdir -p /var/log/kolla/ceilometer", "+++ stat -c %U:%G /var/log/kolla/ceilometer", "++ [[ ceilometer:kolla != \\c\\e\\i\\l\\o\\m\\e\\t\\e\\r\\:\\k\\o\\l\\l\\a ]]", "+++ stat -c %a /var/log/kolla/ceilometer", "++ [[ 2755 != \\7\\5\\5 ]]", "++ chmod 755 /var/log/kolla/ceilometer", "++ . /usr/local/bin/kolla_ceilometer_extend_start", "+++ [[ -n 0 ]]", "+++ [[ gnocchi == \\g\\n\\o\\c\\c\\h\\i ]]", "+++ ceilometer-upgrade ''", "usage: ceilometer-upgrade [-h] [--config-dir DIR] [--config-file PATH]", " [--debug] [--log-config-append PATH]", " [--log-date-format DATE_FORMAT] [--log-dir LOG_DIR]", " [--log-file PATH] [--nodebug]", " [--noskip-gnocchi-resource-types]", " [--noskip-metering-database] [--nouse-journal]", " [--nouse-json] [--nouse-syslog] [--nowatch-log-file]", " [--retry RETRY] [--skip-gnocchi-resource-types]", " [--skip-metering-database]", " [--syslog-log-facility SYSLOG_LOG_FACILITY]", " [--use-journal] [--use-json] [--use-syslog]", " [--version] [--watch-log-file]", "ceilometer-upgrade: error: unrecognized arguments: "], "stdout": "", "stdout_lines": []}

I as far as I can see the issue is related to gnocchi_upgrade und ceilometer_upgrade.
Both commands are run with '' as parameter which cannot be interpreted ("+++ ceilometer-upgrade ''",).

What you expected to happen:
ceilometer_upgrade and gnocchi_upgrade get no parameter attached if BOOSTRAP_PARAMS is empty.

How to reproduce it (minimal and precise):

set up openstack with "out-of-the-box" Kolla and kolla-ansible while activating ceilometer in globals.yml.

**Environment**:
* OS (e.g. from /etc/os-release): Ubuntu 18.04 Server, Debian 10
* Kernel (e.g. `uname -a`): Ubuntu (4.15.0-106-generic), Debian (4.19.0-9-amd64)
* Docker version if applicable (e.g. `docker version`):
Client: Docker Engine - Community
 Version: 19.03.11
 API version: 1.40
 Go version: go1.13.10
 Git commit: 42e35e61f3
 Built: Mon Jun 1 09:12:22 2020
 OS/Arch: linux/amd64
 Experimental: false
* Kolla version (e.g. `git head or tag or stable branch` or pip package version if using release): master

Malware (swiegand) wrote :

And I have the feeling this is related to the conainer rather than ansible.

Malware (swiegand) on 2020-06-24
description: updated
Radosław Piliszek (yoctozepto) wrote :

Did you try using a stable branch? Train is very stable now (Ubuntu/CentOS7), Ussuri should be too.

Mark Goddard (mgoddard) on 2020-06-25
summary: - Ceilomerter deployment with kolla ansible fails on bootstrap
+ Ceilometer deployment with kolla ansible fails on bootstrap
Radosław Piliszek (yoctozepto) wrote :
Changed in kolla:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Radosław Piliszek (yoctozepto)
Changed in kolla:
status: Triaged → In Progress

Reviewed: https://review.opendev.org/739992
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=69b8b7a49fd682bb15ff84fd0b322dc6c21b5007
Submitter: Zuul
Branch: master

commit 69b8b7a49fd682bb15ff84fd0b322dc6c21b5007
Author: Radosław Piliszek <email address hidden>
Date: Wed Jul 8 15:52:11 2020 +0200

    Fix ceilometer-upgrade params

    It seems [1] has introduced a slight regression in that it tries
    by default to pass an empty string to ceilometer-upgrade which
    makes it fail:

    ceilometer-upgrade: error: unrecognized arguments:

    (note there is nothing more on that line)

    This patch fixes that by allowing the params to be of any number,
    including zero.

    Backport to Ussuri.

    [1] 3d587409ee068843cfa28710f09f65af48b69367

    Change-Id: I989a1dd30d1656b5c1fda7a46abbf5f265fbb72c
    Closes-Bug: #1884919

Changed in kolla:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/740413
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=d1b75e09e1c9abc7c4b834eb512511b76bd93f54
Submitter: Zuul
Branch: stable/ussuri

commit d1b75e09e1c9abc7c4b834eb512511b76bd93f54
Author: Radosław Piliszek <email address hidden>
Date: Wed Jul 8 15:52:11 2020 +0200

    Fix ceilometer-upgrade params

    It seems [1] has introduced a slight regression in that it tries
    by default to pass an empty string to ceilometer-upgrade which
    makes it fail:

    ceilometer-upgrade: error: unrecognized arguments:

    (note there is nothing more on that line)

    This patch fixes that by allowing the params to be of any number,
    including zero.

    Backport to Ussuri.

    [1] 3d587409ee068843cfa28710f09f65af48b69367

    Change-Id: I989a1dd30d1656b5c1fda7a46abbf5f265fbb72c
    Closes-Bug: #1884919
    (cherry picked from commit 69b8b7a49fd682bb15ff84fd0b322dc6c21b5007)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers