Ceilometer deployment with kolla ansible fails on bootstrap

Bug #1884919 reported by Malware
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
kolla
Fix Released
High
Radosław Piliszek
Ussuri
Fix Committed
High
Radosław Piliszek
Victoria
Fix Released
High
Radosław Piliszek

Bug Description

**Bug Report**

I installed the core services of Openstack(currently eval AIO) using master branch of kolla and kolla ansible.
After first fire up I activated AODH, gnocchi and ceilometer and rerun the deployment.

What happened:

TASK [ceilometer : Running Ceilometer bootstrap container] **************************************************************************************************************************************************************************************************
fatal: [10.253.90.10 -> 10.253.90.10]: FAILED! => {"changed": true, "msg": "Container exited with non-zero return code 2", "rc": 2, "stderr": "+ sudo -E kolla_set_configs\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO:__main__:Deleting /etc/ceilometer/ceilometer.conf\nINFO:__main__:Copying /var/lib/kolla/config_files/ceilometer.conf to /etc/ceilometer/ceilometer.conf\nINFO:__main__:Setting permission for /etc/ceilometer/ceilometer.conf\nINFO:__main__:Deleting /etc/ceilometer/event_definitions.yaml\nINFO:__main__:Copying /var/lib/kolla/config_files/event_definitions.yaml to /etc/ceilometer/event_definitions.yaml\nINFO:__main__:Setting permission for /etc/ceilometer/event_definitions.yaml\nINFO:__main__:Deleting /etc/ceilometer/event_pipeline.yaml\nINFO:__main__:Copying /var/lib/kolla/config_files/event_pipeline.yaml to /etc/ceilometer/event_pipeline.yaml\nINFO:__main__:Setting permission for /etc/ceilometer/event_pipeline.yaml\nINFO:__main__:Deleting /etc/ceilometer/pipeline.yaml\nINFO:__main__:Copying /var/lib/kolla/config_files/pipeline.yaml to /etc/ceilometer/pipeline.yaml\nINFO:__main__:Setting permission for /etc/ceilometer/pipeline.yaml\nINFO:__main__:Writing out command to execute\n++ cat /run_command\n+ CMD=ceilometer-agent-notification\n+ ARGS=\n+ sudo kolla_copy_cacerts\n+ [[ ! -n '' ]]\n+ . kolla_extend_start\n++ CEILOMETER_LOG_DIR=/var/log/kolla/ceilometer\n++ [[ ! -d /var/log/kolla/ceilometer ]]\n++ mkdir -p /var/log/kolla/ceilometer\n+++ stat -c %U:%G /var/log/kolla/ceilometer\n++ [[ ceilometer:kolla != \\c\\e\\i\\l\\o\\m\\e\\t\\e\\r\\:\\k\\o\\l\\l\\a ]]\n+++ stat -c %a /var/log/kolla/ceilometer\n++ [[ 2755 != \\7\\5\\5 ]]\n++ chmod 755 /var/log/kolla/ceilometer\n++ . /usr/local/bin/kolla_ceilometer_extend_start\n+++ [[ -n 0 ]]\n+++ [[ gnocchi == \\g\\n\\o\\c\\c\\h\\i ]]\n+++ ceilometer-upgrade ''\nusage: ceilometer-upgrade [-h] [--config-dir DIR] [--config-file PATH]\n [--debug] [--log-config-append PATH]\n [--log-date-format DATE_FORMAT] [--log-dir LOG_DIR]\n [--log-file PATH] [--nodebug]\n [--noskip-gnocchi-resource-types]\n [--noskip-metering-database] [--nouse-journal]\n [--nouse-json] [--nouse-syslog] [--nowatch-log-file]\n [--retry RETRY] [--skip-gnocchi-resource-types]\n [--skip-metering-database]\n [--syslog-log-facility SYSLOG_LOG_FACILITY]\n [--use-journal] [--use-json] [--use-syslog]\n [--version] [--watch-log-file]\nceilometer-upgrade: error: unrecognized arguments: \n", "stderr_lines": ["+ sudo -E kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Deleting /etc/ceilometer/ceilometer.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/ceilometer.conf to /etc/ceilometer/ceilometer.conf", "INFO:__main__:Setting permission for /etc/ceilometer/ceilometer.conf", "INFO:__main__:Deleting /etc/ceilometer/event_definitions.yaml", "INFO:__main__:Copying /var/lib/kolla/config_files/event_definitions.yaml to /etc/ceilometer/event_definitions.yaml", "INFO:__main__:Setting permission for /etc/ceilometer/event_definitions.yaml", "INFO:__main__:Deleting /etc/ceilometer/event_pipeline.yaml", "INFO:__main__:Copying /var/lib/kolla/config_files/event_pipeline.yaml to /etc/ceilometer/event_pipeline.yaml", "INFO:__main__:Setting permission for /etc/ceilometer/event_pipeline.yaml", "INFO:__main__:Deleting /etc/ceilometer/pipeline.yaml", "INFO:__main__:Copying /var/lib/kolla/config_files/pipeline.yaml to /etc/ceilometer/pipeline.yaml", "INFO:__main__:Setting permission for /etc/ceilometer/pipeline.yaml", "INFO:__main__:Writing out command to execute", "++ cat /run_command", "+ CMD=ceilometer-agent-notification", "+ ARGS=", "+ sudo kolla_copy_cacerts", "+ [[ ! -n '' ]]", "+ . kolla_extend_start", "++ CEILOMETER_LOG_DIR=/var/log/kolla/ceilometer", "++ [[ ! -d /var/log/kolla/ceilometer ]]", "++ mkdir -p /var/log/kolla/ceilometer", "+++ stat -c %U:%G /var/log/kolla/ceilometer", "++ [[ ceilometer:kolla != \\c\\e\\i\\l\\o\\m\\e\\t\\e\\r\\:\\k\\o\\l\\l\\a ]]", "+++ stat -c %a /var/log/kolla/ceilometer", "++ [[ 2755 != \\7\\5\\5 ]]", "++ chmod 755 /var/log/kolla/ceilometer", "++ . /usr/local/bin/kolla_ceilometer_extend_start", "+++ [[ -n 0 ]]", "+++ [[ gnocchi == \\g\\n\\o\\c\\c\\h\\i ]]", "+++ ceilometer-upgrade ''", "usage: ceilometer-upgrade [-h] [--config-dir DIR] [--config-file PATH]", " [--debug] [--log-config-append PATH]", " [--log-date-format DATE_FORMAT] [--log-dir LOG_DIR]", " [--log-file PATH] [--nodebug]", " [--noskip-gnocchi-resource-types]", " [--noskip-metering-database] [--nouse-journal]", " [--nouse-json] [--nouse-syslog] [--nowatch-log-file]", " [--retry RETRY] [--skip-gnocchi-resource-types]", " [--skip-metering-database]", " [--syslog-log-facility SYSLOG_LOG_FACILITY]", " [--use-journal] [--use-json] [--use-syslog]", " [--version] [--watch-log-file]", "ceilometer-upgrade: error: unrecognized arguments: "], "stdout": "", "stdout_lines": []}

I as far as I can see the issue is related to gnocchi_upgrade und ceilometer_upgrade.
Both commands are run with '' as parameter which cannot be interpreted ("+++ ceilometer-upgrade ''",).

What you expected to happen:
ceilometer_upgrade and gnocchi_upgrade get no parameter attached if BOOSTRAP_PARAMS is empty.

How to reproduce it (minimal and precise):

set up openstack with "out-of-the-box" Kolla and kolla-ansible while activating ceilometer in globals.yml.

**Environment**:
* OS (e.g. from /etc/os-release): Ubuntu 18.04 Server, Debian 10
* Kernel (e.g. `uname -a`): Ubuntu (4.15.0-106-generic), Debian (4.19.0-9-amd64)
* Docker version if applicable (e.g. `docker version`):
Client: Docker Engine - Community
 Version: 19.03.11
 API version: 1.40
 Go version: go1.13.10
 Git commit: 42e35e61f3
 Built: Mon Jun 1 09:12:22 2020
 OS/Arch: linux/amd64
 Experimental: false
* Kolla version (e.g. `git head or tag or stable branch` or pip package version if using release): master

Revision history for this message
Malware (swiegand) wrote :

And I have the feeling this is related to the conainer rather than ansible.

Malware (swiegand)
description: updated
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Did you try using a stable branch? Train is very stable now (Ubuntu/CentOS7), Ussuri should be too.

Mark Goddard (mgoddard)
summary: - Ceilomerter deployment with kolla ansible fails on bootstrap
+ Ceilometer deployment with kolla ansible fails on bootstrap
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :
Changed in kolla:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Radosław Piliszek (yoctozepto)
Changed in kolla:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.opendev.org/739992
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=69b8b7a49fd682bb15ff84fd0b322dc6c21b5007
Submitter: Zuul
Branch: master

commit 69b8b7a49fd682bb15ff84fd0b322dc6c21b5007
Author: Radosław Piliszek <email address hidden>
Date: Wed Jul 8 15:52:11 2020 +0200

    Fix ceilometer-upgrade params

    It seems [1] has introduced a slight regression in that it tries
    by default to pass an empty string to ceilometer-upgrade which
    makes it fail:

    ceilometer-upgrade: error: unrecognized arguments:

    (note there is nothing more on that line)

    This patch fixes that by allowing the params to be of any number,
    including zero.

    Backport to Ussuri.

    [1] 3d587409ee068843cfa28710f09f65af48b69367

    Change-Id: I989a1dd30d1656b5c1fda7a46abbf5f265fbb72c
    Closes-Bug: #1884919

Changed in kolla:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/740413

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/ussuri)

Reviewed: https://review.opendev.org/740413
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=d1b75e09e1c9abc7c4b834eb512511b76bd93f54
Submitter: Zuul
Branch: stable/ussuri

commit d1b75e09e1c9abc7c4b834eb512511b76bd93f54
Author: Radosław Piliszek <email address hidden>
Date: Wed Jul 8 15:52:11 2020 +0200

    Fix ceilometer-upgrade params

    It seems [1] has introduced a slight regression in that it tries
    by default to pass an empty string to ceilometer-upgrade which
    makes it fail:

    ceilometer-upgrade: error: unrecognized arguments:

    (note there is nothing more on that line)

    This patch fixes that by allowing the params to be of any number,
    including zero.

    Backport to Ussuri.

    [1] 3d587409ee068843cfa28710f09f65af48b69367

    Change-Id: I989a1dd30d1656b5c1fda7a46abbf5f265fbb72c
    Closes-Bug: #1884919
    (cherry picked from commit 69b8b7a49fd682bb15ff84fd0b322dc6c21b5007)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.