Running cyborg bootstrap container failed because init cyborg database failed

Bug #1872432 reported by ya.wang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Medium
ya.wang
Train
Fix Released
Medium
Unassigned
Ussuri
Fix Released
Medium
ya.wang

Bug Description

What happened:

When use kolla-ansible to deploy the OpenStack cluster(train version), the deploy failed because the cyborg database init failed. The output message is below:

  TASK [cyborg : Running cyborg bootstrap container] *********************************************************************************************************************************************** **********************
fatal: [vgpu02 -> vgpu02]: FAILED! => {"changed": true, "msg": "Container exited with non-zero return code 2", "rc": 2, "stderr": "+ sudo -E kolla_set_configs\nINFO:__main__:Loading config file
  at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO :__main__:Copying /var/lib/kolla/config_files/cyborg.conf to /etc/cyborg/cyborg.conf\nINFO:__main__:Setting permission for /etc/cyborg/cyborg.conf\nINFO:__main__:Copying /var/lib/kolla/config_fi les/api-paste.ini to /etc/cyborg/api-paste.ini\nINFO:__main__:Setting permission for /etc/cyborg/api-paste.ini\nINFO:__main__:Writing out command to execute\n++ cat /run_command\n+ CMD='cyborg-a pi --config-file /etc/cyborg/cyborg.conf'\n+ ARGS=\n+ sudo kolla_copy_cacerts\n+ [[ ! -n '' ]]\n+ . kolla_extend_start\n++ CYBORG_LOG_DIR=/var/log/kolla/cyborg\n++ [[ ! -d /var/log/kolla/cyborg ]]\n++ mkdir -p /var/log/kolla/cyborg\n+++ stat -c %U:%G /var/log/kolla/cyborg\n++ [[ root:kolla != \\c\\y\\b\\o\\r\\g\\:\\k\\o\\l\\l\\a ]]\n++ chown cyborg:kolla /var/log/kolla/cyborg\n+++ stat -c %a /var/log/kolla/cyborg\n++ [[ 2755 != \\7\\5\\5 ]]\n++ chmod 755 /var/log/kolla/cyborg\n++ . /usr/local/bin/kolla_cyborg_extend_start\n+++ [[ -n 0 ]]\n+++ cyborg-dbsync\nusage: cyborg-dbsy nc [-h] [--config-dir DIR] [--config-file PATH] [--debug]\n [--log-config-append PATH]\n [--log-date-format DATE_FORMAT] [--log-dir LOG_DIR]\n [--log-file PATH] [--nodebug] [--nouse-journal]\n [--nouse-json] [--nouse-syslog] [--nowatch-log-file]\n [--syslog-log-facility SYSLOG_LOG_FACILI TY]\n [--use-journal] [--use-json] [--use-syslog] [--version]\n [--watch-log-file]\n {upgrade,revision,stamp,version,create_schema} .. .\ncyborg-dbsync: error: too few arguments\n", "stderr_lines": ["+ sudo -E kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Vali dating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Copying /var/lib/kolla/config_files/cyborg.con f to /etc/cyborg/cyborg.conf", "INFO:__main__:Setting permission for /etc/cyborg/cyborg.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/api-paste.ini to /etc/cyborg/api-paste.ini", "IN FO:__main__:Setting permission for /etc/cyborg/api-paste.ini", "INFO:__main__:Writing out command to execute", "++ cat /run_command", "+ CMD='cyborg-api --config-file /etc/cyborg/cyborg.conf'", "+ ARGS=", "+ sudo kolla_copy_cacerts", "+ [[ ! -n '' ]]", "+ . kolla_extend_start", "++ CYBORG_LOG_DIR=/var/log/kolla/cyborg", "++ [[ ! -d /var/log/kolla/cyborg ]]", "++ mkdir -p /var/log/kolla /cyborg", "+++ stat -c %U:%G /var/log/kolla/cyborg", "++ [[ root:kolla != \\c\\y\\b\\o\\r\\g\\:\\k\\o\\l\\l\\a ]]", "++ chown cyborg:kolla /var/log/kolla/cyborg", "+++ stat -c %a /var/log/kolla/ cyborg", "++ [[ 2755 != \\7\\5\\5 ]]", "++ chmod 755 /var/log/kolla/cyborg", "++ . /usr/local/bin/kolla_cyborg_extend_start", "+++ [[ -n 0 ]]", "+++ cyborg-dbsync", "usage: cyborg-dbsync [-h] [- -config-dir DIR] [--config-file PATH] [--debug]", " [--log-config-append PATH]", " [--log-date-format DATE_FORMAT] [--log-dir LOG_DIR]", " [--log-file PATH] [--nodebug] [--nouse-journal]", " [--nouse-json] [--nouse-syslog] [--nowatch-log-file]", " [--syslog-log-facility SYSLOG_LOG_FACILI TY]", " [--use-journal] [--use-json] [--use-syslog] [--version]", " [--watch-log-file]", " {upgrade,revision,stamp,version,create_sche ma} ...", "cyborg-dbsync: error: too few arguments"], "stdout": "", "stdout_lines": []}

What you expected to happen:

The deploy can be success.

How to reproduce it (minimal and precise):

Use train version(master version should also has this issue), enable cyborg, and use kolla-ansible to deploy the cluster.

**Environment**:
OS: CentOS7.7.1908
Kernel: 3.10.0-1062.18.1.el7.x86_64
Docker version: 18.09.9

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.opendev.org/719431

Changed in kolla:
assignee: nobody → ya.wang (ya.wang)
status: New → In Progress
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

By the sources it looks like both methods should just work: https://github.com/openstack/cyborg/tree/master/cyborg/db/sqlalchemy/alembic

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.opendev.org/719431
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=43de719c3ac17085423b381e5938d1bf83e3ff65
Submitter: Zuul
Branch: master

commit 43de719c3ac17085423b381e5938d1bf83e3ff65
Author: ya.wang <wang.ya@99cloud.net>
Date: Mon Apr 13 18:01:24 2020 +0800

    Fix that the cyborg database init failed.

    The database init command should be 'cyborg-dbsync upgrade'.

    Change-Id: I4541c3c070ca70a2ee62f2a0e4ac5d70c57ca6e9
    Closes-bug: #1872432

Changed in kolla:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/720370

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/train)

Reviewed: https://review.opendev.org/720370
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=95d82da7e1d85c840646eb6954f812c946c0de17
Submitter: Zuul
Branch: stable/train

commit 95d82da7e1d85c840646eb6954f812c946c0de17
Author: ya.wang <wang.ya@99cloud.net>
Date: Mon Apr 13 18:01:24 2020 +0800

    Fix that the cyborg database init failed.

    The database init command should be 'cyborg-dbsync upgrade'.

    Change-Id: I4541c3c070ca70a2ee62f2a0e4ac5d70c57ca6e9
    Closes-bug: #1872432
    (cherry picked from commit 43de719c3ac17085423b381e5938d1bf83e3ff65)

tags: added: in-stable-train
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.