Cloudkitty database upgrade fail version 8.0.0

[Kleyton]

CentOS 7
kolla-ansible 8.0.0 (python 2.7 / Ansible 2.8.1)
Binary Centos
Official Images

Hi,

I'm trying to deploy/upgrade Cloudkitty using kolla-ansible 8.0.0 (stein). That's the error message.

TASK [cloudkitty : Running Cloudkitty bootstrap container] *******************************************************************************************
fatal: [localhost -> localhost]: FAILED! => {"changed": true, "msg": "Container exited with non-zero return code 1", "rc": 1, "stderr": "+ sudo -E kolla_set_configs\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO:__main__:Deleting /etc/cloudkitty/cloudkitty.conf\nINFO:__main__:Copying /var/lib/kolla/config_files/cloudkitty.conf to /etc/cloudkitty/cloudkitty.conf\nINFO:__main__:Setting permission for /etc/cloudkitty/cloudkitty.conf\nINFO:__main__:Copying /var/lib/kolla/config_files/wsgi-cloudkitty.conf to /etc/httpd/conf.d/wsgi-cloudkitty.conf\nINFO:__main__:Setting permission for /etc/httpd/conf.d/wsgi-cloudkitty.conf\nINFO:__main__:Writing out command to execute\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api.log\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api-error.log\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api-access.log\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-processor.log\n++ cat /run_command\n+ CMD='httpd -DFOREGROUND'\n+ ARGS=\n+ [[ ! -n '' ]]\n+ . kolla_extend_start\n++ [[ ! -d /var/log/kolla/cloudkitty ]]\n+++ stat -c %a /var/log/kolla/cloudkitty\n++ [[ 2755 != \\7\\5\\5 ]]\n++ chmod 755 /var/log/kolla/cloudkitty\n++ . /usr/local/bin/kolla_cloudkitty_extend_start\n+++ [[ centos =~ debian|ubuntu ]]\n+++ rm -rf /var/run/httpd/htcacheclean /run/httpd/htcacheclean '/tmp/httpd*'\n+++ [[ -n 0 ]]\n+++ cloudkitty-dbsync upgrade\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\n+++ cloudkitty-storage-init\n", "stderr_lines": ["+ sudo -E kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Deleting /etc/cloudkitty/cloudkitty.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/cloudkitty.conf to /etc/cloudkitty/cloudkitty.conf", "INFO:__main__:Setting permission for /etc/cloudkitty/cloudkitty.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/wsgi-cloudkitty.conf to /etc/httpd/conf.d/wsgi-cloudkitty.conf", "INFO:__main__:Setting permission for /etc/httpd/conf.d/wsgi-cloudkitty.conf", "INFO:__main__:Writing out command to execute", "INFO:__main__:Setting permission for /var/log/kolla/cloudkitty", "INFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api.log", "INFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api-error.log", "INFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api-access.log", "INFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-processor.log", "++ cat /run_command", "+ CMD='httpd -DFOREGROUND'", "+ ARGS=", "+ [[ ! -n '' ]]", "+ . kolla_extend_start", "++ [[ ! -d /var/log/kolla/cloudkitty ]]", "+++ stat -c %a /var/log/kolla/cloudkitty", "++ [[ 2755 != \\7\\5\\5 ]]", "++ chmod 755 /var/log/kolla/cloudkitty", "++ . /usr/local/bin/kolla_cloudkitty_extend_start", "+++ [[ centos =~ debian|ubuntu ]]", "+++ rm -rf /var/run/httpd/htcacheclean /run/httpd/htcacheclean '/tmp/httpd*'", "+++ [[ -n 0 ]]", "+++ cloudkitty-dbsync upgrade", "INFO [alembic.runtime.migration] Context impl MySQLImpl.", "INFO [alembic.runtime.migration] Will assume non-transactional DDL.", "INFO [alembic.runtime.migration] Context impl MySQLImpl.", "INFO [alembic.runtime.migration] Will assume non-transactional DDL.", "INFO [alembic.runtime.migration] Context impl MySQLImpl.", "INFO [alembic.runtime.migration] Will assume non-transactional DDL.", "+++ cloudkitty-storage-init"], "stdout": "", "stdout_lines": []}

[Mark Goddard (mgoddard)]

I don't see any errors in there. Can you check the cloudkitty log files under /var/log/kolla/cloudkitty?

[Laurent FERRIE (laurent-ob)]

CentOS 7
kolla-ansible 8.0.0 (python 2.7 / Ansible 2.8.3)
Binary Centos
Fresh install for test

enable_glance: "{{ enable_openstack_core | bool }}"
enable_haproxy: "yes"
enable_keepalived: "{{ enable_haproxy | bool }}"
enable_keystone: "{{ enable_openstack_core | bool }}"
enable_mariadb: "yes"
enable_memcached: "yes"
enable_neutron: "{{ enable_openstack_core | bool }}"
enable_nova: "{{ enable_openstack_core | bool }}"
enable_rabbitmq: "{{ 'yes' if om_rpc_transport == 'rabbit' or om_notify_transport == 'rabbit' else 'no' }}"

cloudkitty: "yes"
influxdb: "yes"

Hello,

I meet the same problem attached here the file : kolla_logs/_data/cloudkitty/cloudkitty-api.log

2019-08-13 17:13:26.652 25 WARNING cloudkitty.storage [-] V2 Storage is in beta. Its API is considered stable but its implementation may still evolve.
2019-08-13 17:13:26.857 25 CRITICAL cloudkitty [-] Unhandled error: InfluxDBClientError: get_list_retention_policies() requires a database as a parameter or the client to be using a database
2019-08-13 17:13:26.857 25 ERROR cloudkitty Traceback (most recent call last):
2019-08-13 17:13:26.857 25 ERROR cloudkitty File "/var/lib/kolla/venv/bin/cloudkitty-storage-init", line 10, in <module>
2019-08-13 17:13:26.857 25 ERROR cloudkitty sys.exit(main())
2019-08-13 17:13:26.857 25 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/cli/storage.py", line 32, in main
2019-08-13 17:13:26.857 25 ERROR cloudkitty init_storage_backend()
2019-08-13 17:13:26.857 25 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/cli/storage.py", line 25, in init_storage_backend
2019-08-13 17:13:26.857 25 ERROR cloudkitty backend.init()
2019-08-13 17:13:26.857 25 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/storage/v2/influx.py", line 220, in init
2019-08-13 17:13:26.857 25 ERROR cloudkitty if not self._conn.retention_policy_exists(database, policy):
2019-08-13 17:13:26.857 25 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/storage/v2/influx.py", line 102, in retention_policy_exists
2019-08-13 17:13:26.857 25 ERROR cloudkitty policies = self._conn.get_list_retention_policies(database)
2019-08-13 17:13:26.857 25 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/influxdb/client.py", line 712, in get_list_retention_policies
2019-08-13 17:13:26.857 25 ERROR cloudkitty "get_list_retention_policies() requires a database as a "
2019-08-13 17:13:26.857 25 ERROR cloudkitty InfluxDBClientError: get_list_retention_policies() requires a database as a parameter or the client to be using a database
2019-08-13 17:13:26.857 25 ERROR cloudkitty

[Kleyton (kleytonn)]

Hi Mark,

I don't have any log about this because it's a update process of Cloudkitty from Rocky to Stein using Kolla-ansible 8.0.0. If I install Cloudkitty (Stein) from zero I don't have this problem.

That is another erro from a multinode upgrade from Rocky to Stein using kolla-ansible. I think is something related with database schema upgrade.

TASK [cloudkitty : Running Cloudkitty bootstrap container] *****************************************************************************************************
fatal: [lxl1scnco001 -> lxl1scnco001]: FAILED! => {"changed": true, "msg": "Container exited with non-zero return code 1", "rc": 1, "stderr": "+ sudo -E kolla_set_configs\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO:__main__:Deleting /etc/cloudkitty/cloudkitty.conf\nINFO:__main__:Copying /var/lib/kolla/config_files/cloudkitty.conf to /etc/cloudkitty/cloudkitty.conf\nINFO:__main__:Setting permission for /etc/cloudkitty/cloudkitty.conf\nINFO:__main__:Copying /var/lib/kolla/config_files/wsgi-cloudkitty.conf to /etc/httpd/conf.d/wsgi-cloudkitty.conf\nINFO:__main__:Setting permission for /etc/httpd/conf.d/wsgi-cloudkitty.conf\nINFO:__main__:Writing out command to execute\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api.log\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api-error.log\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api-access.log\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-processor.log\n++ cat /run_command\n+ CMD='httpd -DFOREGROUND'\n+ ARGS=\n+ [[ ! -n '' ]]\n+ . kolla_extend_start\n++ [[ ! -d /var/log/kolla/cloudkitty ]]\n+++ stat -c %a /var/log/kolla/cloudkitty\n++ [[ 755 != \\7\\5\\5 ]]\n++ . /usr/local/bin/kolla_cloudkitty_extend_start\n+++ [[ centos =~ debian|ubuntu ]]\n+++ rm -rf /var/run/httpd/htcacheclean
/run/httpd/htcacheclean '/tmp/httpd*'\n+++ [[ -n 0 ]]\n+++ cloudkitty-dbsync upgrade\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO [alembic.runtime.migration] Running upgrade 4da82e1c11c8 -> 644faa4491fd, Update tenant_id type from uuid to text\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\n+++ cloudkitty-storage-init\n", "stderr_lines": ["+ sudo -E kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Deleting /etc/cloudkitty/cloudkitty.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/cloudkitty.conf to /etc/cloudkitty/cloudkitty.conf", "INFO:__ma...

[Mark Goddard (mgoddard)]

Looks like a problem with the configuration of the connection to influxdb:

2019-08-13 17:13:26.857 25 ERROR cloudkitty InfluxDBClientError: get_list_retention_policies() requires a database as a parameter or the client to be using a database

[Rodrigo Lima (rodrigolima1981)]

It is the same error on my deploy:

2019-08-28 09:25:43.960 26 WARNING cloudkitty.storage [-] V2 Storage is in beta. Its API is considered stable but its implementation may still evolve.
2019-08-28 09:25:44.129 26 CRITICAL cloudkitty [-] Unhandled error: InfluxDBClientError: get_list_retention_policies() requires a database as a parameter or the client to be using a database
2019-08-28 09:25:44.129 26 ERROR cloudkitty Traceback (most recent call last):
2019-08-28 09:25:44.129 26 ERROR cloudkitty File "/var/lib/kolla/venv/bin/cloudkitty-storage-init", line 10, in <module>
2019-08-28 09:25:44.129 26 ERROR cloudkitty sys.exit(main())
2019-08-28 09:25:44.129 26 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/cli/storage.py", line 32, in main
2019-08-28 09:25:44.129 26 ERROR cloudkitty init_storage_backend()
2019-08-28 09:25:44.129 26 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/cli/storage.py", line 25, in init_storage_backend
2019-08-28 09:25:44.129 26 ERROR cloudkitty backend.init()
2019-08-28 09:25:44.129 26 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/storage/v2/influx.py", line 220, in init
2019-08-28 09:25:44.129 26 ERROR cloudkitty if not self._conn.retention_policy_exists(database, policy):
2019-08-28 09:25:44.129 26 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/storage/v2/influx.py", line 102, in retention_policy_exists
2019-08-28 09:25:44.129 26 ERROR cloudkitty policies = self._conn.get_list_retention_policies(database)
2019-08-28 09:25:44.129 26 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/influxdb/client.py", line 712, in get_list_retention_policies
2019-08-28 09:25:44.129 26 ERROR cloudkitty "get_list_retention_policies() requires a database as a "
2019-08-28 09:25:44.129 26 ERROR cloudkitty InfluxDBClientError: get_list_retention_policies() requires a database as a parameter or the client to be using a database
2019-08-28 09:25:44.129 26 ERROR cloudkitty
2019-09-11 08:24:55.287 25 WARNING cloudkitty.storage [-] V2 Storage is in beta. Its API is considered stable but its implementation may still evolve.
2019-09-11 08:24:55.421 25 CRITICAL cloudkitty [-] Unhandled error: InfluxDBClientError: get_list_retention_policies() requires a database as a parameter or the client to be using a database
2019-09-11 08:24:55.421 25 ERROR cloudkitty Traceback (most recent call last):
2019-09-11 08:24:55.421 25 ERROR cloudkitty File "/var/lib/kolla/venv/bin/cloudkitty-storage-init", line 10, in <module>
2019-09-11 08:24:55.421 25 ERROR cloudkitty sys.exit(main())
2019-09-11 08:24:55.421 25 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/cli/storage.py", line 32, in main
2019-09-11 08:24:55.421 25 ERROR cloudkitty init_storage_backend()
2019-09-11 08:24:55.421 25 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/cli/storage.py", line 25, in init_storage_backend
2019-09-11 08:24:55.421 25 ERROR cloudkitty backend.init()
2019-09-11 08:24:55.421 25 ERROR cloudkitty ...

[Mark Goddard (mgoddard)]

Has the storage backend been changed to influxdb?

cloudkitty_storage_backend

[Mark Goddard (mgoddard)]

The default backend was changed in cloudkitty in stein to influxdb, but we are not configuring it correctly.

In kolla-ansible we recently merged a commit with support for the influxdb backend: https://review.opendev.org/#/c/658418/. It forces the default backend to be sqlalchemy. It did not get merged into stein though.

So we are now in a position where we have these defaults:

rocky: sqlalchemy
stein (8.0.0 and 8.0.1): influxdb
master: sqlalchemy (overridden by kolla-ansible for 'backwards compat')

I think we probably just need to move with the times here and set our default to influxdb, possibly backporting https://review.opendev.org/#/c/658418.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/681545

[Mark Goddard (mgoddard)]

Patch switching default to influxdb in cloudkitty, merged in stein cycle: https://review.opendev.org/#/c/615928/.

[Rodrigo Lima (rodrigolima1981)]

Mark,

I need to put this in globals.yml?

enable_cloudkitty: "yes"
cloudkitty_storage_backend: "influxdb"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/681554

[Rodrigo Lima (rodrigolima1981)]

I put this in globals.yml and receive the same error.

[Mark Goddard (mgoddard)]

Rodrigo, I'm afraid it's not quite that simple - we don't have the code to create an influxdb database. I proposed backporting https://review.opendev.org/#/c/681545/, and also created https://review.opendev.org/681554 to switch the default of cloudkitty_storage_backend to influxdb.

If you are able to test these (together), that would be helpful.

[Rodrigo Lima (rodrigolima1981)]

Sure, Trying now.

[Rodrigo Lima (rodrigolima1981)]

Well, now we have a new action on playbook execution:

TASK [cloudkitty : Creating Cloudkitty influxdb database] ***********************************************************************************************************************
changed: [nimbus-ctrl02]

But after the code modifications, receive same error:

TASK [cloudkitty : Running Cloudkitty bootstrap container] **********************************************************************************************************************
fatal: [nimbus-ctrl02 -> nimbus-ctrl01]: FAILED! => {"changed": true, "msg": "Container exited with non-zero return code 1", "rc": 1, "stderr": "+ sudo -E kolla_set_configs\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO:__main__:Copying /var/lib/kolla/config_files/cloudkitty.conf to /etc/cloudkitty/cloudkitty.conf\nINFO:__main__:Setting permission for /etc/cloudkitty/cloudkitty.conf\nINFO:__main__:Copying /var/lib/kolla/config_files/wsgi-cloudkitty.conf to /etc/httpd/conf.d/wsgi-cloudkitty.conf\nINFO:__main__:Setting permission for /etc/httpd/conf.d/wsgi-cloudkitty.conf\nINFO:__main__:Writing out command to execute\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api.log\n++ cat /run_command\n+ CMD='httpd -DFOREGROUND'\n+ ARGS=\n+ [[ ! -n '' ]]\n+ . kolla_extend_start\n++ [[ ! -d /var/log/kolla/cloudkitty ]]\n+++ stat -c %a /var/log/kolla/cloudkitty\n++ [[ 2755 != \\7\\5\\5 ]]\n++ chmod 755 /var/log/kolla/cloudkitty\n++ . /usr/local/bin/kolla_cloudkitty_extend_start\n+++ [[ centos =~ debian|ubuntu ]]\n+++ rm -rf /var/run/httpd/htcacheclean /run/httpd/htcacheclean '/tmp/httpd*'\n+++ [[ -n 0 ]]\n+++ cloudkitty-dbsync upgrade\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\n+++ cloudkitty-storage-init\n", "stderr_lines": ["+ sudo -E kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Copying /var/lib/kolla/config_files/cloudkitty.conf to /etc/cloudkitty/cloudkitty.conf", "INFO:__main__:Setting permission for /etc/cloudkitty/cloudkitty.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/wsgi-cloudkitty.conf to /etc/httpd/conf.d/wsgi-cloudkitty.conf", "INFO:__main__:Setting permission for /etc/httpd/conf.d/wsgi-cloudkitty.conf", "INFO:__main__:Writing out command to execute", "INFO:__main__:Setting permission for /var/log/kolla/cloudkitty", "INFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api.log", "++ cat /run_command", "...

[Mark Goddard (mgoddard)]

Thanks for testing. I'm testing this in CI using this patch: https://review.opendev.org/681555.

[Rodrigo Lima (rodrigolima1981)]

Hi Mark, just for information: Last night I tested master branch that I downloaded on GitHub and cloudkitty deploy works.

[Mark Goddard (mgoddard)]

Thanks. I expect that's because currently on master cloudkitty is using the sqlalchemy backend by default.

[Mark Goddard (mgoddard)]

Another problem I've found is that the config section should be [storage_influxdb], not [storage_influx]. Rodrigo, could you try that?

[Rodrigo Lima (rodrigolima1981)]

Sure, give me a few hours to setup another environment to try

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/681772

[Kleyton (kleytonn)]

Hi,

I'm testing the latest master branch from git (8.1.0.dev469) and is all right now. No more errors installing cloudkitty (Stein). kolla-ansible is the best !!

Thank you all!

[Mark Goddard (mgoddard)]

@Kleyton, just a warning - the changes I have proposed change the default storage backend for cloudkitty to influxdb. This is already the case in Stein, but as we can see in this bug it is incorrectly configured, so it doesn't work. Master is currently configured for sqlalchemy, but if my proposed changes get merged it will become influxdb as well.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/681554
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=27f4876eed21d7e1e615646d3c3da1e9025ced4a
Submitter: Zuul
Branch: master

commit 27f4876eed21d7e1e615646d3c3da1e9025ced4a
Author: Mark Goddard <email address hidden>
Date: Wed Sep 11 18:15:02 2019 +0100

    Switch default cloudkitty storage backend to influxdb

    Backport: stein

    In the Stein release, cloudkitty switched the default storage backend
    from sqlalchemy to influxdb. In kolla-ansible stein configuration, we
    did not explicitly set the storage backend, and so we automatically
    picked up this change. However, prior to
    https://review.opendev.org/#/c/615928/ we did not have full support for
    InfluxDB as a storage backend, and so this has broken the Rocky-Stein
    upgrade (https://bugs.launchpad.net/kolla-ansible/+bug/1838641), which
    fails with this during the DB sync:

    ERROR cloudkitty InfluxDBClientError: get_list_retention_policies()
    requires a database as a parameter or the client to be using a database

    This change synchronises our default with cloudkitty's (influxdb), and
    also provides an upgrade transition to create the influxdb database.

    We also move the cloudkitty_storage_backend variable to
    group_vars/all.yml, since it is used to determine whether to enable
    influxdb.

    Finally, the section name in cloudkitty.conf was incorrect - it was
    storage_influx, but should be storage_influxdb.

    Change-Id: I71f2ed11bd06f58e141d222e2709835b7ddb2c71
    Closes-Bug: #1838641

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/681545
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=220943fad49cba96c93902c91038d8e1785c78b4
Submitter: Zuul
Branch: stable/stein

commit 220943fad49cba96c93902c91038d8e1785c78b4
Author: Rafael Weingärtner <email address hidden>
Date: Fri May 10 13:40:46 2019 -0300

    Cloudkitty InfluxDB Storage backend via Kolla-ansible

    NOTE: Backported feature to Stein, as part of a fix for bug 1838641.

    This proposal will add support to Kolla-Ansible for Cloudkitty
     InfluxDB storage system deployment. The feature of InfluxDB as the
     storage backend for Cloudkitty was created with the following commit
     https://github.com/openstack/cloudkitty/commit/
     c4758e78b49386145309a44623502f8095a2c7ee

    Problem Description
    ===================

    With the addition of support for InfluxDB in Cloudkitty, which is
    achieving general availability via Stein release, we need a method to
    easily configure/support this storage backend system via Kolla-ansible.

    Kolla-ansible is already able to deploy and configure an InfluxDB
    system. Therefore, this proposal will use the InfluxDB deployment
    configured via Kolla-ansible to connect to CloudKitty and use it as a
    storage backend.

    If we do not provide a method for users (operators) to manage
    Cloudkitty storage backend via Kolla-ansible, the user has to execute
    these changes/configurations manually (or via some other set of
    automated scripts), which creates distributed set of configuration
    files, "configurations" scripts that have different versioning schemas
    and life cycles.

    Proposed Change
    ===============

    Architecture
    ------------

    We propose a flag that users can use to make Kolla-ansible configure
    CloudKitty to use InfluxDB as the storage backend system. When
    enabling this flag, Kolla-ansible will also enable the deployment of
    the InfluxDB via Kolla-ansible automatically.

    CloudKitty will be configured accordingly to [1] and [2]. We will also
    externalize the "retention_policy", "use_ssl", and "insecure", to
    allow fine granular configurations to operators. All of these
    configurations will only be used when configured; therefore, when they
    are not set, the default value/behavior defined in Cloudkitty will be
    used. Moreover, when we configure "use_ssl" to "true", the user will
    be able to set "cafile" to a custom trusted CA file. Again, if these
    variables are not set, the default ones in Cloudkitty will be used.

    Implementation
    --------------
    We need to introduce a new variable called
    `cloudkitty_storage_backend`. Valid options are `sqlalchemy` or
    `influxdb`. The default value in Kolla-ansible is `sqlalchemy` for
    backward compatibility. Then, the first step is to change the
    definition for the following variable:
    `/ansible/group_vars/all.yml:enable_influxdb: "{{ enable_monasca |
    bool }}"`

    We also need to enable InfluxDB when CloudKitty is configured to use
    it as the storage backend. Afterwards, we need to create tasks in
    ...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/681772
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=12950be4ac06e5ab341958bd2cfbaac0893fc99f
Submitter: Zuul
Branch: stable/stein

commit 12950be4ac06e5ab341958bd2cfbaac0893fc99f
Author: Mark Goddard <email address hidden>
Date: Wed Sep 11 18:15:02 2019 +0100

    Switch default cloudkitty storage backend to influxdb

    Backport: stein

    In the Stein release, cloudkitty switched the default storage backend
    from sqlalchemy to influxdb. In kolla-ansible stein configuration, we
    did not explicitly set the storage backend, and so we automatically
    picked up this change. However, prior to
    https://review.opendev.org/#/c/615928/ we did not have full support for
    InfluxDB as a storage backend, and so this has broken the Rocky-Stein
    upgrade (https://bugs.launchpad.net/kolla-ansible/+bug/1838641), which
    fails with this during the DB sync:

    ERROR cloudkitty InfluxDBClientError: get_list_retention_policies()
    requires a database as a parameter or the client to be using a database

    This change synchronises our default with cloudkitty's (influxdb), and
    also provides an upgrade transition to create the influxdb database.

    We also move the cloudkitty_storage_backend variable to
    group_vars/all.yml, since it is used to determine whether to enable
    influxdb.

    Finally, the section name in cloudkitty.conf was incorrect - it was
    storage_influx, but should be storage_influxdb.

    Change-Id: I71f2ed11bd06f58e141d222e2709835b7ddb2c71
    Closes-Bug: #1838641
    (cherry picked from commit 27f4876eed21d7e1e615646d3c3da1e9025ced4a)

[Martin Lukas (lxfozzibaer)]

Hello,
I still see this error when installing OpenStack with kolla-ansible and cloudkitty enabled.

What I do:
- I get kolla-ansible and kolla from git using:
  git clone https://github.com/openstack/kolla-ansible -b stable/stein
  git clone https://github.com/openstack/kolla -b stable/stein
- Then I install them:
  pip install ./kolla
  pip install ./kolla-ansible

Then I prepare the /etc/kolla directory with the required files etc. and install OpenStack using kolla-ansible.

In the git log of kolla-ansible I can see the above mentioned commit (and others mentioning influxdb):
...
commit a40520e34a62f4a1e011a645e11ec5e7bb979af0
Merge: 599cbdb 220943f
Author: Zuul <email address hidden>
Date: Thu Sep 26 12:47:52 2019 +0000

    Merge "Cloudkitty InfluxDB Storage backend via Kolla-ansible" into stable/stein

commit 599cbdb8b92544c646928b140fac9cba37de2d98
Merge: b5f3a8d 2827341
Author: Zuul <email address hidden>
Date: Thu Sep 26 11:46:54 2019 +0000

    Merge "Fix kolla_toolbox error handling" into stable/stein

commit 12950be4ac06e5ab341958bd2cfbaac0893fc99f
Author: Mark Goddard <email address hidden>
Date: Wed Sep 11 18:15:02 2019 +0100

    Switch default cloudkitty storage backend to influxdb

    Backport: stein
...

I can also see that the cloudkitty influxdb is created.
01:30:47.424 ___________________________________________________________
01:30:47.424 < TASK [cloudkitty : Creating Cloudkitty influxdb database] >
01:30:47.424 -----------------------------------------------------------

In the influx container I can see that the cloudkitty database exists:
(influxdb)[influxdb@stack01 /]$ influx -host 192.168.41.13
^[^HConnected to http://192.168.41.13:8086 version 1.7.9
InfluxDB shell version: 1.7.9
> SHOW DATABASES
name: databases
name
----
_internal
cloudkitty
>

But the cloudkitty deployment still fails. In the cloudkity-api.log I see that it cannot find the cloudkitty influx database:

...
2019-10-31 13:57:44.670 26 WARNING cloudkitty.storage [-] V2 Storage is in beta. Its API is considered stable but its implementation may still evolve.
2019-10-31 13:57:44.828 26 CRITICAL cloudkitty [-] Unhandled error: InfluxDBClientError: database not found: cloudkitty
2019-10-31 13:57:44.828 26 ERROR cloudkitty Traceback (most recent call last):
2019-10-31 13:57:44.828 26 ERROR cloudkitty File "/var/lib/kolla/venv/bin/cloudkitty-storage-init", line 8, in <module>
2019-10-31 13:57:44.828 26 ERROR cloudkitty sys.exit(main())
2019-10-31 13:57:44.828 26 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/cli/storage.py", line 32, in main
2019-10-31 13:57:44.828 26 ERROR cloudkitty init_storage_backend()
2019-10-31 13:57:44.828 26 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/cli/storage.py", line 25, in init_storage_backend
2019-10-31 13:57:44.828 26 ERROR cloudkitty backend.init()
2019-10-31 13:57:44.828 26 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python2.7/site-packages/cloudkitty/storage/v2/influx.py", line 220, in init
2019-10-31 13:57:44.828 26 ERROR cloudkitty if not self._conn.retention_policy_exists(database, policy):
2019-10-31 13:...

[Martin Lukas (lxfozzibaer)]

Update: Just tried deploying with the master branch of kolla and kolla-ansible.
Failed with a different error, looks like they cannot be used currently:
fatal: [stack01]: FAILED! => {"msg": "The conditional check 'inventory_hostname in groups['prometheus-blackbox-exporter']' failed. The error was: error while evaluating conditional (inventory_hostname in groups['prometheus-blackbox-exporter']): Unable to look up a name or access an attribute in template string ({% if inventory_hostname in groups['prometheus-blackbox-exporter'] %} True {% else %} False {% endif %}).\nMake sure your variable name does not contain invalid characters like '-': argument of type 'AnsibleUndefined' is not iterable\n\nThe error appears to be in '/var/lib/jenkins/workspace/FullSetup/kolla/share/kolla-ansible/ansible/roles/prometheus/tasks/precheck.yml': line 132, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Checking free ports for Prometheus blackbox-exporter\n ^ here\n"}

[Mark Goddard (mgoddard)]

Looks like you need to update your inventory file for master.

[Martin Lukas (lxfozzibaer)]

You are right, sorry for that :-(.
Trying with an updated inventory now ...

[Martin Lukas (lxfozzibaer)]

Stopped testing with master as it requires too many changes.
Back to the original problem:
With kolla-ansible from git using branch stable/stein: Should cloudkitty deployment work? I constantly get the following error:
InfluxDBClientError: database not found: cloudkitty

Although the log shows the influx db creation task:
 ___________________________________________________________
< TASK [cloudkitty : Creating Cloudkitty influxdb database] >
 -----------------------------------------------------------
        \ ^__^
         \ (oo)\_______
            (__)\ )\/\
                ||----w |
                || ||

changed: [stack01]
changed: [stack02]

ok: [stack03]

(sorry for that cowsay thingy :-( ).

So I am currently stuck here, only option would be to disable cloudkitty, which is not what I want to do. Should it work in stable/stein or only in master branch currently?

[Mark Goddard (mgoddard)]

I would expect it to work in stable/stein. Do you have any config overrides for cloudkitty?

[Mark Goddard (mgoddard)]

Please check that the generated cloudkitty config has the expected config for influxdb.

[Martin Lukas (lxfozzibaer)]

No, I don't have any config overrides for cloudkitty.

cloudkitty.conf contains the following section:
[storage_influxdb]
database = cloudkitty
host = cloud.test
port = 8086

globals.yml only contains the following cloudkitty relevant files:
# grep cloudkitty globals.yml
enable_cloudkitty: "yes"
enable_horizon_cloudkitty: "{{ enable_cloudkitty | bool }}"

[Mark Goddard (mgoddard)]

I think I know the problem. The config section was renamed from storage_influx in Stein to storage_influxdb in train. Our backported patches use storage_influxdb in Stein. I'll propose a fix. Could you try changing that section name and restarting the containers, to see if it works?

[Mark Goddard (mgoddard)]

Martin, I have raised https://bugs.launchpad.net/kolla-ansible/+bug/1851352 to cover this new issue, please move discussion there.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/692966

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mark Goddard (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/692966
Reason: Turns out this is not correct - storage_influxdb is the right section name.

[Mark Goddard (mgoddard)]

Martin, I don't have more time to investigate this issue. Can you keep looking, given that you have a proper setup in which to test?

[Martin Lukas (lxfozzibaer)]

Mark, thanks for you support so far! I'll try to investigate further and apply the second fix you provided in the other bug report. We'll see how far I'll come ...

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.1.0 release.

[Kleyton (kleytonn)]

Tested and approved in kolla-ansible 8.1.0. Great job! Thanks

[Radosław Piliszek (yoctozepto)]

Thanks for confirming! :-)