Cloudkitty fails to bootstrap container

Bug #1937352 reported by Boris Lukashev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
New
Undecided
Unassigned

Bug Description

When enabling cloudkitty on a deployed stack, i'm seeing:
```
TASK [cloudkitty : Running Cloudkitty bootstrap container] ***************************************************************************************************************************************************************
[WARNING]: The value "False" (type bool) was converted to "'False'" (type string). If this does not look like what you expect, quote the entire value to ensure it does not change.
fatal: [ctlnode0 -> ctlnode0]: FAILED! => {"changed": true, "msg": "Container exited with non-zero return code 1", "rc": 1, "stderr": "+ sudo -E kolla_set_configs\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO:__main__:Copying /var/lib/kolla/config_files/cloudkitty.conf to /etc/cloudkitty/cloudkitty.conf\nINFO:__main__:Setting permission for /etc/cloudkitty/cloudkitty.conf\nINFO:__main__:Copying /var/lib/kolla/config_files/wsgi-cloudkitty.conf to /etc/apache2/conf-enabled/wsgi-cloudkitty.conf\nINFO:__main__:Setting permission for /etc/apache2/conf-enabled/wsgi-cloudkitty.conf\nINFO:__main__:Writing out command to execute\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty\nINFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api.log\n++ cat /run_command\n+ CMD='apache2 -DFOREGROUND'\n+ ARGS=\n+ sudo kolla_copy_cacerts\n+ [[ ! -n '' ]]\n+ . kolla_extend_start\n++ [[ ! -d /var/log/kolla/cloudkitty ]]\n+++ stat -c %a /var/log/kolla/cloudkitty\n++ [[ 2755 != \\7\\5\\5 ]]\n++ chmod 755 /var/log/kolla/cloudkitty\n++ . /usr/local/bin/kolla_cloudkitty_extend_start\n+++ [[ -n 0 ]]\n+++ cloudkitty-dbsync upgrade\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\nINFO [alembic.runtime.migration] Context impl MySQLImpl.\nINFO [alembic.runtime.migration] Will assume non-transactional DDL.\n+++ cloudkitty-storage-init\n", "stderr_lines": ["+ sudo -E kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Copying /var/lib/kolla/config_files/cloudkitty.conf to /etc/cloudkitty/cloudkitty.conf", "INFO:__main__:Setting permission for /etc/cloudkitty/cloudkitty.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/wsgi-cloudkitty.conf to /etc/apache2/conf-enabled/wsgi-cloudkitty.conf", "INFO:__main__:Setting permission for /etc/apache2/conf-enabled/wsgi-cloudkitty.conf", "INFO:__main__:Writing out command to execute", "INFO:__main__:Setting permission for /var/log/kolla/cloudkitty", "INFO:__main__:Setting permission for /var/log/kolla/cloudkitty/cloudkitty-api.log", "++ cat /run_command", "+ CMD='apache2 -DFOREGROUND'", "+ ARGS=", "+ sudo kolla_copy_cacerts", "+ [[ ! -n '' ]]", "+ . kolla_extend_start", "++ [[ ! -d /var/log/kolla/cloudkitty ]]", "+++ stat -c %a /var/log/kolla/cloudkitty", "++ [[ 2755 != \\7\\5\\5 ]]", "++ chmod 755 /var/log/kolla/cloudkitty", "++ . /usr/local/bin/kolla_cloudkitty_extend_start", "+++ [[ -n 0 ]]", "+++ cloudkitty-dbsync upgrade", "INFO [alembic.runtime.migration] Context impl MySQLImpl.", "INFO [alembic.runtime.migration] Will assume non-transactional DDL.", "INFO [alembic.runtime.migration] Context impl MySQLImpl.", "INFO [alembic.runtime.migration] Will assume non-transactional DDL.", "INFO [alembic.runtime.migration] Context impl MySQLImpl.", "INFO [alembic.runtime.migration] Will assume non-transactional DDL.", "+++ cloudkitty-storage-init"], "stdout": "", "stdout_lines": []}

```
with gnocchi, influx, elastic, and prometheus enabled (monasca disabled). Default backends (none set):
```
$ egrep 'gnoc|cloudk' etc/kolla/globals.yml
enable_cloudkitty: "yes"
#enable_elasticsearch: "{{ 'yes' if enable_central_logging | bool or enable_osprofiler | bool or enable_skydive | bool or enable_monasca | bool or (enable_cloudkitty | bool and cloudkitty_storage_backend == 'elasticsearch') else 'no' }}"
enable_gnocchi: "yes"
enable_gnocchi_statsd: "yes"
#enable_horizon_cloudkitty: "{{ enable_cloudkitty | bool }}"
#enable_influxdb: "{{ enable_monasca | bool or (enable_cloudkitty | bool and cloudkitty_storage_backend == 'influxdb') }}"
ceph_gnocchi_keyring: "ceph.client.gnocchi.keyring"
ceph_gnocchi_user: "gnocchi"
ceph_gnocchi_pool_name: "gnocchi"
#gnocchi_backend_storage: "{% if enable_swift | bool %}swift{% else %}file{% endif %}"
gnocchi_backend_storage: "file"
gnocchi_incoming_storage: "{{ 'redis' if enable_redis | bool else '' }}"
# Valid option is gnocchi
#cloudkitty_collector_backend: "gnocchi"
#cloudkitty_storage_backend: "influxdb"
```
gnocchi is using a file backend because the container doenst have the relevant rados python lib in it (there's a bug out for that already).

Revision history for this message
Mark Goddard (mgoddard) wrote :

There may be logs in /var/log/kolla/cloudkitty/

Revision history for this message
Boris Lukashev (rageltman) wrote :

Cloudkitty is running, but the logs just claim there's no influxdb database for it:
```
2021-07-27 13:44:07.805 19 WARNING cloudkitty.storage [-] V2 Storage is in beta. Its API is considered stable but its implementation may still evolve.
2021-07-27 13:44:07.993 19 CRITICAL cloudkitty [-] Unhandled error: influxdb.exceptions.InfluxDBClientError: database not found: cloudkitty
2021-07-27 13:44:07.993 19 ERROR cloudkitty Traceback (most recent call last):
2021-07-27 13:44:07.993 19 ERROR cloudkitty File "/var/lib/kolla/venv/bin/cloudkitty-storage-init", line 8, in <module>
2021-07-27 13:44:07.993 19 ERROR cloudkitty sys.exit(main())
2021-07-27 13:44:07.993 19 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/cli/storage.py", line 30, in main
2021-07-27 13:44:07.993 19 ERROR cloudkitty init_storage_backend()
2021-07-27 13:44:07.993 19 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/cli/storage.py", line 23, in init_storage_backend
2021-07-27 13:44:07.993 19 ERROR cloudkitty backend.init()
2021-07-27 13:44:07.993 19 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/storage/v2/influx.py", line 283, in init
2021-07-27 13:44:07.993 19 ERROR cloudkitty if not self._conn.retention_policy_exists(database, policy):
2021-07-27 13:44:07.993 19 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/storage/v2/influx.py", line 109, in retention_policy_exists
2021-07-27 13:44:07.993 19 ERROR cloudkitty policies = self._conn.get_list_retention_policies(database)
2021-07-27 13:44:07.993 19 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python3.8/site-packages/influxdb/client.py", line 915, in get_list_retention_policies
2021-07-27 13:44:07.993 19 ERROR cloudkitty rsp = self.query(
2021-07-27 13:44:07.993 19 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python3.8/site-packages/influxdb/client.py", line 536, in query
2021-07-27 13:44:07.993 19 ERROR cloudkitty results = [
2021-07-27 13:44:07.993 19 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python3.8/site-packages/influxdb/client.py", line 537, in <listcomp>
2021-07-27 13:44:07.993 19 ERROR cloudkitty ResultSet(result, raise_errors=raise_errors)
2021-07-27 13:44:07.993 19 ERROR cloudkitty File "/var/lib/kolla/venv/lib/python3.8/site-packages/influxdb/resultset.py", line 25, in __init__
2021-07-27 13:44:07.993 19 ERROR cloudkitty raise InfluxDBClientError(self.error)
2021-07-27 13:44:07.993 19 ERROR cloudkitty influxdb.exceptions.InfluxDBClientError: database not found: cloudkitty
2021-07-27 13:44:07.993 19 ERROR cloudkitty
```
however, when connecting to influx on the admin VIP, i clearly see the database:
```
InfluxDB shell version: 1.8.6
> SHOW DATABASES
name: databases
name
----
_internal
cloudkitty
```
which i doc'd as https://bugs.launchpad.net/kolla-ansible/+bug/1937908

Revision history for this message
Mark Goddard (mgoddard) wrote :

It's a different symptom of the same issue, marked as a dup.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.