Cloudkitty influx storage reports no database present

[Boris Lukashev]

Having gotten cloudkitty to deploy (wallaby, current), i'm seeing very inconsistent behavior, seeming to indicate that one of the influxDBs is not set up correctly in that ~1/3 horizon accesses throws an error, and the backend logs showing:
```
Server-side error: "database not found: cloudkitty". Detail:
Traceback (most recent call last):

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/wsmeext/pecan.py", line 85, in callfunction
    result = f(self, *args, **kwargs)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/api/v1/controllers/report.py", line 147, in summary
    result = storage.total(

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/storage/v2/influx.py", line 386, in total
    total = self._conn.get_total(metric_types, begin, end,

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/cloudkitty/storage/v2/influx.py", line 210, in get_total
    total = self._conn.query(query)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/influxdb/client.py", line 536, in query
    results = [

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/influxdb/client.py", line 537, in <listcomp>
    ResultSet(result, raise_errors=raise_errors)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/influxdb/resultset.py", line 25, in __init__
    raise InfluxDBClientError(self.error)

influxdb.exceptions.InfluxDBClientError: database not found: cloudkitty
: influxdb.exceptions.InfluxDBClientError: database not found: cloudkitty
```
I see the `[cloudkitty : Creating Cloudkitty influxdb database]` task execute against one of the three control nodes, but thinking that maybe its not replicating or something else "weird" is afoot. Unfortunately, `/var/log/kolla/influxdb/` is empty on the control nodes, so not sure what's up with that. Containers are definitely there though, and appear to have outputs logged to the Docker environment itself (`docker logs` works to show accesses, which show 200s from all 3 nodes):
```
[24/Jul/2021:18:14:04 +0000] "GET /query?db=cloudkitty&q=SHOW+RETENTION+POLICIES+ON+%22cloudkitty%22 HTTP/1.1" 200 139
```
```
[24/Jul/2021:18:13:56 +0000] "GET /query?db=cloudkitty&q=SHOW+DATABASES HTTP/1.1" 200 108
```
```
[24/Jul/2021:17:57:47 +0000] "GET /query?db=cloudkitty&q=SELECT+SUM%28qty%29+AS+qty%2C+SUM%28price%29+AS+rate+FROM+%22dataframes%22+WHERE+time+%3E%3D+%272021-07-01T00%3A00%3A00%2B00%3A00%27+AND+time+%3C+%272021-08-01T00%3A00%3A00%2B00%3A00%27+AND+%22project_id%22%3D%27c4a96de54323440ba046aad3c9c3e4d1%27+GROUP+BY+%22project_id%22%2C%22type%22%3B HTTP/1.1" 200 73
```

Not sure what to make of this - is influx actually running as a cluster here? I thought that was reserved for commercial use. I verified that mariadb also has the database on all members, so maybe its something weird going on in keystone? Anyone else seeing this?

[Mark Goddard (mgoddard)]

Influxdb is not clustered, so accessing it via haproxy with multiple servers is expected to fail in this way.

[Boris Lukashev (rageltman)]

Confusing? I didn't make any configuration changes - this is how kolla-ansible installed the stack. So should Kolla be deploying only a single IFDB then?

[Mark Goddard (mgoddard)]

Indeed, this setup is just plain broken as far as I can tell. A workaround would be to place only a single host in the influxdb group.

[Boris Lukashev (rageltman)]

Thanks,kinda wish i had misdiagnosed the problem, humbug. This seems a pretty major flaw in the design...
IIRC cloudkitty was kinda dead there for a while, how long since they came back into the fold (i was out of the deploying new stacks game for a couple of years)? Do they support any actually distributed/replicated back-ends? Docs seem to indicate influx-only and influx plays those we're-half-FOSS-but-really-pay-us-peons games we all love so much. Kolla-ansible doesnt seem to clean up after itself (no ability to remove components), so even altering the multinode config here and re-running with just one server wont fix this AFAIK. What's the official process for cleaning up stuff like this?

[Mark Goddard (mgoddard)]

Official cleanup process is docker rm -rf influxdb. You can also clean up the configs in /etc/kolla and any docker volumes.

Elasticsearch is supported since the Wallaby release.

[Boris Lukashev (rageltman)]

Thank you, updated config and seems to be working - will clean up influx shortly.
I would note that the YAML itself does not present ES as an option for this (i'm already using it for logging so quite convenient):
```
#######################
# Cloudkitty options
#######################
# Valid option is gnocchi
cloudkitty_collector_backend: "gnocchi"
# Valid options are 'sqlalchemy' or 'influxdb'. The default value is
# 'influxdb', which matches the default in Cloudkitty since the Stein release.
# When the backend is "influxdb", we also enable Influxdb.
# Also, when using 'influxdb' as the backend, we trigger the configuration/use
# of Cloudkitty storage backend version 2.
cloudkitty_storage_backend: "elasticsearch"
```