admin/nagios user setup fails on cs:influxdb-14

[Gareth Woolridge]

cs:influxdb-13 (latest yesterday) successfully created the admin influxdb user and dropped the creds in /root/.influx-auth. Also when related to nrpe the charm successfully created the nagios user dropping the creds in /var/lib/nagios/.influx-auth

The recent release of cs:influxdb-14 to the charmstore seems to have caused a regression here where the users and auth files do not get created causing mojo deployments to fail on verification:

2018-05-23 08:48:36 [ERROR] 1 FAIL on influxdb/0:
+ /usr/local/lib/nagios/plugins/check_influxdb.py
error authorizing query: create admin user first or disable authentication
CRITICAL: Exception while attempting to check InfluxDB
Traceback (most recent call last):
File "/usr/local/lib/nagios/plugins/check_influxdb.py", line 29, in <module>
output = subprocess.check_output(cmd)
File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
**kwargs).stdout
File "/usr/lib/python3.5/subprocess.py", line 708, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['influx', '-execute', 'show diagnostics', '-format', 'json']' returned non-zero exit status 1
2018-05-23 08:48:36 [ERROR] Nagios checks failed

[Tom Haddon (mthaddon)]

We should publish the previous version of the charm as cs:influxdb-15 until we can get this fixed, unless the fix is very quick.

[Jamon Camisso (jamon)]

Seeing this bug again intermittently. I tested it a few times across Trusty, Xenial, and Bionic and managed to trigger it (somehow) on Trusty & Xenial. It seems like some kind of race, because changing a config option and resolving the errored unit appears to generate the file.

[Haw Loeung (hloeung)]

It's probably to do with PPA or install source unavailable at the time. Any chance you could post snippet of your juju unit logs?

[Tom Haddon (mthaddon)]

Indeed, I see the following in the charm logs:

2018-07-09 20:20:42 [ERROR] 2018-07-09 20:18:21 INFO juju-log maintenance: Updating apt cache
[...]
2018-07-09 20:20:21 DEBUG install Err:5 https://repos.influxdata.com/ubuntu bionic InRelease
2018-07-09 20:20:21 DEBUG install Connection timed out after 120001 milliseconds
2018-07-09 20:20:24 DEBUG install Reading package lists...
2018-07-09 20:20:24 DEBUG install W: Failed to fetch https://repos.influxdata.com/ubuntu/dists/bionic/InRelease Connection timed out after 120001 milliseconds
2018-07-09 20:20:24 DEBUG install W: Some index files failed to download. They have been ignored, or old ones used instead.

[Tom Haddon (mthaddon)]

I believe this is actually related to https://bugs.launchpad.net/influxdb-charm/+bug/1781177. Even with the above message fixed (which was due to environment restrictions on access to repos.influxdata.com) we're still seeing issues on install - setting a model to retry hooks will make it succeed, but that's obviously not ideal.

[Tom Haddon (mthaddon)]

This should now be fixed as part of lp#1781177. Will mark as fix released.