Support rally checks on HTTPS environments

[Alvaro Uria]

On environments where Keystone runs over TLS, rally fails with:
"""
Max retries exceeded with url: /v3 (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
"""

1) /var/lib/nagios/nagios.novarc does NOT have a OS_CACERT environment variable, because it relies on the configured certificate in /usr/local/share/ca-certificates/openstack-service-checks.crt

2) fcbtest.rallyinit script calls:
rally deployment create --fromenv --name snap_generated

Since the environment doesn't have the OS_CACERT variable, "rally deployment config" shows: https://pastebin.ubuntu.com/p/ftXkktCCG2/ (https_cacert would have been configured if OS_CACERT would have existed)

3) Once the rally deployment environment is recreated, "fcbtest.rally verify start" works as expected

* Fixes needed:
- lib_openstack_service_checks.py method _load_envvars needs to manually import OS_CACERT if the keystone endpoint (in OS_AUTH_URL) is found to be "https". However, OS_CACERT needs to point to a different location (see below)
- Copy /usr/local/share/ca-certificates/openstack-service-checks.crt to /home/nagiososc/snap/fcbtest/current/.rally/ssl/openstack-service-checks.crt (the snap has access to HOME, but since it is strictly confined, it doesn't have access anywhere else)
- run_rally.py should also import OS_CACERT pointing to the "nagiososc" $HOME instead of /usr/local/share/ca-certificates

[Alvaro Uria (aluria)]

I forgot to mention that /etc/juju-proxy.conf variables need to exist if the cloud is behind a proxy, or the test images won't be downloaded and checks will fail.

I ran out of time to verify if "juju model-config" options are propagated on all juju versions (it worked on a lab but not in a production cloud).

[Alvaro Uria (aluria)]

related to bug 1835433

[Paul Goins (vultaire)]

Relevant part of "juju model-config" on one of the charm dev machines: https://pastebin.ubuntu.com/p/ycQDj8JV4V/

The above was set before deploying anything.

Unfortunately, /etc/juju-proxy.conf ends up completely empty.

[Paul Goins (vultaire)]

As far as I can tell:

* The charm does have code to generate /etc/juju-proxy.conf. However, this is deprecated. It's based upon the snap_proxy config setting, which should no longer be used.

* Rather, the proxies for snap should be set via snap directly. These values can be retrieved via "snap get core proxy", and are set via the snap-<proto>-proxy model config variables. (I've confirmed this on one of the charm dev machines.)

* However, even with this set, I'm getting an error like this, as extracted from the output of the check_rally nrpe plugin: Failed to download image. Possibly there is no connection to Internet. Error: HTTPConnectionPool(host='download.cirros-cloud.net', port=80): Max retries exceeded with url: /0.3.5/cirros-0.3.5-x86_64-disk.img (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb30d94a320>: Failed to establish a new connection: [Errno 101] Network is unreachable',)).

[Paul Goins (vultaire)]

Sorry, my last comment was off base and incorrect in several ways. Let me correct it:

* The charm does try to use /etc/juju-proxy.conf if present. It doesn't try to generate it. However, in my case /etc/juju-proxy.conf is indeed empty.

* While proxies for snap should indeed be set via snap directly, that's not the reason for the failure I'm encountering with retrieving images via rally. Snap is not involved based on what I see. I think I linked snap in my brain based upon the other stuff going on in this ticket.

* The error I'm getting above is different. It's a proxy error, but it's likely because I've set {apt,juju,snap}-*-proxy, which are all application-specific. There's no app-specific setting for rally that I'm aware of, and it expects to use /etc/juju-proxy.conf. I'm guessing that's generated via the legacy http-proxy and https-proxy settings, but juju won't let me set both at the same time.

[Paul Goins (vultaire)]

Yes; the issue was indeed that for rally I need to specify the normal proxy vars rather than the per-app ones. The "legacy" proxy vars will generate /etc/juju-proxy.conf.

[Paul Goins (vultaire)]

I believe I've confirmed this; now looking to confirm details and the proposed fixes.

[Paul Goins (vultaire)]

Basically everything seems as Andrea says here. I'm not 100% sure whether adding OS_CACERT to run_rally.py is needed, but it does seem necessary prior to running fcbtest.rallyinit.

[Paul Goins (vultaire)]

Correction, not Andrea, but Alvaro. Sorry!

[Paul Goins (vultaire)]

I'm running my code by a peer first; MR is coming soon.

[Eric Chen (eric-chen)]

This charm is no longer being actively maintained. Please consider using the new Canonical Observability Stack instead.
(https://charmhub.io/topics/canonical-observability-stack)
I will close this new feature request.