[Ussuri - Octavia] check_amphorae reports empty list - NRPE: Unable to read output

[Pedro Guimarães]

Focal / Ussuri deployment

Test failing:
$ /usr/lib/nagios/plugins/check_nrpe -H <target> -c check_octavia_amphorae
NRPE: Unable to read output

$ sudo -u nagios /usr/local/lib/nagios/plugins/check_octavia.py --check amphorae
Traceback (most recent call last):
  File "/usr/local/lib/nagios/plugins/check_octavia.py", line 328, in <module>
    main()
  File "/usr/local/lib/nagios/plugins/check_octavia.py", line 322, in main
    status, message = process_checks(args)
  File "/usr/local/lib/nagios/plugins/check_octavia.py", line 263, in process_checks
    return nagios_exit(args, checks[args.check](connection))
  File "/usr/local/lib/nagios/plugins/check_octavia.py", line 186, in check_amphorae
    bad_amp = [
  File "/usr/local/lib/nagios/plugins/check_octavia.py", line 192, in <listcomp>
    if item["status"] in bad_status_list
KeyError: 'status'

When running check_octavia.py manually on openstack-service-checks, it gets a 200 response with empty body:

b'{"amphorae": [{}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}], "amphorae_links": []}'

Looking into openstackclients snap code:
https://opendev.org/x/snap-openstackclients/src/branch/master/snapcraft.yaml

It seems to use python-octaviaclient instead of openstack's.
In the snap case, it is working correctly.

[Przemyslaw Hausman (phausman)]

Possible patch (that works for me on Focal+Ussuri OpenStack deployment):

--- /usr/local/lib/nagios/plugins/check_octavia.py 2021-02-23 10:53:58.535026981 +0000
+++ /usr/local/lib/nagios/plugins/check_octavia_phausman.py 2021-02-23 10:53:41.843180345 +0000
@@ -171,15 +171,7 @@
 def check_amphorae(connection):
     """Check amphorae status."""
     lb_mgr = connection.load_balancer
-
- resp = lb_mgr.get("/v2/octavia/amphorae")
- # python api is not available yet, use url
- if resp.status_code != 200:
- return [(NAGIOS_STATUS_WARNING, "amphorae api not working")]
-
- data = json.loads(resp.content)
- # ouput is like {"amphorae": [{...}, {...}, ...]}
- items = data.get("amphorae", [])
+ items = lb_mgr.amphorae()

     # raise CRITICAL for ERROR status
     bad_status_list = ["ERROR"]

[Jeff Hillman (jhillman)]

I am also hitting this bug. Przemyslaw's patch worked for me.

[Vladimir Grevtsev (vlgrevtsev)]

+1 for the comments above, still actual.

[Nobuto Murata (nobuto)]

Attaching ~field-high.

[Jose Guedez (jfguedez)]

This seems related to the version of python3-openstackclient, since the charm is relying on the version of the operating system. The library added support for octavia/amphorae queries directly in the version included in Focal. However, calling lb_mgr.amphorae() is not supported in Bionic.

This means that the charm needs to check what is supported. Long term it probably would be better to have the charm bundle python libs instead of relying on OS packages that can vary significantly between LTS versions

[Adam Dyess (addyess)]

An alternate work-around that worked for me in focal/ussuri was to simply change

- resp = lb_mgr.get("/v2/octavia/amphorae")
+ resp = lb_mgr.get("/octavia/amphorae")

I saw in the octavia access logs the following

127.0.0.1 - - [24/Sep/2021:13:36:31 +0000] "GET /v2.0/v2/octavia/amphorae HTTP/1.1" 200 334 "-" "openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10"

Perhaps this would be functional in bionic etc?

[Vladimir Grevtsev (vlgrevtsev)]

FTR, the fix is currently available in cs:~bootstack-charmers-next/openstack-service-checks-3.

[Andrea Ieri (aieri)]

This is also available in (at least) cs:openstack-service-checks-9