volume only deployment creates non-functional api haproxy backend

[Andrea Ieri]

Deploying a volume only unit generates incorrect haproxy config.

To reproduce:

juju deploy cinder cinder-volume --config enabled-services=volume
juju add-relation cinder-volume mysql
juju add-relation cinder-volume rabbitmq-server

The above yields the following haproxy.cfg stanza:

backend cinder_api_<ip>
    balance leastconn
    server cinder-volume-1 <ip>:8766 check

The backend is however not functional as cinder-api is not started and nothing is listening on port 8766

This would be rather harmless, except that when our nagios haproxy check validates backends, it will alert for that bogus entry.

[Joe Guo (guoqiao)]

Hi, One of customer's alerts is blocked by this bug, is there any update, please?

[Ryan Farrell (whereisrysmind)]

This bug was lacking the proper subscriptions and has been long standing, so I've added field-high and canonical-boostack-cres.

The effect is that we have a persistent false alerts in Nagios for this particular cinder configuration.

Workaround:
In the meantime, we can manually remove the nrpe check for haproxy_servers from the relation:
$ juju run -u cinder-ssd/1 -- relation-ids nrpe-external-master
nrpe-external-master:170
$ juju run -u cinder-ssd/1 -- relation-list -r170
nrpe/23
juju run -u nrpe/23 -- relation-get -r170 - cinder-ssd/1
monitors: |
  monitors:
    remote:
      nrpe:
        apache2: {command: check_apache2}
        cinder-volume: {command: check_cinder-volume}
        haproxy: {command: check_haproxy}
        haproxy_queue: {command: check_haproxy_queue}
        haproxy_servers: {command: check_haproxy_servers}
        memcached: {command: check_memcached}
primary: "True"
private-address: 10.36.1.231
#write monitors section to a file
$ juju run -u cinder-ssd/1 -- relation-set -r170 monitors="$(cat monitors.txt)"

[Alvaro Uria (aluria)]

Relations with this service are:

~$ juju config cinder-ssd enabled-services
volume
~$ juju status cinder-ssd --relations | egrep cinder-ssd:
cinder-ssd:cinder-volume-service nova-cloud-controller:cinder-volume-service cinder regular
cinder-ssd:cluster cinder-ssd:cluster cinder-ha peer
cinder-ssd:juju-info filebeat:beats-host juju-info subordinate
cinder-ssd:nrpe-external-master nrpe:nrpe-external-master nrpe-external-master subordinate
glance:image-service cinder-ssd:image-service glance regular
mysql:shared-db cinder-ssd:shared-db mysql-shared regular
rabbitmq-server:amqp cinder-ssd:amqp rabbitmq regular
ssd-storage:storage-backend cinder-ssd:storage-backend cinder-backend subordinate

COMMON_PACKAGES includes apache2 and haproxy. Those packages should be only installed, configured and managed if cinder-api service is enabled.

Same about nrpe.add_haproxy_checks (in update_nrpe_config function). Method needs to be called when cinder-api service is enabled in the unit.

[Alvaro Uria (aluria)]

zaza-openstack lib needs to be updated to support "enabled-services" option. Proposed PR can be found at [1].

1. https://github.com/openstack-charmers/zaza-openstack-tests/pull/337

[Alvaro Uria (aluria)]

Fix manually tested together with nagios and nrpe. See attachment.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738651

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738651
Committed: https://git.openstack.org/cgit/openstack/charm-cinder/commit/?id=71f968faf1f5d8e9a307e7f1d391a699875ed5c1
Submitter: Zuul
Branch: master

commit 71f968faf1f5d8e9a307e7f1d391a699875ed5c1
Author: Alvaro Uria <email address hidden>
Date: Fri Jun 26 15:38:40 2020 +0200

    Ensure API not configured on volume-only units

    COMMON_PACKAGES included haproxy and apache2. However, enabled-services
    can request a unit to only run cinder-volume. Cinder API related
    services should not run or get monitored.

    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/337

    Change-Id: I69f15c3fd164f7114f5498d100b2832caf93fb00
    Closes-Bug: #1779310