Checking if a port is up doesn't make sense when pointing at a load balancer

[Hugh Saunders]

The cinder playbooks check if the cinder api port is responsive before making requests (good idea). However that port is opened by haproxy (or your lb of choice) not the cinder api daemon. Haproxy may have an open port before any backends are up. This check passes because a port is open, but then when the next tasks attempts to access the service, haproxy gives 503 if there are no live backends.

Example:

=====
**11:44:46** TASK: [os_cinder | Add in cinder devices types] *******************************
**11:44:46** skipping: [jenk-heat-477-node1_cinder_api_container-17e0fafe] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node2_cinder_api_container-6ff44cfa] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node3_cinder_api_container-a5bb2d91] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node2_cinder_scheduler_container-c83bb010] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node1_cinder_scheduler_container-cfec39a4] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node3_cinder_scheduler_container-0267c30c] => (item=cinder_backends|dictsort)
**11:44:48** failed: [jenk-heat-477-node4] => (item=(u'lvm', {u'volume_driver': u'cinder.volume.drivers.lvm.LVMISCSIDriver', u'volume_backend_name': u'LVM_iSCSI', u'volume_group': u'cinder-volumes'})) => {"changed": true, "cmd": ". /root/openrc\n cinder type-create \"lvm\"\n cinder type-key \"lvm\" set volume_backend_name=\"LVM_iSCSI\"", "delta": "0:00:01.612136", "end": "2015-12-11 11:44:48.305110", "item": ["lvm", {"volume_backend_name": "LVM_iSCSI", "volume_driver": "cinder.volume.drivers.lvm.LVMISCSIDriver", "volume_group": "cinder-volumes"}], "rc": 1, "start": "2015-12-11 11:44:46.692974", "warnings": []}
**11:44:48** stderr: ERROR: Service Unavailable (HTTP 503)
**11:44:48** ERROR: Service Unavailable (HTTP 503)
=====

Relevant section of code:

https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_cinder/tasks/cinder_backends.yml#L16-L22

There may well be others.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/256414
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=eab4b15aeb1204989355705fe2ecdf184d8bbb8b
Submitter: Jenkins
Branch: master

commit eab4b15aeb1204989355705fe2ecdf184d8bbb8b
Author: Hugh Saunders <email address hidden>
Date: Fri Dec 11 13:10:41 2015 +0000

    Use http request to check cinder api availability

    Using a port open check is not reliable when going through a load
    balancer.

    httplib2 is added as it is required by the Ansible uri module.

    Change-Id: I0e489e8f671b262c56c1442e7af85fd53c061598
    Closes-bug: #1525208
    Closes-Bug: #1510990

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: liberty
Review: https://review.openstack.org/276784

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/276784
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=9d03db957aca27f28af994131dd25a1a70bd05c8
Submitter: Jenkins
Branch: liberty

commit 9d03db957aca27f28af994131dd25a1a70bd05c8
Author: Hugh Saunders <email address hidden>
Date: Fri Dec 11 13:10:41 2015 +0000

    Use http request to check cinder api availability

    Using a port open check is not reliable when going through a load
    balancer.

    httplib2 is added as it is required by the Ansible uri module.

    Change-Id: I0e489e8f671b262c56c1442e7af85fd53c061598
    Closes-bug: #1525208
    Closes-Bug: #1510990
    (cherry picked from commit eab4b15aeb1204989355705fe2ecdf184d8bbb8b)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 13.0.0 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 12.0.11 release.