| 2015-12-11 12:45:02 |
Hugh Saunders |
description |
The cinder playbooks check if the cinder api port is responsive before making requests (good idea). However that port points at haproxy (or your lb of choice), which may have an open port before any backends are up, leading to a 503.
Example:
=====
**11:44:46** TASK: [os_cinder | Add in cinder devices types] *******************************
**11:44:46** skipping: [jenk-heat-477-node1_cinder_api_container-17e0fafe] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node2_cinder_api_container-6ff44cfa] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node3_cinder_api_container-a5bb2d91] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node2_cinder_scheduler_container-c83bb010] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node1_cinder_scheduler_container-cfec39a4] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node3_cinder_scheduler_container-0267c30c] => (item=cinder_backends|dictsort)
**11:44:48** failed: [jenk-heat-477-node4] => (item=(u'lvm', {u'volume_driver': u'cinder.volume.drivers.lvm.LVMISCSIDriver', u'volume_backend_name': u'LVM_iSCSI', u'volume_group': u'cinder-volumes'})) => {"changed": true, "cmd": ". /root/openrc\n cinder type-create \"lvm\"\n cinder type-key \"lvm\" set volume_backend_name=\"LVM_iSCSI\"", "delta": "0:00:01.612136", "end": "2015-12-11 11:44:48.305110", "item": ["lvm", {"volume_backend_name": "LVM_iSCSI", "volume_driver": "cinder.volume.drivers.lvm.LVMISCSIDriver", "volume_group": "cinder-volumes"}], "rc": 1, "start": "2015-12-11 11:44:46.692974", "warnings": []}
**11:44:48** stderr: ERROR: Service Unavailable (HTTP 503)
**11:44:48** ERROR: Service Unavailable (HTTP 503)
=====
Relevant section of code:
https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_cinder/tasks/cinder_backends.yml#L16-L22
There may well be others. |
The cinder playbooks check if the cinder api port is responsive before making requests (good idea). However that port is opened by haproxy (or your lb of choice) not the cinder api daemon. Haproxy may have an open port before any backends are up. This check passes because a port is open, but then when the next tasks attempts to access the service, haproxy gives 503 if there are no live backends.
Example:
=====
**11:44:46** TASK: [os_cinder | Add in cinder devices types] *******************************
**11:44:46** skipping: [jenk-heat-477-node1_cinder_api_container-17e0fafe] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node2_cinder_api_container-6ff44cfa] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node3_cinder_api_container-a5bb2d91] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node2_cinder_scheduler_container-c83bb010] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node1_cinder_scheduler_container-cfec39a4] => (item=cinder_backends|dictsort)
**11:44:46** skipping: [jenk-heat-477-node3_cinder_scheduler_container-0267c30c] => (item=cinder_backends|dictsort)
**11:44:48** failed: [jenk-heat-477-node4] => (item=(u'lvm', {u'volume_driver': u'cinder.volume.drivers.lvm.LVMISCSIDriver', u'volume_backend_name': u'LVM_iSCSI', u'volume_group': u'cinder-volumes'})) => {"changed": true, "cmd": ". /root/openrc\n cinder type-create \"lvm\"\n cinder type-key \"lvm\" set volume_backend_name=\"LVM_iSCSI\"", "delta": "0:00:01.612136", "end": "2015-12-11 11:44:48.305110", "item": ["lvm", {"volume_backend_name": "LVM_iSCSI", "volume_driver": "cinder.volume.drivers.lvm.LVMISCSIDriver", "volume_group": "cinder-volumes"}], "rc": 1, "start": "2015-12-11 11:44:46.692974", "warnings": []}
**11:44:48** stderr: ERROR: Service Unavailable (HTTP 503)
**11:44:48** ERROR: Service Unavailable (HTTP 503)
=====
Relevant section of code:
https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_cinder/tasks/cinder_backends.yml#L16-L22
There may well be others. |
|
