Distro: OpenStack-Ansible
Version: Zed
Point: Master
In a multi-node deployment with multiple controller nodes, a recent patch to Keystone role results in an attempt to hit the Keystone endpoint via VIP before the service has been re-enabled in haproxy.
-=-=-=-=-
TASK [include_tasks] ********************************************************************************************************************************
included: /opt/openstack-ansible/playbooks/common-tasks/haproxy-endpoint-manage.yml for infra1_keystone_container-4e93eb38
TASK [Set haproxy service state] ********************************************************************************************************************
changed: [infra1_keystone_container-4e93eb38 -> loadbalancer1(10.0.236.150)] => (item=loadbalancer1)
TASK [Configure container] **************************************************************************************************************************
included: /opt/openstack-ansible/playbooks/common-tasks/os-lxc-container-setup.yml for infra1_keystone_container-4e93eb38
TASK [Set default bind mounts (bind var/log)] *******************************************************************************************************
ok: [infra1_keystone_container-4e93eb38]
...
TASK [systemd_service : Place the systemd timer] ****************************************************************************************************
skipping: [infra1_keystone_container-4e93eb38] => (item={'service_name': 'keystone-wsgi-public', 'enabled': True, 'state': 'started', 'execstarts': '
/openstack/venvs/uwsgi-25.1.0.dev68-python3/bin/uwsgi --autoload --ini /etc/uwsgi/keystone-wsgi-public.ini', 'execreloads': '/openstack/venvs/uwsgi-2
5.1.0.dev68-python3/bin/uwsgi --reload /run/keystone-wsgi-public/uwsgi/keystone-wsgi-public.pid', 'config_overrides': {}})
TASK [systemd_service : Place the systemd socket] ***************************************************************************************************
TASK [systemd_service : Reload systemd on unit change] **********************************************************************************************
TASK [systemd_service : include_tasks] **************************************************************************************************************
included: /etc/ansible/roles/systemd_service/tasks/systemd_load.yml for infra1_keystone_container-4e93eb38 => (item={'service_name': 'keystone-wsgi-p
ublic', 'enabled': True, 'state': 'started', 'execstarts': '/openstack/venvs/uwsgi-25.1.0.dev68-python3/bin/uwsgi --autoload --ini /etc/uwsgi/keyston
e-wsgi-public.ini', 'execreloads': '/openstack/venvs/uwsgi-25.1.0.dev68-python3/bin/uwsgi --reload /run/keystone-wsgi-public/uwsgi/keystone-wsgi-publ
ic.pid', 'config_overrides': {}})
TASK [systemd_service : Load service keystone-wsgi-public] ******************************************************************************************
ok: [infra1_keystone_container-4e93eb38] => (item=)
TASK [systemd_service : Load timer keystone-wsgi-public] ********************************************************************************************
skipping: [infra1_keystone_container-4e93eb38] => (item=)
TASK [systemd_service : Load socket] ****************************************************************************************************************
TASK [os_keystone : Flush handlers] *****************************************************************************************************************
TASK [os_keystone : Wait for service to be up] ******************************************************************************************************
FAILED - RETRYING: [infra1_keystone_container-4e93eb38]: Wait for service to be up (12 retries left).
FAILED - RETRYING: [infra1_keystone_container-4e93eb38]: Wait for service to be up (11 retries left).
FAILED - RETRYING: [infra1_keystone_container-4e93eb38]: Wait for service to be up (10 retries left).
FAILED - RETRYING: [infra1_keystone_container-4e93eb38]: Wait for service to be up (9 retries left).
FAILED - RETRYING: [infra1_keystone_container-4e93eb38]: Wait for service to be up (8 retries left).
FAILED - RETRYING: [infra1_keystone_container-4e93eb38]: Wait for service to be up (7 retries left).
FAILED - RETRYING: [infra1_keystone_container-4e93eb38]: Wait for service to be up (6 retries left).
FAILED - RETRYING: [infra1_keystone_container-4e93eb38]: Wait for service to be up (5 retries left).
FAILED - RETRYING: [infra1_keystone_container-4e93eb38]: Wait for service to be up (4 retries left).
FAILED - RETRYING: [infra1_keystone_container-4e93eb38]: Wait for service to be up (3 retries left).
FAILED - RETRYING: [infra1_keystone_container-4e93eb38]: Wait for service to be up (2 retries left).
FAILED - RETRYING: [infra1_keystone_container-4e93eb38]: Wait for service to be up (1 retries left).
fatal: [infra1_keystone_container-4e93eb38]: FAILED! => {"attempts": 12, "cache_control": "no-cache", "changed": false, "connection": "close", "conte
nt_length": "107", "content_type": "text/html", "elapsed": 0, "msg": "Status code was 503 and not [300]: HTTP Error 503: Service Unavailable", "redir
ected": false, "status": 503, "url": "http://10.0.236.150:5000"}
-=-=-=-=-
This seems to be related to the change made here:
https://github.com/openstack/openstack-ansible-os_keystone/commit/05c64f7651a93bfa987a939fce680c3d4b13df30
The re-enabling of the node in haproxy is a post_task[1] for os-keystone-install, which hasn't happened yet when the "Wait for service to be up" task is run. The service is reachable directly but not (yet) via the VIP. Reverting the patch seems to resolve this.
[1] https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-keystone-install.yml#L98
Bugfix proposed in https:/
Would be great if you could test it out.