keystone fails with haproxy in maint mode

[Kyle L. Henderson]

Deploying from the master level (pike) to 3 controllers and 1 compute node, the deployment fails on keystone because haproxy is in maintenance mode while keystone is trying to "Ensure service tenant".

In openstack-ansible at the start of deploying keystone it 'disables' haproxy by using the haproxy-endpoint-manage task. However, haproxy is already 'down' and issuing a disable request puts it into maintenance mode.

TASK [Set haproxy service state] ***********************************************
Tuesday 04 April 2017 17:48:56 +0000 (0:00:00.096) 0:42:27.799 *********
changed: [node_192_168_10_64_keystone_container-bf838211 -> 172.29.236.64] => (item=node_192_168_10_64)
changed: [node_192_168_10_64_keystone_container-bf838211 -> 172.29.236.78] => (item=node_192_168_10_78)
changed: [node_192_168_10_64_keystone_container-bf838211 -> 172.29.236.62] => (item=node_192_168_10_62)

In the haproxy logs:

Apr 4 17:48:57 localhost haproxy[9878]: Server keystone_service-back/node_192_168_10_64_keystone_container-bf838211 was DOWN and now enters maintenance.

via: >echo "show stat" | socat /var/run/haproxy.stat stdio | grep -i keystone

keystone_admin-front-1,FRONTEND,,,0,0,4096,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,28,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,
keystone_admin-front-2,FRONTEND,,,0,1,4096,6,2658,1272,0,0,0,,,,,OPEN,,,,,,,,,1,29,0,,,,0,0,0,1,,,,0,0,0,0,6,0,,0,1,6,,,0,0,0,0,,,,,,,,
keystone_admin-back,node_192_168_10_64_keystone_container-bf838211,0,0,0,0,,0,0,0,,0,,0,0,0,0,MAINT,1,1,0,1,1,1764,1764,,1,30,1,,0,,2,0,,0,L4CON,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
keystone_admin-back,node_192_168_10_78_keystone_container-b8e8b96e,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,1764,1764,,1,30,2,,0,,2,0,,0,L4CON,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,Connection refused,,0,0,0,0,
keystone_admin-back,node_192_168_10_62_keystone_container-415c7f56,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,1764,1764,,1,30,3,,0,,2,0,,0,L4CON,,1,0,0,0,0,0,0,0,,,,0,0,,,,,-1,Connection refused,,0,0,0,0,
keystone_admin-back,BACKEND,0,0,0,1,820,6,2658,1272,0,0,,6,0,0,0,DOWN,0,0,0,,1,1764,1764,,1,30,0,,0,,1,0,,1,,,,0,0,0,0,6,0,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,
keystone_service-front-1,FRONTEND,,,0,0,4096,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,31,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,
keystone_service-front-2,FRONTEND,,,0,0,4096,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,32,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,
keystone_service-back,node_192_168_10_64_keystone_container-bf838211,0,0,0,0,,0,0,0,,0,,0,0,0,0,MAINT,1,1,0,1,1,1763,1763,,1,33,1,,0,,2,0,,0,L4CON,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
keystone_service-back,node_192_168_10_78_keystone_container-b8e8b96e,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,1763,1763,,1,33,2,,0,,2,0,,0,L4CON,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,Connection refused,,0,0,0,0,
keystone_service-back,node_192_168_10_62_keystone_container-415c7f56,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,1763,1763,,1,33,3,,0,,2,0,,0,L4CON,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,Connection refused,,0,0,0,0,

Haproxy stays in maintenance mode until it's either recycled or it's manually taken out of maintenance mode. This causes the deployment of keystone to fail since it can't operate against the service.

[Kyle L. Henderson (kyleh)]

Looks like this is the commit that started disabling haproxy prior to installing keystone.

https://github.com/openstack/openstack-ansible/commit/257b60222cc970fecc3a06956621822a16be9bdb

[Jean-Philippe Evrard (jean-philippe-evrard)]

Work already started by Logan there: https://review.openstack.org/#/c/438224/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438224
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=d13e2ebf77772f86b04447bc643636b25b863095
Submitter: Jenkins
Branch: master

commit d13e2ebf77772f86b04447bc643636b25b863095
Author: Logan V <email address hidden>
Date: Sat Feb 25 20:03:51 2017 -0600

    Bypass LB for service setup

    The keystone container can communicate with itself directly for
    service setup in case the load balancer is in maintenance mode
    while the service is being deployed.

    Closes-Bug: #1680075
    Depends-On: I6ecfc422b0e8a7a5fadea08389c1268009ac7f2a
    Change-Id: I379b68682a240d729765da74b3a9ec55df0920ee

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/469553

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/469598

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/469598
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=1c469f978bf7940acd474ff2c7da67f08bad741b
Submitter: Jenkins
Branch: stable/newton

commit 1c469f978bf7940acd474ff2c7da67f08bad741b
Author: Logan V <email address hidden>
Date: Sat Feb 25 20:03:51 2017 -0600

    Bypass LB for service setup

    The keystone container can communicate with itself directly for
    service setup in case the load balancer is in maintenance mode
    while the service is being deployed.

    Closes-Bug: #1680075
    Depends-On: I6ecfc422b0e8a7a5fadea08389c1268009ac7f2a
    Change-Id: I379b68682a240d729765da74b3a9ec55df0920ee
    (cherry picked from commit d13e2ebf77772f86b04447bc643636b25b863095)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/469553
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=c841625c66fa3ca38a9a11f05ece762107598099
Submitter: Jenkins
Branch: stable/ocata

commit c841625c66fa3ca38a9a11f05ece762107598099
Author: Logan V <email address hidden>
Date: Sat Feb 25 20:03:51 2017 -0600

    Bypass LB for service setup

    The keystone container can communicate with itself directly for
    service setup in case the load balancer is in maintenance mode
    while the service is being deployed.

    Closes-Bug: #1680075
    Depends-On: I6ecfc422b0e8a7a5fadea08389c1268009ac7f2a
    Change-Id: I379b68682a240d729765da74b3a9ec55df0920ee
    (cherry picked from commit d13e2ebf77772f86b04447bc643636b25b863095)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_keystone 16.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_keystone 14.2.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_keystone 15.1.5 release.