haproxy fails and then works on subsquent runs

Bug #1973242 reported by admin0
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Undecided
Unassigned

Bug Description

This is not a blocker, but a nuisance and exists on the latest 24.2.0 as well.

#variables

haproxy_keepalived_external_vip_cidr: "10.60.0.99/24"
haproxy_keepalived_internal_vip_cidr: "172.29.236.9/22"
haproxy_keepalived_external_interface: br-corp
haproxy_keepalived_internal_interface: br-mgmt
haproxy_stats_enabled: True
haproxy_stats_bind_address: "cloud.domain.com"
haproxy_ssl_self_signed_regen: false
haproxy_user_ssl_cert: /opt/ssl/cert.pem
haproxy_user_ssl_key: /opt/ssl/key.pem

openstack-ansible haproxy-install.yml

#run1

TASK [haproxy_server : meta] *******************************************************************************************************************************

RUNNING HANDLER [keepalived : restart keepalived] **********************************************************************************************************
changed: [r2c1]

RUNNING HANDLER [keepalived : reload keepalived] ***********************************************************************************************************
fatal: [r2c1]: FAILED! => {"changed": true, "cmd": ["systemctl", "reload", "keepalived.service"], "delta": "0:00:00.006272", "end": "2022-05-12 20:21:06.103979", "msg": "non-zero return code", "rc": 1, "start": "2022-05-12 20:21:06.097707", "stderr": "keepalived.service is not active, cannot reload.", "stderr_lines": ["keepalived.service is not active, cannot reload."], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [haproxy_server : regen pem] ***************************************************************************************************************
failed: [r2c1] (item=cloud.domain.com) => {"ansible_loop_var": "item", "changed": true, "cmd": "cat /etc/haproxy/ssl/haproxy_r2c1-cloud.domain.com.crt /etc/haproxy/ssl/haproxy_r2c1-cloud.domain.com-ca.crt /etc/haproxy/ssl/haproxy_r2c1-cloud.domain.com.key > /etc/haproxy/ssl/haproxy_r2c1-cloud.domain.com.pem\n", "delta": "0:00:00.003777", "end": "2022-05-12 20:21:06.313199", "item": "cloud.domain.com", "msg": "non-zero return code", "rc": 1, "start": "2022-05-12 20:21:06.309422", "stderr": "cat: /etc/haproxy/ssl/haproxy_r2c1-cloud.domain.com-ca.crt: No such file or directory", "stderr_lines": ["cat: /etc/haproxy/ssl/haproxy_r2c1-cloud.domain.com-ca.crt: No such file or directory"], "stdout": "", "stdout_lines": []}
failed: [r2c1] (item=172.29.236.9) => {"ansible_loop_var": "item", "changed": true, "cmd": "cat /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9.crt /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9-ca.crt /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9.key > /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9.pem\n", "delta": "0:00:00.003539", "end": "2022-05-12 20:21:06.489924", "item": "172.29.236.9", "msg": "non-zero return code", "rc": 1, "start": "2022-05-12 20:21:06.486385", "stderr": "cat: /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9-ca.crt: No such file or directory", "stderr_lines": ["cat: /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9-ca.crt: No such file or directory"], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [haproxy_server : Regenerate haproxy configuration] ****************************************************************************************
changed: [r2c1]

RUNNING HANDLER [haproxy_server : Get package facts] *******************************************************************************************************
ok: [r2c1]

RUNNING HANDLER [haproxy_server : Get package facts] *******************************************************************************************************
ok: [r2c1]

RUNNING HANDLER [haproxy_server : Restart rsyslog] *********************************************************************************************************
changed: [r2c1]

RUNNING HANDLER [haproxy_server : Reload haproxy] **********************************************************************************************************
changed: [r2c1]

RUNNING HANDLER [pki : cert installed] *********************************************************************************************************************

NO MORE HOSTS LEFT *****************************************************************************************************************************************

PLAY RECAP *************************************************************************************************************************************************
r2c1 : ok=47 changed=24 unreachable=0 failed=2 skipped=17 rescued=0 ignored=0

run2 # first one will pass, but fail in 2nd one

RUNNING HANDLER [keepalived : reload keepalived] ***********************************************************************************************************
fatal: [r2c2]: FAILED! => {"changed": true, "cmd": ["systemctl", "reload", "keepalived.service"], "delta": "0:00:00.012108", "end": "2022-05-12 20:32:27.600247", "msg": "non-zero return code", "rc": 1, "start": "2022-05-12 20:32:27.588139", "stderr": "keepalived.service is not active, cannot reload.", "stderr_lines": ["keepalived.service is not active, cannot reload."], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [haproxy_server : regen pem] ***************************************************************************************************************
failed: [r2c2] (item=cloud.domain.com) => {"ansible_loop_var": "item", "changed": true, "cmd": "cat /etc/haproxy/ssl/haproxy_r2c2-cloud.domain.com.crt /etc/haproxy/ssl/haproxy_r2c2-cloud.domain.com-ca.crt /etc/haproxy/ssl/haproxy_r2c2-cloud.domain.com.key > /etc/haproxy/ssl/haproxy_r2c2-cloud.domain.com.pem\n", "delta": "0:00:00.003372", "end": "2022-05-12 20:32:27.820339", "item": "cloud.domain.com", "msg": "non-zero return code", "rc": 1, "start": "2022-05-12 20:32:27.816967", "stderr": "cat: /etc/haproxy/ssl/haproxy_r2c2-cloud.domain.com-ca.crt: No such file or directory", "stderr_lines": ["cat: /etc/haproxy/ssl/haproxy_r2c2-cloud.domain.com-ca.crt: No such file or directory"], "stdout": "", "stdout_lines": []}
failed: [r2c2] (item=172.29.236.9) => {"ansible_loop_var": "item", "changed": true, "cmd": "cat /etc/haproxy/ssl/haproxy_r2c2-172.29.236.9.crt /etc/haproxy/ssl/haproxy_r2c2-172.29.236.9-ca.crt /etc/haproxy/ssl/haproxy_r2c2-172.29.236.9.key > /etc/haproxy/ssl/haproxy_r2c2-172.29.236.9.pem\n", "delta": "0:00:00.006515", "end": "2022-05-12 20:32:28.113107", "item": "172.29.236.9", "msg": "non-zero return code", "rc": 1, "start": "2022-05-12 20:32:28.106592", "stderr": "cat: /etc/haproxy/ssl/haproxy_r2c2-172.29.236.9-ca.crt: No such file or directory", "stderr_lines": ["cat: /etc/haproxy/ssl/haproxy_r2c2-172.29.236.9-ca.crt: No such file or directory"], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [haproxy_server : Regenerate haproxy configuration] ****************************************************************************************
changed: [r2c2]

RUNNING HANDLER [haproxy_server : Get package facts] *******************************************************************************************************
ok: [r2c2]

RUNNING HANDLER [haproxy_server : Restart rsyslog] *********************************************************************************************************
changed: [r2c2]

RUNNING HANDLER [haproxy_server : Reload haproxy] **********************************************************************************************************
changed: [r2c2]

RUNNING HANDLER [pki : cert installed] *********************************************************************************************************************

NO MORE HOSTS LEFT *****************************************************************************************************************************************

PLAY RECAP *************************************************************************************************************************************************
r2c1 : ok=42 changed=0 unreachable=0 failed=0 skipped=17 rescued=0 ignored=0
r2c2 : ok=47 changed=24 unreachable=0 failed=2 skipped=17 rescued=0 ignored=0

run3 # first 2 pass, fails on the 3rd one

TASK [haproxy_server : Make log socket available to chrooted filesystem] ***********************************************************************************
changed: [r2c3]

TASK [haproxy_server : meta] *******************************************************************************************************************************

RUNNING HANDLER [keepalived : restart keepalived] **********************************************************************************************************
changed: [r2c3]

RUNNING HANDLER [keepalived : reload keepalived] ***********************************************************************************************************
fatal: [r2c3]: FAILED! => {"changed": true, "cmd": ["systemctl", "reload", "keepalived.service"], "delta": "0:00:00.008997", "end": "2022-05-12 20:35:20.579170", "msg": "non-zero return code", "rc": 1, "start": "2022-05-12 20:35:20.570173", "stderr": "keepalived.service is not active, cannot reload.", "stderr_lines": ["keepalived.service is not active, cannot reload."], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [haproxy_server : regen pem] ***************************************************************************************************************
failed: [r2c3] (item=cloud.domain.com) => {"ansible_loop_var": "item", "changed": true, "cmd": "cat /etc/haproxy/ssl/haproxy_r2c3-cloud.domain.com.crt /etc/haproxy/ssl/haproxy_r2c3-cloud.domain.com-ca.crt /etc/haproxy/ssl/haproxy_r2c3-cloud.domain.com.key > /etc/haproxy/ssl/haproxy_r2c3-cloud.domain.com.pem\n", "delta": "0:00:00.003953", "end": "2022-05-12 20:35:20.793785", "item": "cloud.domain.com", "msg": "non-zero return code", "rc": 1, "start": "2022-05-12 20:35:20.789832", "stderr": "cat: /etc/haproxy/ssl/haproxy_r2c3-cloud.domain.com-ca.crt: No such file or directory", "stderr_lines": ["cat: /etc/haproxy/ssl/haproxy_r2c3-cloud.domain.com-ca.crt: No such file or directory"], "stdout": "", "stdout_lines": []}
failed: [r2c3] (item=172.29.236.9) => {"ansible_loop_var": "item", "changed": true, "cmd": "cat /etc/haproxy/ssl/haproxy_r2c3-172.29.236.9.crt /etc/haproxy/ssl/haproxy_r2c3-172.29.236.9-ca.crt /etc/haproxy/ssl/haproxy_r2c3-172.29.236.9.key > /etc/haproxy/ssl/haproxy_r2c3-172.29.236.9.pem\n", "delta": "0:00:00.003311", "end": "2022-05-12 20:35:20.964046", "item": "172.29.236.9", "msg": "non-zero return code", "rc": 1, "start": "2022-05-12 20:35:20.960735", "stderr": "cat: /etc/haproxy/ssl/haproxy_r2c3-172.29.236.9-ca.crt: No such file or directory", "stderr_lines": ["cat: /etc/haproxy/ssl/haproxy_r2c3-172.29.236.9-ca.crt: No such file or directory"], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [haproxy_server : Regenerate haproxy configuration] ****************************************************************************************
changed: [r2c3]

RUNNING HANDLER [haproxy_server : Get package facts] *******************************************************************************************************
ok: [r2c3]

RUNNING HANDLER [haproxy_server : Restart rsyslog] *********************************************************************************************************
changed: [r2c3]

RUNNING HANDLER [haproxy_server : Reload haproxy] **********************************************************************************************************
changed: [r2c3]

RUNNING HANDLER [pki : cert installed] *********************************************************************************************************************

NO MORE HOSTS LEFT *****************************************************************************************************************************************

PLAY RECAP *************************************************************************************************************************************************
r2c1 : ok=42 changed=0 unreachable=0 failed=0 skipped=17 rescued=0 ignored=0
r2c2 : ok=42 changed=0 unreachable=0 failed=0 skipped=17 rescued=0 ignored=0
r2c3 : ok=47 changed=24 unreachable=0 failed=2 skipped=17 rescued=0 ignored=0

run4 # finally no breaks .. all success \o/

TASK [haproxy_server : Prevent SELinux from preventing haproxy from binding to arbitrary ports] ************************************************************

TASK [haproxy_server : Create log directory if it does not exist] ******************************************************************************************
ok: [r2c3]

TASK [haproxy_server : Ensure empty file is availble to bind mount log socket] *****************************************************************************
ok: [r2c3]

TASK [haproxy_server : Make log socket available to chrooted filesystem] ***********************************************************************************
ok: [r2c3]

TASK [haproxy_server : meta] *******************************************************************************************************************************

TASK [haproxy_server : include_tasks] **********************************************************************************************************************

PLAY RECAP *************************************************************************************************************************************************
r2c1 : ok=42 changed=0 unreachable=0 failed=0 skipped=17 rescued=0 ignored=0
r2c2 : ok=42 changed=0 unreachable=0 failed=0 skipped=17 rescued=0 ignored=0
r2c3 : ok=42 changed=0 unreachable=0 failed=0 skipped=17 rescued=0 ignored=0

Revision history for this message
Jonathan Rosser (jrosser) wrote :

I don't think we can say much about the haproxy error, because there is also an error from keepalived which is deployed first.

["keepalived.service is not active, cannot reload."]

This needs to be understood and fixed first.

Revision history for this message
admin0 (shashi-eu) wrote :
Download full text (12.6 KiB)

## cert expired, so update cert with new content
## openstack-ansible haproxy-install.yml

TASK [haproxy_server : Create log directory if it does not exist] ******************************************************************************************
ok: [r2c1]

TASK [haproxy_server : Ensure empty file is availble to bind mount log socket] *****************************************************************************
ok: [r2c1]

TASK [haproxy_server : Make log socket available to chrooted filesystem] ***********************************************************************************
ok: [r2c1]

TASK [haproxy_server : meta] *******************************************************************************************************************************

RUNNING HANDLER [haproxy_server : regen pem] ***************************************************************************************************************
failed: [r2c1] (item=cloud.domain.com) => {"ansible_loop_var": "item", "changed": true, "cmd": "cat /etc/haproxy/ssl/haproxy_r2c1-cloud.domain.com.crt /etc/haproxy/ssl/haproxy_r2c1-cloud.domain.com-ca.crt /etc/haproxy/ssl/haproxy_r2c1-cloud.domain.com.key > /etc/haproxy/ssl/haproxy_r2c1-cloud.domain.com.pem\n", "delta": "0:00:00.004872", "end": "2022-08-08 10:19:36.218365", "item": "cloud.domain.com", "msg": "non-zero return code", "rc": 1, "start": "2022-08-08 10:19:36.213493", "stderr": "cat: /etc/haproxy/ssl/haproxy_r2c1-cloud.domain.com-ca.crt: No such file or directory", "stderr_lines": ["cat: /etc/haproxy/ssl/haproxy_r2c1-cloud.domain.com-ca.crt: No such file or directory"], "stdout": "", "stdout_lines": []}
failed: [r2c1] (item=172.29.236.9) => {"ansible_loop_var": "item", "changed": true, "cmd": "cat /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9.crt /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9-ca.crt /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9.key > /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9.pem\n", "delta": "0:00:00.004155", "end": "2022-08-08 10:19:36.428401", "item": "172.29.236.9", "msg": "non-zero return code", "rc": 1, "start": "2022-08-08 10:19:36.424246", "stderr": "cat: /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9-ca.crt: No such file or directory", "stderr_lines": ["cat: /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9-ca.crt: No such file or directory"], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [pki : cert installed] *********************************************************************************************************************

NO MORE HOSTS LEFT *****************************************************************************************************************************************

PLAY RECAP *************************************************************************************************************************************************
r2c1 : ok=42 changed=1 unreachable=0 failed=1 skipped=17 rescued=0 ignored=0
r2c2 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
r2c3 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

EXIT NOTICE [Playbook execution failure]...

Revision history for this message
Dmitriy Rabotyagov (noonedeadpunk) wrote (last edit ):

I believe this patch should fix the bug https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/852399

Clean forgot to mention the bug when posting the change. Would be great if you could check on it.

Changed in openstack-ansible:
status: New → In Progress
Changed in openstack-ansible:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.