keepalived haproxy check script too simple

[Chenjun Shen]

location: playbooks/vars/configs/keepalived_haproxy.yml

This part:

keepalived_scripts:
  haproxy_check_script:
    check_script: "killall -0 haproxy"

We found "killall -0 haproxy" is too simple, and might cause problem.

For example:
In our envs, we provide load balance as a service in neutron. In the neutron-agent container, there is also a haproxy process running there. Even we stop the haproxy on the control nodes, haproxy process in the container still lists in the ps output of control nodes.

Then "killall -0 haproxy" will still give exit code = 0, which makes keepalived not recognize that haproxy process is down.

We suggest to change the check script as below, which only checks the process running locally, not influenced by the one in the container:

keepalived_scripts:
  haproxy_check_script:
    check_script: "service haproxy status"

If this would be accepted, I could also provide a patch or whatever as wished.

Best regards,
Chenjun Shen

[Jean-Philippe Evrard (jean-philippe-evrard)]

If collocating haproxies, that could indeed be a problem (which is gonna be happening with neutron metadata or LBaaS).

We could move keepalived and haproxy into containers, or change the check script to be namespaced. I haven't checked how "service haproxy status" works under systemd. Probably worth investigating too.

[Chenjun Shen (cshen)]

Hi Jean,

"service haproxy status" worked for my company under Ubuntu 16.04.

Alternative will be "kill -0 `cat /var/run/haproxy.pid`" under Ubuntu/Systemd, which also worked for us.

[Chenjun Shen (cshen)]

Patch submitted.

https://review.openstack.org/451409

[Jean-Philippe Evrard (jean-philippe-evrard)]

Thank you!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/451409
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=6d757cade5fe4fcefa83ff27f134787e6881e6d2
Submitter: Jenkins
Branch: master

commit 6d757cade5fe4fcefa83ff27f134787e6881e6d2
Author: Chenjun Shen <email address hidden>
Date: Wed Mar 29 15:37:09 2017 +0200

    Update haproxy check script in keepalived.

    Fixes-Bug: 1674742

    Change-Id: If4bdc9d8882ba902e14508e3c28a78b8c0f74b09

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/463869

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/463869
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=3ac6ae0d4be619596a04c76d70bb7156df0beb95
Submitter: Jenkins
Branch: stable/ocata

commit 3ac6ae0d4be619596a04c76d70bb7156df0beb95
Author: Chenjun Shen <email address hidden>
Date: Wed Mar 29 15:37:09 2017 +0200

    Update haproxy check script in keepalived.

    Fixes-Bug: 1674742

    Change-Id: If4bdc9d8882ba902e14508e3c28a78b8c0f74b09
    (cherry picked from commit 6d757cade5fe4fcefa83ff27f134787e6881e6d2)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/513814

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/513814
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=0513d1cd6858d71fc50e65c14c987b0edfc128ae
Submitter: Zuul
Branch: stable/newton

commit 0513d1cd6858d71fc50e65c14c987b0edfc128ae
Author: Chenjun Shen <email address hidden>
Date: Wed Mar 29 15:37:09 2017 +0200

    Update haproxy check script in keepalived.

    Fixes-Bug: 1674742

    Change-Id: If4bdc9d8882ba902e14508e3c28a78b8c0f74b09
    (cherry picked from commit 6d757cade5fe4fcefa83ff27f134787e6881e6d2)