haproxy doesn't stop by resource disable p_haproxy

[Ilya Shakhat]

Steps to reproduce:
1. pcs resource disable p_haproxy
2. check that 'pcs resource' reports that p_haproxy is stopped:
 Clone Set: clone_p_haproxy [p_haproxy]
     Stopped: [ node-2.domain.tld node-4.domain.tld node-5.domain.tld ]
It is expected that there are no haproxy processes in the system, however:
root@node-2:~# ps aux | grep hapr
haproxy 1191 0.0 0.0 35564 6752 ? Ss Aug19 0:42 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
haproxy 1457 0.0 0.0 35824 6784 ? Ss Aug19 0:42 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
haproxy 4529 0.0 0.0 36068 7148 ? Ss Aug19 0:51 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
haproxy 4750 0.0 0.0 36212 7272 ? Ss Aug19 1:03 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
haproxy 6590 0.2 0.0 38004 7832 ? Ss 13:32 0:04 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
haproxy 7158 0.4 0.0 45268 16204 ? Ss Aug19 7:07 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
haproxy 7637 0.0 0.0 36472 7400 ? Ss Aug19 1:12 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
root 7849 0.0 0.0 8864 648 pts/18 S+ 14:03 0:00 grep --color=auto hapr

[Ilya Shakhat (shakhat)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "189"
  build_id: "2015-08-18_17-24-26"
  nailgun_sha: "d0b727cdd0d8e7ce5e17e6ea1306d835bfdfb5e7"
  python-fuelclient_sha: "6ad5e0eb4dbbca6cade3444554606927ecd3f16f"
  fuel-agent_sha: "57145b1d8804389304cd04322ba0fb3dc9d30327"
  fuel-nailgun-agent_sha: "e01693992d7a0304d926b922b43f3b747c35964c"
  astute_sha: "e24ca066bf6160bc1e419aaa5d486cad1aaa937d"
  fuel-library_sha: "7c80eed2119260cc15a700068b9eb20ccc773926"
  fuel-ostf_sha: "235f21b230fea15724d625b2dc44ade0464527e2"
  fuelmain_sha: "c9dad194e82a60bf33060eae635fff867116a9ce"

[Bogdan Dobrelya (bogdando)]

UX for this issue is quite sad, so I believe this should be a high prio bug

[Bogdan Dobrelya (bogdando)]

reproduced
root@node-1:~# pgrep -fla haproxy
7487 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
root@node-1:~# OCF_ROOT=/usr/lib/ocf /usr/lib/ocf/resource.d/fuel/ns_haproxy stop
DEBUG: default: check_ns(): recieved netns list: haproxy
DEBUG: default: get_variables(): set up variables and PIDFILE name
DEBUG: default: check_ns(): recieved netns list: haproxy
DEBUG: default: get_variables(): set up variables and PIDFILE name
INFO: haproxy daemon is not running
INFO: Stopped haproxy daemon.
root@node-1:~# pgrep -fla haproxy
7487 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid

[Alex Schultz (alex-schultz)]

What version of the iso was used for this? There was a fix to stop this, https://review.openstack.org/#/c/214284/

[Bogdan Dobrelya (bogdando)]

related logging issue

[Bogdan Dobrelya (bogdando)]

Note, another related issue is that the haproxy processes left after the pcs disable or OCF stop command, would also remain running orphaned.

[Bogdan Dobrelya (bogdando)]

@Alex, your fix could make this bug resolved partially, but AFAICT, there is a flaw in the OCF script that removes the pid file leaving the haproxy process running.

[Alex Schultz (alex-schultz)]

Yes, there probably is a bug in the ocf script as well. I missed the version info when I first looked at this bug.

[Alex Schultz (alex-schultz)]

I think I tracked down the stop issue due to a leading space on the pid causing some argument errors which ended up keeping the process from being properly stopped.

root@node-1:~# OCF_ROOT=/usr/lib/ocf /usr/lib/ocf/resource.d/fuel/ns_haproxy stop
DEBUG: default: check_ns(): recieved netns list: haproxy
DEBUG: default: get_variables(): set up variables and PIDFILE name
DEBUG: ip netns exec haproxy /usr/sbin/haproxy
DEBUG: default: check_ns(): recieved netns list: haproxy
DEBUG: default: get_variables(): set up variables and PIDFILE name
INFO: haproxy daemon running
/usr/lib/ocf/resource.d/fuel/ns_haproxy: line 475: [: too many arguments
/usr/lib/ocf/resource.d/fuel/ns_haproxy: line 448: [: /proc/: binary operator expected
/usr/lib/ocf/resource.d/fuel/ns_haproxy: line 481: [: too many arguments
DEBUG: Delete pid file: /var/run/haproxy.pid with content 8637
INFO: Stopped haproxy daemon.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/215282

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/215282
Reason: https://review.openstack.org/#/c/215283

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/215283
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=58319254e501b6bf07261e588ceb08d56c439dbe
Submitter: Jenkins
Branch: master

commit 58319254e501b6bf07261e588ceb08d56c439dbe
Author: Sergii Golovatiuk <email address hidden>
Date: Thu Aug 20 20:46:41 2015 +0200

    Restart haproxy in ns properly

    Fix the regression introduced by
    I720aff6ff0515b2b0ceb4b6ce821e2eaff70688a as it tried to kill by
    processgroup without identifying it. PID != PGRP

    Additionally, ps -o pgid= <PID> may return a leading space which breaks
    the logic. This change removes the spaces from the ps results to ensure
    we can properly use the pgid. This change also updates the restart logic
    to not attempt to call the script within the namespace as that should be
    properly handled by the ocf script.

    Closes-Bug: 1487175
    Closes-Bug: 1487056

    Change-Id: I9030553ecbe6af05076e473f7d3a376fc8f5f660
    Signed-off-by: Sergii Golovatiuk <email address hidden>
    Co-Authored-By: Alex Schultz <email address hidden>