Insufficient monitoring of p_haproxy and vip__management corosync resources
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Invalid
|
Critical
|
Aleksey Zvyagintsev |
Bug Description
Detailed bug description:
Currently we have the following function to check status of haproxy, that basically checks that in the ps output we have process
haproxy_status() {
# check and make PID file dir
local PID_DIR="$( dirname ${PIDFILE} )"
if [ ! -d "${PID_DIR}" ] ; then
# no need to chown, root is user for haproxy
fi
if [ -n "${PIDFILE}" -a -f "${PIDFILE}" ]; then
# haproxy is probably running
# get pid from pidfile
if [ -n "${PID}" ]; then
fi
fi
# haproxy is not running
ocf_log info "haproxy daemon is not running"
return $OCF_NOT_RUNNING
}
Only one check, that process is running is not enough to ensure that haproxy works as expected.
Below you can find situations, when pcs will not know that resource is not functional.
Steps to reproduce:
(1) Simulation of an overloaded system
Let assume having too few or no CPU time for haproxy. This can be simulated by suspending the haproxy; In case of “kill -STOP $(pidof haproxy)“ on the haproxy owning the VIP, the haproxy instance would be suspended and therefore no OpenStack commands would be possible even though the haproxy is running thus CRM cannot recognize that VIP movement would be needed
(2) Simulation of a forwarding error on the network
This can be simulated by applying an “iptables -P INPUT DROP” in the haproxy namespace, thus the haproxy instance would be unresponsive but CRM cannot recognize that VIP movement would be needed since the haproxy itself is running and the VIP address responds to the ARP probes.
Expected results:
resource should be moved to another node
Actual result:
pcs reports that resource is running, no problems detected
tags: | added: customer-found |
Changed in mos: | |
status: | New → Confirmed |
Changed in fuel: | |
status: | New → Confirmed |
importance: | Undecided → Critical |
Changed in mos: | |
importance: | Undecided → Critical |
no longer affects: | mos |
Changed in fuel: | |
status: | Confirmed → Opinion |
Changed in fuel: | |
status: | Opinion → In Progress |
assignee: | nobody → Aleksey Zvyagintsev (azvyagintsev) |
milestone: | 9.x-updates → 9.2-mu-3 |
Pacemaker's goal it to monitor the service itself, not its ability to work properly.
We can imagine another case where haproxy works, but can't reach backends located on other controllers.
We can't demand deep insight from pacemaker scripts in such cases.
Especially case (2) is completely artificial.
I'm not sure this even fixable