Pacemaker default configurations are unreasonable

[Dan Xu]

Tripleo uses pacemaker's default configurations of virtual ip monitor interval and haproxy monitor interval, which is
sudo pcs resource show ip-192.168.36.17
Resource: ip-192.168.36.17 (class=ocf provider=heartbeat type=IPaddr2)
    Attributes: ip=192.168.36.17 cidr_netmask=32
    Operations: start interval=0s timeout=20s (ip-192.168.36.17-start-interval-0s)
                stop interval=0s timeout=20s (ip-192.168.36.17-stop-interval-0s)
                monitor interval=10s timeout=20s (ip-192.168.36.17-monitor-interval-10s)

sudo pcs resource show haproxy
Resource: haproxy (class=systemd type=haproxy)
    Operations: start interval=0s timeout=200s (haproxy-start-interval-0s)
                stop interval=0s timeout=200s (haproxy-stop-interval-0s)
                monitor interval=60s (haproxy-monitor-interval-60s)

The first configuration will make the outage time of openstack service to be more than 20s.
The second configuration may make the outage time of haproxy processes to be 60s (the worst case).

All these make the HA not very effective (outage time is too long).

[Michele Baldessari (michele)]

Can you expand here in detail what failure scenario is suboptimal?

When the VIP moves it does not really matter if haproxy still needs to be monitored or not, because connections will go to the moved VIP. Are you saying the VIP timeouts need to be lower?

[Tim Rozet (trozet)]

Yes I think the VIP monitor interval and timeout needs to be lower. It should be as low as possible, without adverse effects. Right now if the interval is at 10s, and timeout 20s, it seems possible that a VIP failure could take 30 seconds to failover. Right? I hope we can lower this to at least only a couple seconds.

[Dan Xu (xudan)]

Agree with Tim Rozet.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/554672

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/554672
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=8975100f773a2dcfdc39eaf6e979a763dae82866
Submitter: Zuul
Branch: master

commit 8975100f773a2dcfdc39eaf6e979a763dae82866
Author: Michele Baldessari <email address hidden>
Date: Tue Mar 20 20:29:51 2018 +0100

    Allow VIP resource to have customized ops

    Introduce meta_params and op_params which will be
    passed to the pcs resource create comand when creating
    the VIPs. This allows us to specify custom ops for VIPs
    in the following manner, by setting hiera. E.g.:
    tripleo::profile::pacemaker::haproxy_bundle::op_params: 'start timeout=200s stop timeout=200s monitor timeout=5s'

    Change-Id: I9a1c700051fc6dfc302e1d94347df2956442354e
    Depends-On: Iadf0cd3805f72141563707f43130945c9d362f5c
    Related-Bug: #1719540

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/687234

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/687235

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/687234
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=27b0847c54f536950be2aaa210ff6a036a3e5fc5
Submitter: Zuul
Branch: stable/rocky

commit 27b0847c54f536950be2aaa210ff6a036a3e5fc5
Author: Michele Baldessari <email address hidden>
Date: Tue Mar 20 20:29:51 2018 +0100

    Allow VIP resource to have customized ops

    Introduce meta_params and op_params which will be
    passed to the pcs resource create comand when creating
    the VIPs. This allows us to specify custom ops for VIPs
    in the following manner, by setting hiera. E.g.:
    tripleo::profile::pacemaker::haproxy_bundle::op_params: 'start timeout=200s stop timeout=200s monitor timeout=5s'

    Change-Id: I9a1c700051fc6dfc302e1d94347df2956442354e
    Depends-On: Iadf0cd3805f72141563707f43130945c9d362f5c
    Related-Bug: #1719540
    (cherry picked from commit 8975100f773a2dcfdc39eaf6e979a763dae82866)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/687235
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=a341443018bff04d50960fcc81b8be0ebad337a7
Submitter: Zuul
Branch: stable/queens

commit a341443018bff04d50960fcc81b8be0ebad337a7
Author: Michele Baldessari <email address hidden>
Date: Tue Mar 20 20:29:51 2018 +0100

    Allow VIP resource to have customized ops

    Introduce meta_params and op_params which will be
    passed to the pcs resource create comand when creating
    the VIPs. This allows us to specify custom ops for VIPs
    in the following manner, by setting hiera. E.g.:
    tripleo::profile::pacemaker::haproxy_bundle::op_params: 'start timeout=200s stop timeout=200s monitor timeout=5s'

    Change-Id: I9a1c700051fc6dfc302e1d94347df2956442354e
    Depends-On: Iadf0cd3805f72141563707f43130945c9d362f5c
    Related-Bug: #1719540
    (cherry picked from commit 8975100f773a2dcfdc39eaf6e979a763dae82866)
    (cherry picked from commit 27b0847c54f536950be2aaa210ff6a036a3e5fc5)