[ovn-octavia-provider] HM not working for FIPs

[Luis Tomas Bolivar]

When an OVN Load Balancer has HealthMonitors associated to its pool/members, if a member is detected as error, there is not traffic being sent to it. However, if the OVN LoadBalancer has a FIP associated to the VIP, when using the FIP to access the Load Balancer, the traffic is loadbalanced also using the members that are in error status.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/864817
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/9cb8cd5054e64ab7fd13b4b63259b639b4f26c50
Submitter: "Zuul (22348)"
Branch: master

commit 9cb8cd5054e64ab7fd13b4b63259b639b4f26c50
Author: Luis Tomas Bolivar <email address hidden>
Date: Mon Nov 21 14:23:37 2022 +0100

    Make clear distinction between health check and health monitor

    There was quite a mix between (Octavia) Health Monitors and
    (OVN) Load Balancer Health Checks. This patch tries to make a
    more clear distinction between Octavia HM and OVN LB HCs.

    This patch also add a reference to the Octavia Health Monitors
    Ids to the external_ids of the OVN NB Load_Balancer entries.

    Related-Bug: #1997418

    Change-Id: Ib8499d7c4ea102e183ead31f063a3c0a70af6e23

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865906

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865907

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865908

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865909

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865909
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/b3d20d2204a901ab913b6a91faf3f83ee78d12b9
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b3d20d2204a901ab913b6a91faf3f83ee78d12b9
Author: Luis Tomas Bolivar <email address hidden>
Date: Mon Nov 21 14:23:37 2022 +0100

    Make clear distinction between health check and health monitor

    There was quite a mix between (Octavia) Health Monitors and
    (OVN) Load Balancer Health Checks. This patch tries to make a
    more clear distinction between Octavia HM and OVN LB HCs.

    This patch also add a reference to the Octavia Health Monitors
    Ids to the external_ids of the OVN NB Load_Balancer entries.

    Related-Bug: #1997418

    Change-Id: Ib8499d7c4ea102e183ead31f063a3c0a70af6e23
    (cherry picked from commit 9cb8cd5054e64ab7fd13b4b63259b639b4f26c50)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865906
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/fe3b69206f7d3b31a7149d2051451ddd6fdcc45a
Submitter: "Zuul (22348)"
Branch: stable/zed

commit fe3b69206f7d3b31a7149d2051451ddd6fdcc45a
Author: Luis Tomas Bolivar <email address hidden>
Date: Mon Nov 21 14:23:37 2022 +0100

    Make clear distinction between health check and health monitor

    There was quite a mix between (Octavia) Health Monitors and
    (OVN) Load Balancer Health Checks. This patch tries to make a
    more clear distinction between Octavia HM and OVN LB HCs.

    This patch also add a reference to the Octavia Health Monitors
    Ids to the external_ids of the OVN NB Load_Balancer entries.

    Related-Bug: #1997418

    Change-Id: Ib8499d7c4ea102e183ead31f063a3c0a70af6e23
    (cherry picked from commit 9cb8cd5054e64ab7fd13b4b63259b639b4f26c50)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865907
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/27ce0c0fe0b83589ceb7bf132d1f80fa5b2fbab6
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 27ce0c0fe0b83589ceb7bf132d1f80fa5b2fbab6
Author: Luis Tomas Bolivar <email address hidden>
Date: Mon Nov 21 14:23:37 2022 +0100

    Make clear distinction between health check and health monitor

    There was quite a mix between (Octavia) Health Monitors and
    (OVN) Load Balancer Health Checks. This patch tries to make a
    more clear distinction between Octavia HM and OVN LB HCs.

    This patch also add a reference to the Octavia Health Monitors
    Ids to the external_ids of the OVN NB Load_Balancer entries.

    Related-Bug: #1997418

    Change-Id: Ib8499d7c4ea102e183ead31f063a3c0a70af6e23
    (cherry picked from commit 9cb8cd5054e64ab7fd13b4b63259b639b4f26c50)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865908
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/130ed10bedf4100d03d1c38507471fadfdf77e72
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 130ed10bedf4100d03d1c38507471fadfdf77e72
Author: Luis Tomas Bolivar <email address hidden>
Date: Mon Nov 21 14:23:37 2022 +0100

    Make clear distinction between health check and health monitor

    There was quite a mix between (Octavia) Health Monitors and
    (OVN) Load Balancer Health Checks. This patch tries to make a
    more clear distinction between Octavia HM and OVN LB HCs.

    This patch also add a reference to the Octavia Health Monitors
    Ids to the external_ids of the OVN NB Load_Balancer entries.

    Related-Bug: #1997418

    Change-Id: Ib8499d7c4ea102e183ead31f063a3c0a70af6e23
    (cherry picked from commit 9cb8cd5054e64ab7fd13b4b63259b639b4f26c50)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/873860
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/ba4ea1134b1da4ce1dcd186d4962d12270699fef
Submitter: "Zuul (22348)"
Branch: master

commit ba4ea1134b1da4ce1dcd186d4962d12270699fef
Author: Luis Tomas Bolivar <email address hidden>
Date: Fri Feb 17 08:18:13 2023 +0100

    Ensure HM also apply to FIPs associated to LB VIPs

    Currently, if a FIP gets associated to a LB with HealthMonitors
    it is not included as a new OVN Load Balancer Health Checks. This
    means that if the VIP is used, traffic will not be redirected to
    the dead members, buit if the FIP is used there is no health checks
    being applied and traffic will reach dead members.

    This patch adds the extra functionality so that an extra OVN
    Load Balancer Health Check is created for the FIPs associated to
    the Load Balancer.

    Closes-Bug: #1997418

    Change-Id: Idbf1fb15076518092ce5fdaa57500d29342f51be

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875595

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875596

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875597

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875598

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875596
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/f95301ef5139d443cad17b1d211532cbdbd8a4dc
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit f95301ef5139d443cad17b1d211532cbdbd8a4dc
Author: Luis Tomas Bolivar <email address hidden>
Date: Fri Feb 17 08:18:13 2023 +0100

    Ensure HM also apply to FIPs associated to LB VIPs

    Currently, if a FIP gets associated to a LB with HealthMonitors
    it is not included as a new OVN Load Balancer Health Checks. This
    means that if the VIP is used, traffic will not be redirected to
    the dead members, buit if the FIP is used there is no health checks
    being applied and traffic will reach dead members.

    This patch adds the extra functionality so that an extra OVN
    Load Balancer Health Check is created for the FIPs associated to
    the Load Balancer.

    Closes-Bug: #1997418

    Change-Id: Idbf1fb15076518092ce5fdaa57500d29342f51be
    (cherry picked from commit ba4ea1134b1da4ce1dcd186d4962d12270699fef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875598
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/8aff4293cd22af0e9c0adb7db6b0470ab2df493b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 8aff4293cd22af0e9c0adb7db6b0470ab2df493b
Author: Luis Tomas Bolivar <email address hidden>
Date: Fri Feb 17 08:18:13 2023 +0100

    Ensure HM also apply to FIPs associated to LB VIPs

    Currently, if a FIP gets associated to a LB with HealthMonitors
    it is not included as a new OVN Load Balancer Health Checks. This
    means that if the VIP is used, traffic will not be redirected to
    the dead members, buit if the FIP is used there is no health checks
    being applied and traffic will reach dead members.

    This patch adds the extra functionality so that an extra OVN
    Load Balancer Health Check is created for the FIPs associated to
    the Load Balancer.

    Closes-Bug: #1997418

    Change-Id: Idbf1fb15076518092ce5fdaa57500d29342f51be
    (cherry picked from commit ba4ea1134b1da4ce1dcd186d4962d12270699fef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875597
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/461320ad319e2628b5e73ce86a5683fd24b933b6
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 461320ad319e2628b5e73ce86a5683fd24b933b6
Author: Luis Tomas Bolivar <email address hidden>
Date: Fri Feb 17 08:18:13 2023 +0100

    Ensure HM also apply to FIPs associated to LB VIPs

    Currently, if a FIP gets associated to a LB with HealthMonitors
    it is not included as a new OVN Load Balancer Health Checks. This
    means that if the VIP is used, traffic will not be redirected to
    the dead members, buit if the FIP is used there is no health checks
    being applied and traffic will reach dead members.

    This patch adds the extra functionality so that an extra OVN
    Load Balancer Health Check is created for the FIPs associated to
    the Load Balancer.

    Closes-Bug: #1997418

    Change-Id: Idbf1fb15076518092ce5fdaa57500d29342f51be
    (cherry picked from commit ba4ea1134b1da4ce1dcd186d4962d12270699fef)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovn-octavia-provider 4.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875595
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/b22039aaf1a16d60d5b4617436dfc81ce080560c
Submitter: "Zuul (22348)"
Branch: stable/zed

commit b22039aaf1a16d60d5b4617436dfc81ce080560c
Author: Luis Tomas Bolivar <email address hidden>
Date: Fri Feb 17 08:18:13 2023 +0100

    Ensure HM also apply to FIPs associated to LB VIPs

    Currently, if a FIP gets associated to a LB with HealthMonitors
    it is not included as a new OVN Load Balancer Health Checks. This
    means that if the VIP is used, traffic will not be redirected to
    the dead members, buit if the FIP is used there is no health checks
    being applied and traffic will reach dead members.

    This patch adds the extra functionality so that an extra OVN
    Load Balancer Health Check is created for the FIPs associated to
    the Load Balancer.

    Closes-Bug: #1997418

    Change-Id: Idbf1fb15076518092ce5fdaa57500d29342f51be
    (cherry picked from commit ba4ea1134b1da4ce1dcd186d4962d12270699fef)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovn-octavia-provider 1.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovn-octavia-provider 2.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovn-octavia-provider 3.1.0 release.

[Kurt Garloff (kgarloff)]

https://bugs.launchpad.net/neutron/+bug/1956035
looks like a duplicate to this one.

[Mathias (mattfechner)]

i think this bug is unfortunately not to 100 % percent solved, the healthmonitor, not closed the connection to the errored member

So in Detail:

The ERROR operating status of the members did not cause the LB to no longer distribute requests to the ERROR'ed members -- but in a installation released with ovn-octavia-provider we observe in a example, 50%
of the requests fail due to 2/4 ERROR'ed members.
Release of this environment is Zed.

we tracked this bug in our project here:

https://github.com/SovereignCloudStack/issues/issues/268

[Fernando Royo (froyoredhat)]

Hi Mathias,

I make some test after seeing your comment and I have reproduced the issue when the FIP is added after the HM already was created, if the FIP is added before the HM is created it works fine.

I also reproduce the issue on master and on all stables branches (untill wallaby).

We would create a patch to cover this case.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/886603

[Sven Kieske (s-kieske)]

I can't reset the state from "Fix Released". I guess this is the reason that the bot doesn't post updates regarding:

https://review.opendev.org/c/openstack/ovn-octavia-provider/+/886603

I wanted to ask if it's okay if I create backports for these?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovn-octavia-provider wallaby-eom release.