[ovn-octavia-provider] HM not working for FIPs

Bug #1997418 reported by Luis Tomas Bolivar
28
This bug affects 6 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
Luis Tomas Bolivar

Bug Description

When an OVN Load Balancer has HealthMonitors associated to its pool/members, if a member is detected as error, there is not traffic being sent to it. However, if the OVN LoadBalancer has a FIP associated to the VIP, when using the FIP to access the Load Balancer, the traffic is loadbalanced also using the members that are in error status.

Changed in neutron:
assignee: nobody → Luis Tomas Bolivar (ltomasbo)
tags: added: ovn-octavia-provider
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ovn-octavia-provider (master)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/864817
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/9cb8cd5054e64ab7fd13b4b63259b639b4f26c50
Submitter: "Zuul (22348)"
Branch: master

commit 9cb8cd5054e64ab7fd13b4b63259b639b4f26c50
Author: Luis Tomas Bolivar <email address hidden>
Date: Mon Nov 21 14:23:37 2022 +0100

    Make clear distinction between health check and health monitor

    There was quite a mix between (Octavia) Health Monitors and
    (OVN) Load Balancer Health Checks. This patch tries to make a
    more clear distinction between Octavia HM and OVN LB HCs.

    This patch also add a reference to the Octavia Health Monitors
    Ids to the external_ids of the OVN NB Load_Balancer entries.

    Related-Bug: #1997418

    Change-Id: Ib8499d7c4ea102e183ead31f063a3c0a70af6e23

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ovn-octavia-provider (stable/zed)

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865906

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ovn-octavia-provider (stable/yoga)

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865907

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ovn-octavia-provider (stable/xena)

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865908

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ovn-octavia-provider (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865909

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ovn-octavia-provider (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865909
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/b3d20d2204a901ab913b6a91faf3f83ee78d12b9
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b3d20d2204a901ab913b6a91faf3f83ee78d12b9
Author: Luis Tomas Bolivar <email address hidden>
Date: Mon Nov 21 14:23:37 2022 +0100

    Make clear distinction between health check and health monitor

    There was quite a mix between (Octavia) Health Monitors and
    (OVN) Load Balancer Health Checks. This patch tries to make a
    more clear distinction between Octavia HM and OVN LB HCs.

    This patch also add a reference to the Octavia Health Monitors
    Ids to the external_ids of the OVN NB Load_Balancer entries.

    Related-Bug: #1997418

    Change-Id: Ib8499d7c4ea102e183ead31f063a3c0a70af6e23
    (cherry picked from commit 9cb8cd5054e64ab7fd13b4b63259b639b4f26c50)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ovn-octavia-provider (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865906
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/fe3b69206f7d3b31a7149d2051451ddd6fdcc45a
Submitter: "Zuul (22348)"
Branch: stable/zed

commit fe3b69206f7d3b31a7149d2051451ddd6fdcc45a
Author: Luis Tomas Bolivar <email address hidden>
Date: Mon Nov 21 14:23:37 2022 +0100

    Make clear distinction between health check and health monitor

    There was quite a mix between (Octavia) Health Monitors and
    (OVN) Load Balancer Health Checks. This patch tries to make a
    more clear distinction between Octavia HM and OVN LB HCs.

    This patch also add a reference to the Octavia Health Monitors
    Ids to the external_ids of the OVN NB Load_Balancer entries.

    Related-Bug: #1997418

    Change-Id: Ib8499d7c4ea102e183ead31f063a3c0a70af6e23
    (cherry picked from commit 9cb8cd5054e64ab7fd13b4b63259b639b4f26c50)

tags: added: in-stable-zed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ovn-octavia-provider (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865907
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/27ce0c0fe0b83589ceb7bf132d1f80fa5b2fbab6
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 27ce0c0fe0b83589ceb7bf132d1f80fa5b2fbab6
Author: Luis Tomas Bolivar <email address hidden>
Date: Mon Nov 21 14:23:37 2022 +0100

    Make clear distinction between health check and health monitor

    There was quite a mix between (Octavia) Health Monitors and
    (OVN) Load Balancer Health Checks. This patch tries to make a
    more clear distinction between Octavia HM and OVN LB HCs.

    This patch also add a reference to the Octavia Health Monitors
    Ids to the external_ids of the OVN NB Load_Balancer entries.

    Related-Bug: #1997418

    Change-Id: Ib8499d7c4ea102e183ead31f063a3c0a70af6e23
    (cherry picked from commit 9cb8cd5054e64ab7fd13b4b63259b639b4f26c50)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ovn-octavia-provider (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/865908
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/130ed10bedf4100d03d1c38507471fadfdf77e72
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 130ed10bedf4100d03d1c38507471fadfdf77e72
Author: Luis Tomas Bolivar <email address hidden>
Date: Mon Nov 21 14:23:37 2022 +0100

    Make clear distinction between health check and health monitor

    There was quite a mix between (Octavia) Health Monitors and
    (OVN) Load Balancer Health Checks. This patch tries to make a
    more clear distinction between Octavia HM and OVN LB HCs.

    This patch also add a reference to the Octavia Health Monitors
    Ids to the external_ids of the OVN NB Load_Balancer entries.

    Related-Bug: #1997418

    Change-Id: Ib8499d7c4ea102e183ead31f063a3c0a70af6e23
    (cherry picked from commit 9cb8cd5054e64ab7fd13b4b63259b639b4f26c50)

tags: added: in-stable-xena
Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ovn-octavia-provider (master)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/873860
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/ba4ea1134b1da4ce1dcd186d4962d12270699fef
Submitter: "Zuul (22348)"
Branch: master

commit ba4ea1134b1da4ce1dcd186d4962d12270699fef
Author: Luis Tomas Bolivar <email address hidden>
Date: Fri Feb 17 08:18:13 2023 +0100

    Ensure HM also apply to FIPs associated to LB VIPs

    Currently, if a FIP gets associated to a LB with HealthMonitors
    it is not included as a new OVN Load Balancer Health Checks. This
    means that if the VIP is used, traffic will not be redirected to
    the dead members, buit if the FIP is used there is no health checks
    being applied and traffic will reach dead members.

    This patch adds the extra functionality so that an extra OVN
    Load Balancer Health Check is created for the FIPs associated to
    the Load Balancer.

    Closes-Bug: #1997418

    Change-Id: Idbf1fb15076518092ce5fdaa57500d29342f51be

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ovn-octavia-provider (stable/zed)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ovn-octavia-provider (stable/yoga)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ovn-octavia-provider (stable/xena)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ovn-octavia-provider (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875598

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ovn-octavia-provider (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875596
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/f95301ef5139d443cad17b1d211532cbdbd8a4dc
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit f95301ef5139d443cad17b1d211532cbdbd8a4dc
Author: Luis Tomas Bolivar <email address hidden>
Date: Fri Feb 17 08:18:13 2023 +0100

    Ensure HM also apply to FIPs associated to LB VIPs

    Currently, if a FIP gets associated to a LB with HealthMonitors
    it is not included as a new OVN Load Balancer Health Checks. This
    means that if the VIP is used, traffic will not be redirected to
    the dead members, buit if the FIP is used there is no health checks
    being applied and traffic will reach dead members.

    This patch adds the extra functionality so that an extra OVN
    Load Balancer Health Check is created for the FIPs associated to
    the Load Balancer.

    Closes-Bug: #1997418

    Change-Id: Idbf1fb15076518092ce5fdaa57500d29342f51be
    (cherry picked from commit ba4ea1134b1da4ce1dcd186d4962d12270699fef)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ovn-octavia-provider (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875598
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/8aff4293cd22af0e9c0adb7db6b0470ab2df493b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 8aff4293cd22af0e9c0adb7db6b0470ab2df493b
Author: Luis Tomas Bolivar <email address hidden>
Date: Fri Feb 17 08:18:13 2023 +0100

    Ensure HM also apply to FIPs associated to LB VIPs

    Currently, if a FIP gets associated to a LB with HealthMonitors
    it is not included as a new OVN Load Balancer Health Checks. This
    means that if the VIP is used, traffic will not be redirected to
    the dead members, buit if the FIP is used there is no health checks
    being applied and traffic will reach dead members.

    This patch adds the extra functionality so that an extra OVN
    Load Balancer Health Check is created for the FIPs associated to
    the Load Balancer.

    Closes-Bug: #1997418

    Change-Id: Idbf1fb15076518092ce5fdaa57500d29342f51be
    (cherry picked from commit ba4ea1134b1da4ce1dcd186d4962d12270699fef)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ovn-octavia-provider (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875597
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/461320ad319e2628b5e73ce86a5683fd24b933b6
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 461320ad319e2628b5e73ce86a5683fd24b933b6
Author: Luis Tomas Bolivar <email address hidden>
Date: Fri Feb 17 08:18:13 2023 +0100

    Ensure HM also apply to FIPs associated to LB VIPs

    Currently, if a FIP gets associated to a LB with HealthMonitors
    it is not included as a new OVN Load Balancer Health Checks. This
    means that if the VIP is used, traffic will not be redirected to
    the dead members, buit if the FIP is used there is no health checks
    being applied and traffic will reach dead members.

    This patch adds the extra functionality so that an extra OVN
    Load Balancer Health Check is created for the FIPs associated to
    the Load Balancer.

    Closes-Bug: #1997418

    Change-Id: Idbf1fb15076518092ce5fdaa57500d29342f51be
    (cherry picked from commit ba4ea1134b1da4ce1dcd186d4962d12270699fef)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ovn-octavia-provider 4.0.0.0rc1

This issue was fixed in the openstack/ovn-octavia-provider 4.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ovn-octavia-provider (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/875595
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/b22039aaf1a16d60d5b4617436dfc81ce080560c
Submitter: "Zuul (22348)"
Branch: stable/zed

commit b22039aaf1a16d60d5b4617436dfc81ce080560c
Author: Luis Tomas Bolivar <email address hidden>
Date: Fri Feb 17 08:18:13 2023 +0100

    Ensure HM also apply to FIPs associated to LB VIPs

    Currently, if a FIP gets associated to a LB with HealthMonitors
    it is not included as a new OVN Load Balancer Health Checks. This
    means that if the VIP is used, traffic will not be redirected to
    the dead members, buit if the FIP is used there is no health checks
    being applied and traffic will reach dead members.

    This patch adds the extra functionality so that an extra OVN
    Load Balancer Health Check is created for the FIPs associated to
    the Load Balancer.

    Closes-Bug: #1997418

    Change-Id: Idbf1fb15076518092ce5fdaa57500d29342f51be
    (cherry picked from commit ba4ea1134b1da4ce1dcd186d4962d12270699fef)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ovn-octavia-provider 1.3.0

This issue was fixed in the openstack/ovn-octavia-provider 1.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ovn-octavia-provider 2.1.0

This issue was fixed in the openstack/ovn-octavia-provider 2.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ovn-octavia-provider 3.1.0

This issue was fixed in the openstack/ovn-octavia-provider 3.1.0 release.

Revision history for this message
Kurt Garloff (kgarloff) wrote :

https://bugs.launchpad.net/neutron/+bug/1956035
looks like a duplicate to this one.

Revision history for this message
Mathias (mattfechner) wrote :

i think this bug is unfortunately not to 100 % percent solved, the healthmonitor, not closed the connection to the errored member

So in Detail:

The ERROR operating status of the members did not cause the LB to no longer distribute requests to the ERROR'ed members -- but in a installation released with ovn-octavia-provider we observe in a example, 50%
of the requests fail due to 2/4 ERROR'ed members.
Release of this environment is Zed.

we tracked this bug in our project here:

https://github.com/SovereignCloudStack/issues/issues/268

Revision history for this message
Fernando Royo (froyoredhat) wrote :

Hi Mathias,

I make some test after seeing your comment and I have reproduced the issue when the FIP is added after the HM already was created, if the FIP is added before the HM is created it works fine.

I also reproduce the issue on master and on all stables branches (untill wallaby).

We would create a patch to cover this case.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ovn-octavia-provider (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/886603

Revision history for this message
Sven Kieske (s-kieske) wrote :

I can't reset the state from "Fix Released". I guess this is the reason that the bot doesn't post updates regarding:

https://review.opendev.org/c/openstack/ovn-octavia-provider/+/886603

I wanted to ask if it's okay if I create backports for these?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ovn-octavia-provider wallaby-eom

This issue was fixed in the openstack/ovn-octavia-provider wallaby-eom release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.