failover of ACTIVE_STANDBY LBs can take a lot of time in amphorav1

[Gregory Thiemonge]

There is an issue in amphorav1 when 2 amps of an A/S LB are missing and a failover is triggered, the failover can take between 15 and 40 min, depending on the retry/interval config in the worker.

when recreating the 1st amphora, the failover flow also updates the 2nd amphora (it updates the VRRP config in all the amphorae):

https://opendev.org/openstack/octavia/src/commit/8b196bb3bb05c9d87dae688750457dbb944d4d1b/octavia/controller/worker/v1/flows/amphora_flows.py#L290-L308

but if the 2nd amphora is also failing, those tasks skip the update after "timeout_dict" (but they are not failing, and they set the status of the amp to ERROR)

however, AmphoraIndexVRRPStart has a few issues:
- it calls amphora_driver.start_vrrp_service, this function skips non-ALLOCATED amp [1], but the task doesn't reload the DB object, if a previous task sets it in ERROR, it is still ALLOCATED in the object passed by taskflow (note: in v2, the object is reloaded from the DB)
- it passes timeout_dict to amphora_driver.start_vrrp_service but start_vrrp_service doesn't pass it to _populate_amphora_api_version [2], in case the amp is missing, the short timeout in timeout_dict is not used (the function timeouts after ~15 min in devstack instead of 2 or 3 min)

[1] https://opendev.org/openstack/octavia/src/commit/8b196bb3bb05c9d87dae688750457dbb944d4d1b/octavia/amphorae/drivers/keepalived/vrrp_rest_driver.py#L84
[2] https://opendev.org/openstack/octavia/src/commit/8b196bb3bb05c9d87dae688750457dbb944d4d1b/octavia/amphorae/drivers/keepalived/vrrp_rest_driver.py#L91

We can fix both issues but there's a more interesting lead

the failover flow invokes 3 successive tasks:
- AmphoraIndexUpdateVRRPInterface
- AmphoraIndexVRRPUpdate
- AmphoraIndexVRRPStart

AmphoraIndexUpdateVRRPInterface provides an 'amp_vrrp_int' that is not null when it succeeds.
when the amphora is not reachable, amp_vrrp_int is None [3]
This value can be passed to AmphoraIndexVRRPUpdate and AmphoraIndexVRRPStart, and we can skip the execution of the functions if it is None, it would prevent the amphora driver from attempting to connect to a missing amphora.

[3] https://opendev.org/openstack/octavia/src/commit/8b196bb3bb05c9d87dae688750457dbb944d4d1b/octavia/controller/worker/v1/tasks/amphora_driver_tasks.py#L321

[Gregory Thiemonge (gthiemonge)]

I think we need coverage in octavia-tempest-plugin for this situation

[Gregory Thiemonge (gthiemonge)]

while it doesn't take as much time, amphora failover in amphorav2 can be improved using the same method

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/octavia/+/893536

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/octavia/+/893537

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/893537
Committed: https://opendev.org/openstack/octavia/commit/cce348040d43b92b59146adeb15a4f1486946a64
Submitter: "Zuul (22348)"
Branch: master

commit cce348040d43b92b59146adeb15a4f1486946a64
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 11:35:34 2023 -0400

    Fix timeout duration in start_vrrp_service during failovers

    The timeout_dict argument was not passed to the
    _populate_amphora_api_version, it could have made the failover of broken
    amphora really long.

    Related-Bug: #2033894

    Change-Id: I497f2d8587e25eda0d169aa919cb6fbede7e9fc6

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/893536
Committed: https://opendev.org/openstack/octavia/commit/ca70587f0bcb4b2a05bd5c243f111140a5762114
Submitter: "Zuul (22348)"
Branch: master

commit ca70587f0bcb4b2a05bd5c243f111140a5762114
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 09:08:37 2023 -0400

    Reduce duration of failovers with amphora in ERROR

    In the failover flow, there are multiple tasks for the configuration of
    VRRP for the other amphorae of the load balancer, but during outage the
    other amps may not be available. To prevent the tasks from attempting
    connections to unreachable amphorae, we can detect in the first task
    that an amp is unreachable and pass this information to the other tasks.

    Those connection attempts could have taken a lot of time, between 15 min
    and 40 min depending on the configuration of Octavia and the provider
    driver (amphorav1 or amphorav2)

    Closes-Bug: #2033894

    Change-Id: Ib33a0b8d2875e4ff97c65933fe9360bb06994d32

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/octavia/+/897783

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/octavia/+/897784

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/octavia/+/897786

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/octavia/+/897787

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/octavia/+/897789

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/octavia/+/897790

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/octavia/+/898101

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/octavia/+/898102

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/octavia/+/898104

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/octavia/+/898105

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/octavia/+/898112

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/octavia/+/898113

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/897783
Committed: https://opendev.org/openstack/octavia/commit/54eb1624f5762e9b848f10f712152faf02f8b84d
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 54eb1624f5762e9b848f10f712152faf02f8b84d
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 11:35:34 2023 -0400

    Fix timeout duration in start_vrrp_service during failovers

    The timeout_dict argument was not passed to the
    _populate_amphora_api_version, it could have made the failover of broken
    amphora really long.

    Related-Bug: #2033894

    Change-Id: I497f2d8587e25eda0d169aa919cb6fbede7e9fc6
    (cherry picked from commit cce348040d43b92b59146adeb15a4f1486946a64)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/897784
Committed: https://opendev.org/openstack/octavia/commit/6fdc2079da3392fdfbdf4a80df05bfe7d149ca95
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 6fdc2079da3392fdfbdf4a80df05bfe7d149ca95
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 09:08:37 2023 -0400

    Reduce duration of failovers with amphora in ERROR

    In the failover flow, there are multiple tasks for the configuration of
    VRRP for the other amphorae of the load balancer, but during outage the
    other amps may not be available. To prevent the tasks from attempting
    connections to unreachable amphorae, we can detect in the first task
    that an amp is unreachable and pass this information to the other tasks.

    Those connection attempts could have taken a lot of time, between 15 min
    and 40 min depending on the configuration of Octavia and the provider
    driver (amphorav1 or amphorav2)

    Closes-Bug: #2033894

    Change-Id: Ib33a0b8d2875e4ff97c65933fe9360bb06994d32
    (cherry picked from commit ca70587f0bcb4b2a05bd5c243f111140a5762114)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/897789
Committed: https://opendev.org/openstack/octavia/commit/5c98d901337cb0d6e4c575e3280cb77839720cfe
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 5c98d901337cb0d6e4c575e3280cb77839720cfe
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 11:35:34 2023 -0400

    Fix timeout duration in start_vrrp_service during failovers

    The timeout_dict argument was not passed to the
    _populate_amphora_api_version, it could have made the failover of broken
    amphora really long.

    Related-Bug: #2033894

    Change-Id: I497f2d8587e25eda0d169aa919cb6fbede7e9fc6
    (cherry picked from commit cce348040d43b92b59146adeb15a4f1486946a64)
    (cherry picked from commit 54eb1624f5762e9b848f10f712152faf02f8b84d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/897790
Committed: https://opendev.org/openstack/octavia/commit/97b1b8387e42f8361b612a0b81b1c60b6e14db37
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 97b1b8387e42f8361b612a0b81b1c60b6e14db37
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 09:08:37 2023 -0400

    Reduce duration of failovers with amphora in ERROR

    In the failover flow, there are multiple tasks for the configuration of
    VRRP for the other amphorae of the load balancer, but during outage the
    other amps may not be available. To prevent the tasks from attempting
    connections to unreachable amphorae, we can detect in the first task
    that an amp is unreachable and pass this information to the other tasks.

    Those connection attempts could have taken a lot of time, between 15 min
    and 40 min depending on the configuration of Octavia and the provider
    driver (amphorav1 or amphorav2)

    Closes-Bug: #2033894

    Note: stable/2023.1 and older, the patch also includes modifications in
          octavia/controller/worker/v1/

    Conflicts:
            octavia/controller/worker/v2/tasks/amphora_driver_tasks.py

    Change-Id: Ib33a0b8d2875e4ff97c65933fe9360bb06994d32
    (cherry picked from commit ca70587f0bcb4b2a05bd5c243f111140a5762114)
    (cherry picked from commit 6fdc2079da3392fdfbdf4a80df05bfe7d149ca95)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/897786
Committed: https://opendev.org/openstack/octavia/commit/1dadaff530e252e8e96ea8b4c6dcec902a797516
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 1dadaff530e252e8e96ea8b4c6dcec902a797516
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 11:35:34 2023 -0400

    Fix timeout duration in start_vrrp_service during failovers

    The timeout_dict argument was not passed to the
    _populate_amphora_api_version, it could have made the failover of broken
    amphora really long.

    Related-Bug: #2033894

    Change-Id: I497f2d8587e25eda0d169aa919cb6fbede7e9fc6
    (cherry picked from commit cce348040d43b92b59146adeb15a4f1486946a64)
    (cherry picked from commit 54eb1624f5762e9b848f10f712152faf02f8b84d)
    (cherry picked from commit 5c98d901337cb0d6e4c575e3280cb77839720cfe)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/897787
Committed: https://opendev.org/openstack/octavia/commit/dd849b4c5c1a1d33e4fd4a9afa2f35903d4d9aac
Submitter: "Zuul (22348)"
Branch: stable/zed

commit dd849b4c5c1a1d33e4fd4a9afa2f35903d4d9aac
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 09:08:37 2023 -0400

    Reduce duration of failovers with amphora in ERROR

    In the failover flow, there are multiple tasks for the configuration of
    VRRP for the other amphorae of the load balancer, but during outage the
    other amps may not be available. To prevent the tasks from attempting
    connections to unreachable amphorae, we can detect in the first task
    that an amp is unreachable and pass this information to the other tasks.

    Those connection attempts could have taken a lot of time, between 15 min
    and 40 min depending on the configuration of Octavia and the provider
    driver (amphorav1 or amphorav2)

    Closes-Bug: #2033894

    Note: stable/2023.1 and older, the patch also includes modifications in
          octavia/controller/worker/v1/

    Conflicts:
            octavia/controller/worker/v2/tasks/amphora_driver_tasks.py

    Change-Id: Ib33a0b8d2875e4ff97c65933fe9360bb06994d32
    (cherry picked from commit ca70587f0bcb4b2a05bd5c243f111140a5762114)
    (cherry picked from commit 6fdc2079da3392fdfbdf4a80df05bfe7d149ca95)
    (cherry picked from commit 97b1b8387e42f8361b612a0b81b1c60b6e14db37)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/898101
Committed: https://opendev.org/openstack/octavia/commit/9e5a6340156113ec3a014e64b3bad6617cd1c082
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 9e5a6340156113ec3a014e64b3bad6617cd1c082
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 11:35:34 2023 -0400

    Fix timeout duration in start_vrrp_service during failovers

    The timeout_dict argument was not passed to the
    _populate_amphora_api_version, it could have made the failover of broken
    amphora really long.

    Related-Bug: #2033894

    Change-Id: I497f2d8587e25eda0d169aa919cb6fbede7e9fc6
    (cherry picked from commit cce348040d43b92b59146adeb15a4f1486946a64)
    (cherry picked from commit 54eb1624f5762e9b848f10f712152faf02f8b84d)
    (cherry picked from commit 5c98d901337cb0d6e4c575e3280cb77839720cfe)
    (cherry picked from commit 1dadaff530e252e8e96ea8b4c6dcec902a797516)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/898102
Committed: https://opendev.org/openstack/octavia/commit/68cb6e38800d03746236d0ed3e910658afe54323
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 68cb6e38800d03746236d0ed3e910658afe54323
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 09:08:37 2023 -0400

    Reduce duration of failovers with amphora in ERROR

    In the failover flow, there are multiple tasks for the configuration of
    VRRP for the other amphorae of the load balancer, but during outage the
    other amps may not be available. To prevent the tasks from attempting
    connections to unreachable amphorae, we can detect in the first task
    that an amp is unreachable and pass this information to the other tasks.

    Those connection attempts could have taken a lot of time, between 15 min
    and 40 min depending on the configuration of Octavia and the provider
    driver (amphorav1 or amphorav2)

    Closes-Bug: #2033894

    Note: stable/2023.1 and older, the patch also includes modifications in
          octavia/controller/worker/v1/

    Conflicts:
            octavia/controller/worker/v2/tasks/amphora_driver_tasks.py

    Change-Id: Ib33a0b8d2875e4ff97c65933fe9360bb06994d32
    (cherry picked from commit ca70587f0bcb4b2a05bd5c243f111140a5762114)
    (cherry picked from commit 6fdc2079da3392fdfbdf4a80df05bfe7d149ca95)
    (cherry picked from commit 97b1b8387e42f8361b612a0b81b1c60b6e14db37)
    (cherry picked from commit dd849b4c5c1a1d33e4fd4a9afa2f35903d4d9aac)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/898104
Committed: https://opendev.org/openstack/octavia/commit/b44b71fee325db5cfeba6def1dd6d1ff8f1b198b
Submitter: "Zuul (22348)"
Branch: stable/xena

commit b44b71fee325db5cfeba6def1dd6d1ff8f1b198b
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 11:35:34 2023 -0400

    Fix timeout duration in start_vrrp_service during failovers

    The timeout_dict argument was not passed to the
    _populate_amphora_api_version, it could have made the failover of broken
    amphora really long.

    Related-Bug: #2033894

    Change-Id: I497f2d8587e25eda0d169aa919cb6fbede7e9fc6
    (cherry picked from commit cce348040d43b92b59146adeb15a4f1486946a64)
    (cherry picked from commit 54eb1624f5762e9b848f10f712152faf02f8b84d)
    (cherry picked from commit 5c98d901337cb0d6e4c575e3280cb77839720cfe)
    (cherry picked from commit 1dadaff530e252e8e96ea8b4c6dcec902a797516)
    (cherry picked from commit 9e5a6340156113ec3a014e64b3bad6617cd1c082)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/898105
Committed: https://opendev.org/openstack/octavia/commit/4ca9b894ec1b0234eebb57da828c9bf37d26e2a4
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 4ca9b894ec1b0234eebb57da828c9bf37d26e2a4
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 09:08:37 2023 -0400

    Reduce duration of failovers with amphora in ERROR

    In the failover flow, there are multiple tasks for the configuration of
    VRRP for the other amphorae of the load balancer, but during outage the
    other amps may not be available. To prevent the tasks from attempting
    connections to unreachable amphorae, we can detect in the first task
    that an amp is unreachable and pass this information to the other tasks.

    Those connection attempts could have taken a lot of time, between 15 min
    and 40 min depending on the configuration of Octavia and the provider
    driver (amphorav1 or amphorav2)

    Closes-Bug: #2033894

    Note: stable/2023.1 and older, the patch also includes modifications in
          octavia/controller/worker/v1/

    Conflicts:
            octavia/controller/worker/v2/tasks/amphora_driver_tasks.py

    Change-Id: Ib33a0b8d2875e4ff97c65933fe9360bb06994d32
    (cherry picked from commit ca70587f0bcb4b2a05bd5c243f111140a5762114)
    (cherry picked from commit 6fdc2079da3392fdfbdf4a80df05bfe7d149ca95)
    (cherry picked from commit 97b1b8387e42f8361b612a0b81b1c60b6e14db37)
    (cherry picked from commit dd849b4c5c1a1d33e4fd4a9afa2f35903d4d9aac)
    (cherry picked from commit 68cb6e38800d03746236d0ed3e910658afe54323)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/898112
Committed: https://opendev.org/openstack/octavia/commit/353b9ed77ff696c86b1d44f41aefea9d7da4d6c7
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 353b9ed77ff696c86b1d44f41aefea9d7da4d6c7
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 11:35:34 2023 -0400

    Fix timeout duration in start_vrrp_service during failovers

    The timeout_dict argument was not passed to the
    _populate_amphora_api_version, it could have made the failover of broken
    amphora really long.

    Related-Bug: #2033894

    Change-Id: I497f2d8587e25eda0d169aa919cb6fbede7e9fc6
    (cherry picked from commit cce348040d43b92b59146adeb15a4f1486946a64)
    (cherry picked from commit 54eb1624f5762e9b848f10f712152faf02f8b84d)
    (cherry picked from commit 5c98d901337cb0d6e4c575e3280cb77839720cfe)
    (cherry picked from commit 1dadaff530e252e8e96ea8b4c6dcec902a797516)
    (cherry picked from commit 9e5a6340156113ec3a014e64b3bad6617cd1c082)
    (cherry picked from commit b44b71fee325db5cfeba6def1dd6d1ff8f1b198b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/octavia/+/898113
Committed: https://opendev.org/openstack/octavia/commit/8e2088336458a809f72acb7e36eb60d7955ee93a
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 8e2088336458a809f72acb7e36eb60d7955ee93a
Author: Gregory Thiemonge <email address hidden>
Date: Fri Sep 1 09:08:37 2023 -0400

    Reduce duration of failovers with amphora in ERROR

    In the failover flow, there are multiple tasks for the configuration of
    VRRP for the other amphorae of the load balancer, but during outage the
    other amps may not be available. To prevent the tasks from attempting
    connections to unreachable amphorae, we can detect in the first task
    that an amp is unreachable and pass this information to the other tasks.

    Those connection attempts could have taken a lot of time, between 15 min
    and 40 min depending on the configuration of Octavia and the provider
    driver (amphorav1 or amphorav2)

    Closes-Bug: #2033894

    Note: stable/2023.1 and older, the patch also includes modifications in
          octavia/controller/worker/v1/

    Conflicts:
            octavia/controller/worker/v2/tasks/amphora_driver_tasks.py

    Change-Id: Ib33a0b8d2875e4ff97c65933fe9360bb06994d32
    (cherry picked from commit ca70587f0bcb4b2a05bd5c243f111140a5762114)
    (cherry picked from commit 6fdc2079da3392fdfbdf4a80df05bfe7d149ca95)
    (cherry picked from commit 97b1b8387e42f8361b612a0b81b1c60b6e14db37)
    (cherry picked from commit dd849b4c5c1a1d33e4fd4a9afa2f35903d4d9aac)
    (cherry picked from commit 68cb6e38800d03746236d0ed3e910658afe54323)
    (cherry picked from commit 4ca9b894ec1b0234eebb57da828c9bf37d26e2a4)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/octavia 10.1.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/octavia 11.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/octavia wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/octavia xena-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/octavia 14.0.0.0rc1 release candidate.