[HA] keepalived_state_change does not finish "handle_initial_state"execution

Bug #1917793 reported by Rodolfo Alonso
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Rodolfo Alonso

Bug Description

As seen in some logs [1], when the process "keepalived_state_change" is spawned, the first task done is to read the HA interface status (backup, primary). Sometimes the process never finishes this initial task.

[1]https://72f7db0ba35c6ad18335-0a8a55712d031506235c83f14141b923.ssl.cf2.rackcdn.com/776701/9/check/neutron-functional-with-uwsgi/50d999b/testr_results.html

Changed in neutron:
status: New → Confirmed
importance: Undecided → Critical
tags: added: functional-tests gate-failure
description: updated
Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

https://bugs.launchpad.net/neutron/+bug/1921866 was marked as a duplicated of this LP bug.

tags: added: neutron-proactive-backport-potential
Changed in neutron:
status: Confirmed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/796865

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/victoria)

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/796866

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/796998

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/796999

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/796865
Committed: https://opendev.org/openstack/neutron/commit/8576690c19015f869dc2e454d16752b69490b99e
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 8576690c19015f869dc2e454d16752b69490b99e
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Mar 5 17:30:14 2021 +0000

    Read keepalived initial state in parallel to interface monitoring

    The initial router state method is now a thread that is executed
    in parallel with the "ip monitor" thread. If by any circumstance
    this thread does not read the interface IP addresses on time, the
    state of the router will be defined as "backup".

    Related-Bug: #1917793

    Change-Id: If76c6ee1734f544abdd4196431351d4328ad26fd
    (cherry picked from commit c1ade52fda59163510dd0959ab72c442d1d07027)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/796866
Committed: https://opendev.org/openstack/neutron/commit/a3ba317b8c58776cc3fdde80272871cf6aa6714d
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit a3ba317b8c58776cc3fdde80272871cf6aa6714d
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Mar 5 17:30:14 2021 +0000

    Read keepalived initial state in parallel to interface monitoring

    The initial router state method is now a thread that is executed
    in parallel with the "ip monitor" thread. If by any circumstance
    this thread does not read the interface IP addresses on time, the
    state of the router will be defined as "backup".

    Related-Bug: #1917793

    Change-Id: If76c6ee1734f544abdd4196431351d4328ad26fd
    (cherry picked from commit c1ade52fda59163510dd0959ab72c442d1d07027)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/796998
Committed: https://opendev.org/openstack/neutron/commit/a23accea9b912d844ff937123d2ece022698edf6
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit a23accea9b912d844ff937123d2ece022698edf6
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Mar 5 17:30:14 2021 +0000

    Read keepalived initial state in parallel to interface monitoring

    The initial router state method is now a thread that is executed
    in parallel with the "ip monitor" thread. If by any circumstance
    this thread does not read the interface IP addresses on time, the
    state of the router will be defined as "backup".

    Related-Bug: #1917793

    Conflicts:
        neutron/agent/l3/keepalived_state_change.py

    Change-Id: If76c6ee1734f544abdd4196431351d4328ad26fd
    (cherry picked from commit c1ade52fda59163510dd0959ab72c442d1d07027)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/train)

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/796999
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.