Connectivity during update can be lost with some version of openvswitch.

Bug #1888651 reported by Sofer Athlan-Guyot
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Sofer Athlan-Guyot

Bug Description

Hi,

originally reported there: https://bugzilla.redhat.com/show_bug.cgi?id=1858745#c5

Basically openvswitch update can be very disruptive, and end up with the service stopped. We then need to manually restart it after update.

Problem is that it takes ~2min to reach the update tasks that restart the service after the yum upgrade[1].

We need to implement the hack used for upgrade, to make the update less disruptive.

[1] here, https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/tripleo-packages/tripleo-packages-baremetal-puppet.yaml#L591-L596

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/742632

Changed in tripleo:
status: Triaged → In Progress
Changed in tripleo:
milestone: victoria-1 → victoria-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/743947

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/743949

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/742632
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8ca2b598cbee79177f36448b421692b4cd9c8c8d
Submitter: Zuul
Branch: master

commit 8ca2b598cbee79177f36448b421692b4cd9c8c8d
Author: Sofer Athlan-Guyot <email address hidden>
Date: Thu Jul 23 12:17:31 2020 +0200

    Add openvswitch special treatment to update too.

    Openvswitch package update can be disruptive during update too. So we
    factor out the code used for upgrade to re-use for update and
    undercloud update.

    We also fix a issue where yum/yumdownloader is not available anymore.
    This cause the special treatment to not happen anymore what not matter
    what.

    Instead of duplicating the ansible sequence, this has been ported to a
    ansible module that replicate the action defined in the templates.

    We also make sure to not try to restart the openvswitch service if
    nothing has been changed during the ovs upgrade task. This cover two
    cases. First, it's an optimization, if nothing has been done, we
    don't need to check the service. Second, and most importantly, if ovs
    isn't installed (this can happen on some minimal installation) then we
    don't try to restart the non-existent service.

    We keep the tasks at step 3 that make sure again that the openvswitch
    service is started. This is a failsafe and as such we ignore any
    error there. This also support the overcloud-minimal type of
    installation.

    Depends-On: https://review.opendev.org/742968
    Change-Id: I4fde50e353ab7b2e6b03d963f585f6abe35ca32d
    Closes-Bug: #1888651
    Related-Bug: #1888821

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/ussuri)

Reviewed: https://review.opendev.org/743947
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c2b828fbb8b1e9f085ecbd6d5abd085733f3b7cb
Submitter: Zuul
Branch: stable/ussuri

commit c2b828fbb8b1e9f085ecbd6d5abd085733f3b7cb
Author: Sofer Athlan-Guyot <email address hidden>
Date: Thu Jul 23 12:17:31 2020 +0200

    Add openvswitch special treatment to update too.

    Openvswitch package update can be disruptive during update too. So we
    factor out the code used for upgrade to re-use for update and
    undercloud update.

    We also fix a issue where yum/yumdownloader is not available anymore.
    This cause the special treatment to not happen anymore what not matter
    what.

    Instead of duplicating the ansible sequence, this has been ported to a
    ansible module that replicate the action defined in the templates.

    We also make sure to not try to restart the openvswitch service if
    nothing has been changed during the ovs upgrade task. This cover two
    cases. First, it's an optimization, if nothing has been done, we
    don't need to check the service. Second, and most importantly, if ovs
    isn't installed (this can happen on some minimal installation) then we
    don't try to restart the non-existent service.

    We keep the tasks at step 3 that make sure again that the openvswitch
    service is started. This is a failsafe and as such we ignore any
    error there. This also support the overcloud-minimal type of
    installation.

    Depends-On: https://review.opendev.org/743837
    Change-Id: I4fde50e353ab7b2e6b03d963f585f6abe35ca32d
    Closes-Bug: #1888651
    Related-Bug: #1888821
    (cherry picked from commit 8ca2b598cbee79177f36448b421692b4cd9c8c8d)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/743949
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ed65866d438b16698bc2b8aff8bbf3b0610ddd71
Submitter: Zuul
Branch: stable/train

commit ed65866d438b16698bc2b8aff8bbf3b0610ddd71
Author: Sofer Athlan-Guyot <email address hidden>
Date: Thu Jul 23 12:17:31 2020 +0200

    Add openvswitch special treatment to update too.

    Openvswitch package update can be disruptive during update too. So we
    factor out the code used for upgrade to re-use for update and
    undercloud update.

    We also fix a issue where yum/yumdownloader is not available anymore.
    This cause the special treatment to not happen anymore what not matter
    what.

    Instead of duplicating the ansible sequence, this has been ported to a
    ansible module that replicate the action defined in the templates.

    We also make sure to not try to restart the openvswitch service if
    nothing has been changed during the ovs upgrade task. This cover two
    cases. First, it's an optimization, if nothing has been done, we
    don't need to check the service. Second, and most importantly, if ovs
    isn't installed (this can happen on some minimal installation) then we
    don't try to restart the non-existent service.

    We keep the tasks at step 3 that make sure again that the openvswitch
    service is started. This is a failsafe and as such we ignore any
    error there. This also support the overcloud-minimal type of
    installation.

    Depends-On: https://review.opendev.org/743838
    Change-Id: I4fde50e353ab7b2e6b03d963f585f6abe35ca32d
    Closes-Bug: #1888651
    Related-Bug: #1888821
    (cherry picked from commit 8ca2b598cbee79177f36448b421692b4cd9c8c8d)
    (cherry picked from commit c2b828fbb8b1e9f085ecbd6d5abd085733f3b7cb)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.4.0

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.