Migration to port groups may not work on multiple controllers deployment

Bug #1789921 reported by Daniel Alvarez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-ovn
Fix Released
Undecided
Daniel Alvarez

Bug Description

The migration task to port groups has been included as part of the db sync task. This task won't run on instances that doesn't hold an active OVSDB lock so the only way to make it work would be:

1. Stopping neutron-server on all controllers
2. Starting neutron-server on one single controller
3. Starting neutron-server on the rest of controllers

If we don't perform 1 and just restart neutron-server as part of an update/upgrade process it may happen that the db sync task never holds a lock and it won't ever run:

When all neutron-servers are running:

1. Restart neutron-server with the new (port groups) version.
   The lock will be on another neutron-server instance so the migration task won't run.
2. Repeat 1 for as many controllers present in the cloud

This bug applies for the db sync task in 'repair' mode.

Changed in networking-ovn:
assignee: nobody → Daniel Alvarez (dalvarezs)
Changed in networking-ovn:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-ovn (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/611672

tags: added: networking-ovn-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-ovn (stable/rocky)

Reviewed: https://review.openstack.org/611672
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=03dc5ae1c3208ddf68d0afe5f1b29a04d65a2135
Submitter: Zuul
Branch: stable/rocky

commit 03dc5ae1c3208ddf68d0afe5f1b29a04d65a2135
Author: Daniel Alvarez <email address hidden>
Date: Thu Aug 30 19:08:33 2018 +0200

    Attempt the migration to Port Groups from Maintenance worker

    The migration to port groups task is now moved to the maintenance
    worker. This worker holds a distributed lock within OVSDB so we'll
    make sure that the migration will be performed just once.

    During an update/upgrade, it may happen that newer version of
    neutron-server can't migrate to Port Groups as the lock is held
    by other (old instance). When all servers have been updated, the
    maintenance task will perform the migration just once on the cloud
    making sure that normal operation will happen.

    We can remove this task in later cycles as this is just a migration
    path from Address Sets to Port Groups for implementing Neutron
    Security Groups.

    TODO: functional test to validate the migration path to PGs.

    Change-Id: I227ec967f450b26b12f19d687e94029e6ef1e558
    Closes-Bug: #1789921
    Signed-off-by: Daniel Alvarez <email address hidden>
    (cherry picked from commit 17b15f2901e5ac2e51948670cab0772279b5e2df)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn 6.0.0.0b1

This issue was fixed in the openstack/networking-ovn 6.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-ovn (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/681565

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-ovn (stable/queens)

Reviewed: https://review.opendev.org/681565
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=1016280a92f89e5d6fa981529494544b0c11bc91
Submitter: Zuul
Branch: stable/queens

commit 1016280a92f89e5d6fa981529494544b0c11bc91
Author: Daniel Alvarez <email address hidden>
Date: Thu Aug 30 19:08:33 2018 +0200

    Attempt the migration to Port Groups from Maintenance worker

    The migration to port groups task is now moved to the maintenance
    worker. This worker holds a distributed lock within OVSDB so we'll
    make sure that the migration will be performed just once.

    During an update/upgrade, it may happen that newer version of
    neutron-server can't migrate to Port Groups as the lock is held
    by other (old instance). When all servers have been updated, the
    maintenance task will perform the migration just once on the cloud
    making sure that normal operation will happen.

    We can remove this task in later cycles as this is just a migration
    path from Address Sets to Port Groups for implementing Neutron
    Security Groups.

    TODO: functional test to validate the migration path to PGs.

    Change-Id: I227ec967f450b26b12f19d687e94029e6ef1e558
    Closes-Bug: #1789921
    Signed-off-by: Daniel Alvarez <email address hidden>
    (cherry picked from commit 17b15f2901e5ac2e51948670cab0772279b5e2df)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn 4.0.4

This issue was fixed in the openstack/networking-ovn 4.0.4 release.

Changed in networking-ovn:
status: In Progress → Fix Released
tags: removed: networking-ovn-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn 5.1.0

This issue was fixed in the openstack/networking-ovn 5.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.