neutron

db-sync script may fail when migrating from ovs to ovn

Bug #1939704 reported by Jakub Libosvar on 2021-08-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Medium	Jakub Libosvar

Bug Description

During the steps where port bindings are updated [1] there might be a neutron-server l3 gateway scheduling going on and the port binding for gateway router ports can change its gateway chassis. It makes the db sync script fail with following traceback

2021-08-12 06:21:40.738 48 INFO networking_ovn.cmd.neutron_ovn_db_sync_util [req-988ef422-ae85-41d7-a4a0-1e7b6c68a29e - - - - -] Migrating Neutron database from OVS to OVN
2021-08-12 06:21:41.300 48 CRITICAL neutron_ovn_db_sync_util [req-988ef422-ae85-41d7-a4a0-1e7b6c68a29e - - - - -] Unhandled error: neutron_lib.exceptions.ObjectNotFound: Object PortBinding(port_id=a95eaa4b-fe5f-4770-9a03-2e76f20c2870, host=controller-2.redhat.local) not found.
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module>
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util sys.exit(main())
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.6/site-packages/networking_ovn/cmd/neutron_ovn_db_sync_util.py", line 233, in main
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util db_migration.migrate_neutron_database_to_ovn(core_plugin)
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.6/site-packages/networking_ovn/ml2/db_migration.py", line 66, in migrate_neutron_database_to_ovn
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util pb.update()
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.6/site-packages/neutron/objects/base.py", line 337, in decorator
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util return func(self, *args, **kwargs)
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.6/site-packages/neutron/objects/base.py", line 906, in update
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util self._get_composite_keys()))
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 86, in update_object
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util db_obj = _safe_get_object(obj_cls, context, **kwargs)
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util File "/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 80, in _safe_get_object
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util id="%s(%s)" % (obj_cls.db_model.__name__, key))
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util neutron_lib.exceptions.ObjectNotFound: Object PortBinding(port_id=a95eaa4b-fe5f-4770-9a03-2e76f20c2870, host=controller-2.redhat.local) not found.
2021-08-12 06:21:41.300 48 ERROR neutron_ovn_db_sync_util

[1] https://opendev.org/openstack/neutron/src/commit/caac890c8e121e1bebe33f4e93ab79d4c294db35/neutron/plugins/ml2/drivers/ovn/db_migration.py#L73-L83

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-12: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/804405

Changed in neutron:
status:	New → In Progress

Miguel Lavalle (minsel) on 2021-08-12

Changed in neutron:
importance:	Undecided → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-16: Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/804405
Committed: https://opendev.org/openstack/neutron/commit/9e0c075bf169ed6f3d768cf23be6f47aeae0f98e
Submitter: "Zuul (22348)"
Branch: master

commit 9e0c075bf169ed6f3d768cf23be6f47aeae0f98e
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 12 16:48:59 2021 +0200

ovn: Don't fail db-sync if port binding changes

    During migration from OVS to OVN it can happen that gateway ports are
    scheduled to a different gateway chassis when Neutron is running. This
    patch doesn't fail in such case. The migration procedure runs the db
    sync twice in a row so it should be good to not perform any action when
    this happens and let the next migration handle that.

    Change-Id: I28a4a5fef20d5049f4887d43006947b434de3d78
    Closes-Bug: #1939704
    Signed-off-by: Jakub Libosvar <email address hidden>

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-16: Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/804723

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-08-18: Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/804723
Committed: https://opendev.org/openstack/neutron/commit/069faf570a5d75064520f3f42f53545be8998cd2
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 069faf570a5d75064520f3f42f53545be8998cd2
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 12 16:48:59 2021 +0200

ovn: Don't fail db-sync if port binding changes

    Change-Id: I28a4a5fef20d5049f4887d43006947b434de3d78
    Closes-Bug: #1939704
    Signed-off-by: Jakub Libosvar <email address hidden>
    (cherry picked from commit 9e0c075bf169ed6f3d768cf23be6f47aeae0f98e)

tags:

added: in-stable-wallaby

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-10: Fix included in openstack/neutron 18.1.1

This issue was fixed in the openstack/neutron 18.1.1 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-15: Fix included in openstack/neutron 19.0.0.0rc1

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

Revision history for this message

Rodolfo Alonso (rodolfo-alonso-hernandez) wrote on 2021-10-05:

Hello:

The script is still failing when executing the method "migrate_neutron_database_to_ovn". This error could be caused by concurrent operations modifying the DB registers (port bindings or trunks).

In this method, when the port bindings are retrieved, the objects returned [1] are outside a DB transaction context. When the "update" command is executed, the DB object stored in the OVO is not fresh and could have been changed. That could cause the error pasted here: https://paste.opendev.org/show/809786/.

In order to guarantee the transactionality, we need to enclose the whole method inside a writer context.

Regards.

[1]https://review.opendev.org/c/openstack/neutron/+/804723/1/neutron/plugins/ml2/drivers/ovn/db_migration.py#57

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-05: Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/812481

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-06: Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/812481
Committed: https://opendev.org/openstack/neutron/commit/dfcbb4cce0f206bbf824a1062de028223a27388f
Submitter: "Zuul (22348)"
Branch: master

commit dfcbb4cce0f206bbf824a1062de028223a27388f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Oct 5 11:12:20 2021 +0000

Execute "migrate_neutron_database_to_ovn" inside the same DB ctx

    All DB commands executed in "migrate_neutron_database_to_ovn"
    should be executed inside the same write context. That will prevent
    the DB transaction errors reported in the BZ. When a OVO object is
    retrieved and updated, both DB commands should be executed in the
    same DB transaction to ensure the DB object stored in the OVO
    (that is a stateful representation of the requested columns) is
    fresh and not modified by other transaction.

Related-Bug: #1939704
Change-Id: I754e2f9ad5b896fcc14ff30bd2901ccec2471907

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-15: Fix proposed to neutron (master)

#10

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/814181

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-10: Fix proposed to neutron (stable/xena)

#11

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/817437

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-10: Fix proposed to neutron (stable/wallaby)

#12

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/817438

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-13: Fix merged to neutron (master)

#13

Reviewed: https://review.opendev.org/c/openstack/neutron/+/814181
Committed: https://opendev.org/openstack/neutron/commit/662045f519a164ae683a37f5cf1c4b30369a3386
Submitter: "Zuul (22348)"
Branch: master

commit 662045f519a164ae683a37f5cf1c4b30369a3386
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Oct 15 12:41:52 2021 +0000

[OVN] Execute OVN migration transactions independently

    Instead of executing all Neutron DB changes in one single
    transaction, in order to avoid clashes with other DB operations
    each element will be treated independently.

For the "PortBinding" registers update, each element will be updated
in an independent transaction.

    For the "Trunk" registers update, all subports "PortBinding"
    registers belonging to this trunk will be updated in an independent
    transaction.

Closes-Bug: #1939704
Change-Id: I6f78750af34001b3707f1a6ee3a0882b427a5dcc

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-16: Related fix proposed to neutron (stable/xena)

#14

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/818057

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-16: Related fix proposed to neutron (stable/wallaby)

#15

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/818059

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-22: Related fix merged to neutron (stable/xena)

#16

Reviewed: https://review.opendev.org/c/openstack/neutron/+/818057
Committed: https://opendev.org/openstack/neutron/commit/785ce06b3d06b4477730641ad431c99072f35bac
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 785ce06b3d06b4477730641ad431c99072f35bac
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Oct 5 11:12:20 2021 +0000

Execute "migrate_neutron_database_to_ovn" inside the same DB ctx

    Related-Bug: #1939704
    Change-Id: I754e2f9ad5b896fcc14ff30bd2901ccec2471907
    (cherry picked from commit dfcbb4cce0f206bbf824a1062de028223a27388f)

tags:

added: in-stable-xena

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-22: Fix merged to neutron (stable/xena)

#17

Reviewed: https://review.opendev.org/c/openstack/neutron/+/817437
Committed: https://opendev.org/openstack/neutron/commit/084d210d1077eeee2ccf2ae0d3dcf32cd18db484
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 084d210d1077eeee2ccf2ae0d3dcf32cd18db484
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Oct 15 12:41:52 2021 +0000

[OVN] Execute OVN migration transactions independently

    Instead of executing all Neutron DB changes in one single
    transaction, in order to avoid clashes with other DB operations
    each element will be treated independently.

For the "PortBinding" registers update, each element will be updated
in an independent transaction.

    For the "Trunk" registers update, all subports "PortBinding"
    registers belonging to this trunk will be updated in an independent
    transaction.

    Closes-Bug: #1939704
    Change-Id: I6f78750af34001b3707f1a6ee3a0882b427a5dcc
    (cherry picked from commit 662045f519a164ae683a37f5cf1c4b30369a3386)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-22: Related fix merged to neutron (stable/wallaby)

#18

Reviewed: https://review.opendev.org/c/openstack/neutron/+/818059
Committed: https://opendev.org/openstack/neutron/commit/98e98f995c53a9e4426a96686d884ec2333ade8a
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 98e98f995c53a9e4426a96686d884ec2333ade8a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Oct 5 11:12:20 2021 +0000

Execute "migrate_neutron_database_to_ovn" inside the same DB ctx

    Related-Bug: #1939704
    Change-Id: I754e2f9ad5b896fcc14ff30bd2901ccec2471907
    (cherry picked from commit dfcbb4cce0f206bbf824a1062de028223a27388f)
    (cherry picked from commit 785ce06b3d06b4477730641ad431c99072f35bac)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-22: Fix merged to neutron (stable/wallaby)

#19

Reviewed: https://review.opendev.org/c/openstack/neutron/+/817438
Committed: https://opendev.org/openstack/neutron/commit/8c663c5a2a74a17b62d9d36a38fd3f19627f11e1
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 8c663c5a2a74a17b62d9d36a38fd3f19627f11e1
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Oct 15 12:41:52 2021 +0000

[OVN] Execute OVN migration transactions independently

    Instead of executing all Neutron DB changes in one single
    transaction, in order to avoid clashes with other DB operations
    each element will be treated independently.

For the "PortBinding" registers update, each element will be updated
in an independent transaction.

    For the "Trunk" registers update, all subports "PortBinding"
    registers belonging to this trunk will be updated in an independent
    transaction.

    Closes-Bug: #1939704
    Change-Id: I6f78750af34001b3707f1a6ee3a0882b427a5dcc
    (cherry picked from commit 662045f519a164ae683a37f5cf1c4b30369a3386)