neutron

neutron-ovn-db-sync-util can fail with KeyError

Bug #2045811 reported by Brian Haley
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Brian Haley

Bug Description

If the netron-ovn-db-sync-util is run while neutron-server is active (which is not recommended), it can randomly fail if there are active API calls in flight to create networks and/or subnets.

This is an example traceback I've seen many times in a production environment:

WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-7fc12422-6fae-4ec9-98bc-8a114f30c9e3 - - - - -] DHCP options for subnet 0662e4fd-f8b4-4d29-8ba7-5846bd19e45d is present in Neutron but out of sync for OVN
CRITICAL neutron_ovn_db_sync_util [req-7fc12422-6fae-4ec9-98bc-8a114f30c9e3 - - - - -] Unhandled error: KeyError: 'neutron-93ad1c21-d2cf-448a-8fae-21c71f44dc5c'
ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
ERROR neutron_ovn_db_sync_util File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module>
ERROR neutron_ovn_db_sync_util sys.exit(main())
ERROR neutron_ovn_db_sync_util File "/usr/lib/python3/dist-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 219, in main
ERROR neutron_ovn_db_sync_util synchronizer.do_sync()
ERROR neutron_ovn_db_sync_util File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 98, in do_sync
ERROR neutron_ovn_db_sync_util self.sync_networks_ports_and_dhcp_opts(ctx)
ERROR neutron_ovn_db_sync_util File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 871, in sync_networks_ports_and_dhcp_opts
ERROR neutron_ovn_db_sync_util self._sync_subnet_dhcp_options(
ERROR neutron_ovn_db_sync_util File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py", line 645, in _sync_subnet_dhcp_options
ERROR neutron_ovn_db_sync_util network = db_networks[utils.ovn_name(subnet['network_id'])]
ERROR neutron_ovn_db_sync_util KeyError: 'neutron-93ad1c21-d2cf-448a-8fae-21c71f44dc5c'

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/902841

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/902841
Committed: https://opendev.org/openstack/neutron/commit/e4323e1f209ea1c63fe7af5275ea2b96f52b8740
Submitter: "Zuul (22348)"
Branch: master

commit e4323e1f209ea1c63fe7af5275ea2b96f52b8740
Author: Brian Haley <email address hidden>
Date: Wed Dec 6 16:37:24 2023 -0500

    Fix KeyError failure in _sync_subnet_dhcp_options()

    If the netron-ovn-db-sync-util is run while neutron-server
    is active (which is not recommended), it can randomly fail
    if there are active API calls in flight to create networks
    and/or subnets.

    Skip the subnet and log a warning if detected.

    Closes-bug: #2045811
    Change-Id: Ic5d9608277dd5c4881b3e4b494e1864be0bed1b4

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/2024.1)

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/neutron/+/915296

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/2024.1)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/915296
Committed: https://opendev.org/openstack/neutron/commit/51961e195fa9ac474af45b366831cc521cbc6e34
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 51961e195fa9ac474af45b366831cc521cbc6e34
Author: Brian Haley <email address hidden>
Date: Wed Dec 6 16:37:24 2023 -0500

    Fix KeyError failure in _sync_subnet_dhcp_options()

    If the netron-ovn-db-sync-util is run while neutron-server
    is active (which is not recommended), it can randomly fail
    if there are active API calls in flight to create networks
    and/or subnets.

    Skip the subnet and log a warning if detected.

    Closes-bug: #2045811
    Change-Id: Ic5d9608277dd5c4881b3e4b494e1864be0bed1b4
    (cherry picked from commit e4323e1f209ea1c63fe7af5275ea2b96f52b8740)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/915345

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/915346

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/915345
Committed: https://opendev.org/openstack/neutron/commit/e34c95c4b91dee65acff6f82203fd5e7a87628ce
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit e34c95c4b91dee65acff6f82203fd5e7a87628ce
Author: Brian Haley <email address hidden>
Date: Wed Dec 6 16:37:24 2023 -0500

    Fix KeyError failure in _sync_subnet_dhcp_options()

    If the netron-ovn-db-sync-util is run while neutron-server
    is active (which is not recommended), it can randomly fail
    if there are active API calls in flight to create networks
    and/or subnets.

    Skip the subnet and log a warning if detected.

    Conflicts:
      neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py

    Closes-bug: #2045811
    Change-Id: Ic5d9608277dd5c4881b3e4b494e1864be0bed1b4
    (cherry picked from commit e4323e1f209ea1c63fe7af5275ea2b96f52b8740)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/915346
Committed: https://opendev.org/openstack/neutron/commit/2ec0ef090979b0bbdfd642ab9f1b85f9deff6af9
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 2ec0ef090979b0bbdfd642ab9f1b85f9deff6af9
Author: Brian Haley <email address hidden>
Date: Wed Dec 6 16:37:24 2023 -0500

    Fix KeyError failure in _sync_subnet_dhcp_options()

    If the netron-ovn-db-sync-util is run while neutron-server
    is active (which is not recommended), it can randomly fail
    if there are active API calls in flight to create networks
    and/or subnets.

    Skip the subnet and log a warning if detected.

    Conflicts:
      neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py

    Closes-bug: #2045811
    Change-Id: Ic5d9608277dd5c4881b3e4b494e1864be0bed1b4
    (cherry picked from commit e4323e1f209ea1c63fe7af5275ea2b96f52b8740)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/915956

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/915956
Committed: https://opendev.org/openstack/neutron/commit/4fc3c71546db1572f1f4e82b25e7ea119987e1d6
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 4fc3c71546db1572f1f4e82b25e7ea119987e1d6
Author: Brian Haley <email address hidden>
Date: Wed Dec 6 16:37:24 2023 -0500

    Fix KeyError failure in _sync_subnet_dhcp_options()

    If the netron-ovn-db-sync-util is run while neutron-server
    is active (which is not recommended), it can randomly fail
    if there are active API calls in flight to create networks
    and/or subnets.

    Skip the subnet and log a warning if detected.

    Conflicts:
      neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py

    Closes-bug: #2045811
    Change-Id: Ic5d9608277dd5c4881b3e4b494e1864be0bed1b4
    (cherry picked from commit e4323e1f209ea1c63fe7af5275ea2b96f52b8740)
    (cherry picked from commit 2ec0ef090979b0bbdfd642ab9f1b85f9deff6af9)

tags: added: in-stable-zed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (unmaintained/yoga)

Fix proposed to branch: unmaintained/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/916446

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (unmaintained/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/916446
Committed: https://opendev.org/openstack/neutron/commit/608914c2ce997b1a2d512c2220e2d142c3ca6016
Submitter: "Zuul (22348)"
Branch: unmaintained/yoga

commit 608914c2ce997b1a2d512c2220e2d142c3ca6016
Author: Brian Haley <email address hidden>
Date: Wed Dec 6 16:37:24 2023 -0500

    Fix KeyError failure in _sync_subnet_dhcp_options()

    If the netron-ovn-db-sync-util is run while neutron-server
    is active (which is not recommended), it can randomly fail
    if there are active API calls in flight to create networks
    and/or subnets.

    Skip the subnet and log a warning if detected.

    Conflicts:
      neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py

    Closes-bug: #2045811
    Change-Id: Ic5d9608277dd5c4881b3e4b494e1864be0bed1b4
    (cherry picked from commit e4323e1f209ea1c63fe7af5275ea2b96f52b8740)
    (cherry picked from commit 2ec0ef090979b0bbdfd642ab9f1b85f9deff6af9)
    (cherry picked from commit 4fc3c71546db1572f1f4e82b25e7ea119987e1d6)

tags: added: in-unmaintained-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 21.2.1

This issue was fixed in the openstack/neutron 21.2.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.