external_gateways KeyError when running add_gw_port_info_to_logical_router_port

[Will Szumski]

Version: 2023.1
Driver: OVN

I'm seeing the following error:

```
Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_gw_port_info_to_logical_router_port' (it runs every 600.00 seconds): KeyError: 'external_gateways'
2024-03-14 14:37:08.974 1026 ERROR futurist.periodics Traceback (most recent call last):
2024-03-14 14:37:08.974 1026 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.10/site-packages/futurist/periodics.py", line 290, in run
2024-03-14 14:37:08.974 1026 ERROR futurist.periodics work()
2024-03-14 14:37:08.974 1026 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.10/site-packages/futurist/periodics.py", line 64, in __call__
2024-03-14 14:37:08.974 1026 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs)
2024-03-14 14:37:08.974 1026 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.10/site-packages/futurist/periodics.py", line 178, in decorator
2024-03-14 14:37:08.974 1026 ERROR futurist.periodics return f(*args, **kwargs)
2024-03-14 14:37:08.974 1026 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 982, in add_gw_port_info_to_logical_router_port
2024-03-14 14:37:08.974 1026 ERROR futurist.periodics ext_gw['network_id'] for ext_gw in router['external_gateways']]
2024-03-14 14:37:08.974 1026 ERROR futurist.periodics KeyError: 'external_gateways'
2024-03-14 14:37:08.974 1026 ERROR futurist.periodics

```

This is triggered every ten minutes. Does the external_gateways field only exist if you have the external-gateway-multihoming enabled? Is this likely to cause a functional issue? The patch that added the changed suggested it was only informational:

https://review.opendev.org/c/openstack/neutron/+/908822

[Brian Haley (brian-haley)]

Looks like you are correct about the issue, the multiple gateway support was added in 2032.2 in [0] so the backport you mention should have been modified when it was cherry-picked to 2023.1, it needs to get the external gateway from the router object something like:

  router.get(l3.EXTERNAL_GW_INFO)

but that code in the maintenance task will need more tweaking than that, and we need to make sure there is a test since it should have failed our gate tests.

[0] https://review.opendev.org/c/openstack/neutron/+/873593

[Brian Haley (brian-haley)]

And it's actually this change that broke things:

https://review.opendev.org/c/openstack/neutron/+/908823/

[Brian Haley (brian-haley)]

And the reason the test passed is it mocked the router object returned:

        routers_db = [{
            'id': uuidutils.generate_uuid(),
            'external_gateways': [{'network_id': ext_net_id}]}]

So it essentially had the valid key.

This is one of the drawbacks of unit tests.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/913408

[Brian Haley (brian-haley)]

Can you test my proposed patch? I don't have a 2023.1 environment running at the moment. Thanks.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/913408
Committed: https://opendev.org/openstack/neutron/commit/9bb37523a50c3d5b8f062c15ec21719904c094ab
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 9bb37523a50c3d5b8f062c15ec21719904c094ab
Author: Brian Haley <email address hidden>
Date: Fri Mar 15 10:53:41 2024 -0400

    [stable/2023.1 only] Fix incorrect cherry-pick for ext gw

    A cherry pick [0] to fix log warnings accidentally had
    code that only supported multiple external gateways,
    which wasn't merged until 2023.2. Change to conform to
    2023.1 code where there is only one external_gateway_info
    object in a router.

    [0] https://review.opendev.org/c/openstack/neutron/+/908823/

    Closes-bug: #2057942
    Change-Id: I340bc7a83a65d20ec66edcb505784dfad2372d9f

[Sam Schmitt (samcat116)]

I am seeing this on a new 2023.2 deployment as well

[Brian Haley (brian-haley)]

Sam - can you confirm what version you are running and provide a traceback?

From what I can see the change that introduced multiple external gateways [0] is in 2023.2 and later.

$ git describe a221764751de05e42069f1c097b1025bd9c4fc52
23.0.0.0b2-53-ga221764751

[0] https://review.opendev.org/c/openstack/neutron/+/873593

[Michal Nasiadka (mnasiadka)]

I'm seeing the same on 2023.2:

2024-04-23 18:50:01.010 30 ERROR futurist.periodics [None req-e70cb2bd-b2bf-409e-baa0-bea4c7a85bfb - - - - - -] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_gw_port_info_to_logical_router_port' (it runs every 600.00 seconds): KeyError: 'external_gateways'
2024-04-23 18:50:01.010 30 ERROR futurist.periodics Traceback (most recent call last):
2024-04-23 18:50:01.010 30 ERROR futurist.periodics File "/var/lib/kolla/venv/lib64/python3.9/site-packages/futurist/periodics.py", line 290, in run
2024-04-23 18:50:01.010 30 ERROR futurist.periodics work()
2024-04-23 18:50:01.010 30 ERROR futurist.periodics File "/var/lib/kolla/venv/lib64/python3.9/site-packages/futurist/periodics.py", line 64, in __call__
2024-04-23 18:50:01.010 30 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs)
2024-04-23 18:50:01.010 30 ERROR futurist.periodics File "/var/lib/kolla/venv/lib64/python3.9/site-packages/futurist/periodics.py", line 178, in decorator
2024-04-23 18:50:01.010 30 ERROR futurist.periodics return f(*args, **kwargs)
2024-04-23 18:50:01.010 30 ERROR futurist.periodics File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 977, in add_gw_port_info_to_logical_router_port
2024-04-23 18:50:01.010 30 ERROR futurist.periodics ext_gw['network_id'] for ext_gw in router['external_gateways']]
2024-04-23 18:50:01.010 30 ERROR futurist.periodics KeyError: 'external_gateways'

Using Kolla images built today (from latest stable/2023.2 Neutron code tarballs) and OVN 23.09