neutron

[OVN] neutron-ovn-db-sync-util wipes out hash ring nodes

Bug #1879950 reported by Jakub Libosvar on 2020-05-21

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	Jakub Libosvar

Bug Description

Neutron return 500 when attempting to change name of an existing router:

$ openstack router set --name router-flat 29ead671-3957-4bf2-9dbf-e1bff0a12ab2
HttpException: 500: Server Error for url: http://192.168.121.189:9696/v2.0/routers/29ead671-3957-4bf2-9dbf-e1bff0a12ab2, Request Failed: internal server error while processing your request.

May 21 13:29:47 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] Running txn n=1 command(idx=0): DbSetCommand(table=Chassis, record=141047c5-57f5-443a-a50f-db65612c047b, col_values=(('external_ids', {'neutron:liveness_check_at': '2020-05-21T13:29:47.048155+00:00'}),)) {{(pid=3400) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] Running txn n=1 command(idx=0): DbSetCommand(table=Chassis, record=141047c5-57f5-443a-a50f-db65612c047b, col_values=(('external_ids', {'neutron:metadata_liveness_check_at': '2020-05-21T13:29:47.055462+00:00'}),)) {{(pid=3400) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-f37ed5e0-ef6a-4fc0-9670-537d4935f18e None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-cb4458e4-ff28-4d3d-ad22-21e13625998f None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-cb4458e4-ff28-4d3d-ad22-21e13625998f None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-f37ed5e0-ef6a-4fc0-9670-537d4935f18e None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster
May 21 13:29:47 devstack neutron-server[3378]: DEBUG neutron.db.agents_db [None req-29c44001-08f7-4ff1-9885-2a8fc4dc7e61 None None] Agent healthcheck: found 2 active agents {{(pid=3400) agent_health_check /opt/stack/neutron/neutron/db/agents_db.py:327}}
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-38aed43c-4b9a-4521-9e69-349701e72f62 service neutron] 192.168.121.189 "GET /v2.0/ports?device_id=aff97800-74d0-4339-83d5-3ae53bbc9d97&fields=binding%3Ahost_id&fields=binding%3Avif_type HTTP/1.1" status: 200 len: 265 time: 0.0434918
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-d77dcd0a-bac0-49ec-ba88-73e2a220cb6c service neutron] 192.168.121.189 "GET /v2.0/ports?tenant_id=9635d9bacb954ae78b28222e6eba4070&device_id=aff97800-74d0-4339-83d5-3ae53bbc9d97 HTTP/1.1" status: 200 len: 1456 time: 0.0404735
May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-9312b103-df29-4f4c-9ba6-93dbf47080af service neutron] Attributes excluded by policy engine: ['vlan_transparent'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}}
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-9312b103-df29-4f4c-9ba6-93dbf47080af service neutron] 192.168.121.189 "GET /v2.0/networks?id=336d1bc4-69c6-4b70-b48c-36b190d979db HTTP/1.1" status: 200 len: 930 time: 0.0695074
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-48e27f2d-f333-4b8a-b992-fa46d6a849ba service neutron] 192.168.121.189 "GET /v2.0/floatingips?fixed_ip_address=10.0.0.2&port_id=3d0ce33c-fcf8-4073-95b5-ebf4c5722c74 HTTP/1.1" status: 200 len: 1032 time: 0.0478008
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-2354eb84-e80c-4abb-91f3-25701844207b service neutron] 192.168.121.189 "GET /v2.0/floatingips?fixed_ip_address=fd4f%3A707c%3A5223%3A0%3Af816%3A3eff%3Afeac%3Aa720&port_id=3d0ce33c-fcf8-4073-95b5-ebf4c5722c74 HTTP/1.1" status: 200 len: 217 time: 0.0307662
May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] Attributes excluded by policy engine: ['shared'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}}
May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] Attributes excluded by policy engine: ['shared'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}}
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] 192.168.121.189 "GET /v2.0/subnets?id=5cce10c1-b3c3-4e3f-b709-2066dcc2c785&id=28dfb2ed-2662-4638-a2d4-9369c64b2f3c HTTP/1.1" status: 200 len: 1599 time: 0.0369835
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-a6779a14-962e-461e-b806-be974da65e35 service neutron] 192.168.121.189 "GET /v2.0/ports?network_id=336d1bc4-69c6-4b70-b48c-36b190d979db&device_owner=network%3Adhcp HTTP/1.1" status: 200 len: 1441 time: 0.0365410
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-7db1d18d-7e32-4a44-a9e8-e855fd16a079 service neutron] 192.168.121.189 "GET /v2.0/ports?network_id=336d1bc4-69c6-4b70-b48c-36b190d979db&device_owner=network%3Adhcp HTTP/1.1" status: 200 len: 1441 time: 0.0359824
May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8a3492de-0578-4c0b-bab6-4bf86f630196 service neutron] 192.168.121.189 "GET /v2.0/networks/336d1bc4-69c6-4b70-b48c-36b190d979db?fields=segments HTTP/1.1" status: 200 len: 212 time: 0.0682638
May 21 13:29:49 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-211dc48a-64e4-4e49-b630-88d45c543861 service neutron] 192.168.121.189 "GET /v2.0/networks/336d1bc4-69c6-4b70-b48c-36b190d979db?fields=provider%3Aphysical_network&fields=provider%3Anetwork_type HTTP/1.1" status: 200 len: 277 time: 0.0675299
May 21 13:29:56 devstack neutron-server[3378]: DEBUG neutron.wsgi [-] (3395) accepted ('192.168.121.189', 54086) {{(pid=3395) server /usr/local/lib/python3.6/site-packages/eventlet/wsgi.py:985}}
May 21 13:29:56 devstack neutron-server[3378]: INFO neutron.api.v2.resource [None req-c451591f-e2b5-45d3-964e-e26a45f847bb demo admin] show failed (client error): The resource could not be found.
May 21 13:29:56 devstack neutron-server[3378]: INFO neutron.wsgi [None req-c451591f-e2b5-45d3-964e-e26a45f847bb demo admin] 192.168.121.189 "GET /v2.0/routers/router1 HTTP/1.1" status: 404 len: 312 time: 0.0663443
May 21 13:29:57 devstack neutron-server[3378]: INFO neutron.wsgi [None req-82148186-c9ef-4e98-977c-0efbfeafbf84 demo admin] 192.168.121.189 "GET /v2.0/routers?name=router1 HTTP/1.1" status: 200 len: 859 time: 0.1023729
May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron.api.v2.base [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Request body: {'router': {'name': 'router-flat'}} {{(pid=3395) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:719}}
May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, after_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Chassis candidates with external connectivity: ['30a0cb52-356d-4f2d-be32-821652201fc7'] {{(pid=3395) get_candidates_for_scheduling /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py:1412}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.scheduler.l3_ovn_scheduler [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Gateway lrp-d7c38c52-9350-47bf-9275-118a24538a9f scheduled on chassis ['30a0cb52-356d-4f2d-be32-821652201fc7'] {{(pid=3395) _schedule_gateway /opt/stack/neutron/neutron/scheduler/l3_ovn_scheduler.py:85}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(name=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, resource={'id': '29ead671-3957-4bf2-9dbf-e1bff0a12ab2', 'name': 'router-flat', 'tenant_id': '9635d9bacb954ae78b28222e6eba4070', 'admin_state_up': True, 'status': 'ACTIVE', 'external_gateway_info': {'network_id': 'c1b6e05b-b622-4565-8a46-c03cd25b7c4b', 'external_fixed_ips': [{'subnet_id': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f', 'ip_address': '172.24.4.51'}, {'subnet_id': 'c2e143e5-8f48-45e0-b23e-6bb497381e7b', 'ip_address': '2001:db8::1'}], 'enable_snat': True}, 'gw_port_id': 'd7c38c52-9350-47bf-9275-118a24538a9f', 'description': '', 'routes': [], 'tags': [], 'created_at': '2020-05-20T11:27:32Z', 'updated_at': '2020-05-21T13:29:57Z', 'revision_number': 9, 'project_id': '9635d9bacb954ae78b28222e6eba4070'}, resource_type=routers, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DeleteLRouterExtGwCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DeleteNATRuleInLRouterCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, type=snat, logical_ip=10.0.0.0/26, external_ip=172.24.4.51, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=0.0.0.0/0, nexthop=172.24.4.1, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=::/0, nexthop=2001:db8::2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelLRouterPortCommand(name=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): AddLRouterPortCommand(name=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, may_exist=True, columns={'mac': 'fa:16:3e:54:cf:1a', 'networks': ['2001:db8::1/64', '172.24.4.51/24'], 'external_ids': {'neutron:revision_number': '6', 'neutron:subnet_ids': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f c2e143e5-8f48-45e0-b23e-6bb497381e7b', 'neutron:network_name': 'neutron-c1b6e05b-b622-4565-8a46-c03cd25b7c4b', 'neutron:router_name': '29ead671-3957-4bf2-9dbf-e1bff0a12ab2'}, 'options': {}, 'gateway_chassis': ['30a0cb52-356d-4f2d-be32-821652201fc7']}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): SetLRouterPortInLSwitchPortCommand(lswitch_port=d7c38c52-9350-47bf-9275-118a24538a9f, lrouter_port=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): AddStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'ip_prefix': '0.0.0.0/0', 'nexthop': '172.24.4.1', 'external_ids': {'neutron:is_ext_gw': 'true', 'neutron:subnet_id': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f'}}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): AddStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'ip_prefix': '::/0', 'nexthop': '2001:db8::2', 'external_ids': {'neutron:is_ext_gw': 'true', 'neutron:subnet_id': 'c2e143e5-8f48-45e0-b23e-6bb497381e7b'}}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): AddNATRuleInLRouterCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'type': 'snat', 'logical_ip': '10.0.0.0/26', 'external_ip': '172.24.4.51'}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): UpdateLRouterCommand(name=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'external_ids': {'neutron:router_name': 'router-flat', 'neutron:gw_port_id': 'd7c38c52-9350-47bf-9275-118a24538a9f', 'neutron:revision_number': '9'}, 'enabled': True}, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=0.0.0.0/0, nexthop=172.24.4.1, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=::/0, nexthop=2001:db8::2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}}
May 21 13:29:58 devstack neutron-server[3378]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
May 21 13:29:58 devstack neutron-server[3378]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Traceback (most recent call last):
May 21 13:29:58 devstack neutron-server[3378]: File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
May 21 13:29:58 devstack neutron-server[3378]: txn.results.put(txn.do_commit())
May 21 13:29:58 devstack neutron-server[3378]: File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit
May 21 13:29:58 devstack neutron-server[3378]: raise RuntimeError(msg)
May 21 13:29:58 devstack neutron-server[3378]: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
May 21 13:29:58 devstack neutron-server[3378]:
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Unable to update router 29ead671-3957-4bf2-9dbf-e1bff0a12ab2. Error: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Unable to update lrouter for 29ead671-3957-4bf2-9dbf-e1bff0a12ab2: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin Traceback (most recent call last):
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin original_router)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin 'error': e})
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin context, router_id, added, removed, txn=txn)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin next(self.gen)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin yield t
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin next(self.gen)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin del self._nested_txns_map[cur_thread_id]
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin self.result = self.commit()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin raise result.ex
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin txn.results.put(txn.do_commit())
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin raise RuntimeError(msg)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.db.extraroute_db [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Added routes are [] {{(pid=3395) _update_extra_routes /opt/stack/neutron/neutron/db/extraroute_db.py:113}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.db.extraroute_db [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Removed routes are [] {{(pid=3395) _update_extra_routes /opt/stack/neutron/neutron/db/extraroute_db.py:121}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}}
May 21 13:29:58 devstack neutron-server[3378]: ERROR root [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router\n original_router)\n', ' File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router\n \'error\': e})\n', ' File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router\n context, router_id, added, removed, txn=txn)\n', ' File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__\n next(self.gen)\n', ' File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction\n yield t\n', ' File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__\n next(self.gen)\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction\n del self._nested_txns_map[cur_thread_id]\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__\n self.result = self.commit()\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit\n raise result.ex\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run\n txn.results.put(txn.do_commit())\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit\n raise RuntimeError(msg)\n', 'RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}\n']: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] update failed: No details.: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource Traceback (most recent call last):
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource original_router)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource 'error': e})
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource context, router_id, added, removed, txn=txn)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource next(self.gen)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource yield t
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource next(self.gen)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource del self._nested_txns_map[cur_thread_id]
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.result = self.commit()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise result.ex
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource txn.results.put(txn.do_commit())
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise RuntimeError(msg)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource During handling of the above exception, another exception occurred:
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource Traceback (most recent call last):
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource result = method(request=request, **args)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 625, in update
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return self._update(request, id, body, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 685, in _update
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource obj = obj_updater(request.context, id, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 148, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource revert_router)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/extraroute_db.py", line 72, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource context, id, router)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/l3_db.py", line 1963, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource id, router)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return method(*args, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise()
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/l3_db.py", line 287, in update_router
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource updated = self._make_router_dict(router_db)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/l3_db.py", line 198, in _make_router_dict
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource 'network_id': router.gw_port['network_id'],
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 287, in __get__
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return self.impl.get(instance_state(instance), dict_)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 723, in get
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource value = self.callable_(state, passive)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 710, in _load_for_state
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource % (orm_util.state_str(state), self.key)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource
May 21 13:29:58 devstack neutron-server[3378]: INFO neutron.wsgi [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] 192.168.121.189 "PUT /v2.0/routers/29ead671-3957-4bf2-9dbf-e1bff0a12ab2 HTTP/1.1" status: 500 len: 368 time: 1.8027527

Tags:

Revision history for this message

Jakub Libosvar (libosvar) wrote on 2020-05-21:

The trace above is wrong, was caused by wiped hash ring nodes from the DB. That was caused by running neutron-ovn-db-sync-util

summary:

- Unable to update router name with ovn
+ neutron-ovn-db-sync-util wipes out hash ring nodes

Jakub Libosvar (libosvar) on 2020-05-21

Changed in neutron:
assignee:	nobody → Jakub Libosvar (libosvar)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-21: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/729997

Changed in neutron:
status:	New → In Progress

Maciej Jozefczyk (maciejjozefczyk) on 2020-05-22

summary:	- neutron-ovn-db-sync-util wipes out hash ring nodes + [OVN] neutron-ovn-db-sync-util wipes out hash ring nodes
Changed in neutron:
importance:	Undecided → High

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-22: Fix merged to neutron (master)

Reviewed: https://review.opendev.org/729997
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f5c391077139b9ca61349b59e1e5270b42125cd2
Submitter: Zuul
Branch: master

commit f5c391077139b9ca61349b59e1e5270b42125cd2
Author: Jakub Libosvar <email address hidden>
Date: Thu May 21 15:16:54 2020 +0000

ovn: Override clean_hash_ring for sync util

    ML2 OVN Mechanism driver wipes all the nodes from ovn_hash_ring table on
    initialization. The DB sync untility inherits from the mechanism driver
    calling the cleanup whenever sync util is executed. This leads to nodes
    being deleted while neutron-server is typically running, causing serious
    issues to neutron-server until it is restarted.

This patch overrides the clean method to no-op.

    Change-Id: I5ea8278c0387009ff429572662d64e7664f493e7
    Closes-Bug: #1879950
    Signed-off-by: Jakub Libosvar <email address hidden>

Changed in neutron:
status:	In Progress → Fix Released

Bernard Cafarelli (bcafarel) on 2020-06-04

tags:

added: neutron-proactive-backport-potential

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.