neutron

"subnet" register in the DB can have network_id=NULL

Bug #1839658 reported by Rodolfo Alonso
26
This bug affects 9 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
Rodolfo Alonso

Bug Description

(A bit more readable version of the following text is found at http://paste.openstack.org/show/756523/)

When deleting a cluster with several networks and subnets, the following error was found when executing the shell command 'openstack subnet list':

019-08-06 08:26:10.000 28 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200 len: 248 time: 0.0012720
2019-08-06 08:26:10.803 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.15', 41484) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] GET failed.: AttributeError: 'Subnet' object has no attribute '_obj_network_id'
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 131, in index
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return self.get(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 141, in get
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation **query_params)}
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1075, in get_subnets
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation for subnet_object in subnet_objs
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py", line 134, in _make_subnet_dict
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation 'network_id': subnet['network_id'],
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 774, in __getitem__
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return getattr(self, name)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 68, in getter
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return getattr(self, attrname)
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation AttributeError: 'Subnet' object has no attribute '_obj_network_id'
2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation
2019-08-06 08:26:11.523 28 INFO neutron.wsgi [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] 172.17.1.15 "GET /v2.0/subnets HTTP/1.1" status: 500 len: 344 time: 0.7188911
2019-08-06 08:26:12.002 34 DEBUG neutron.wsgi [-] (34) accepted ('172.17.1.15', 41734) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2019-08-06 08:26:12.004 34 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200 len: 248 time: 0.0012681

Which means that some subnets where left behind without an associated network. In this case ns/openshift-kube-controller-manager-subnet:
MariaDB [ovs_neutron]> select * from subnets;
+----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+
| project_id | id | name | network_id | ip_version | cidr | gateway_ip | enable_dhcp | ipv6_ra_mode | ipv6_address_mode | subnetpool_id | standard_attr_id | segment_id |
+----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+
| 755fea49ab0643d0b731073e87e739e0 | 26953b3b-fa6c-46a0-9e60-6a7d5cadf580 | lb-mgmt-subnet | 4bfa167b-fcf9-4ad8-b5f3-99c6141b20a4 | 4 | 172.24.0.0/16 | 172.24.0.1 | 1 | NULL | NULL | NULL | 8 | NULL |
| dcb34bf98dea46928294ff6762d9c167 | 5ea30bc8-6eb1-4494-96e6-b37d9a70fb09 | ns/openshift-kube-controller-manager-subnet | NULL | 4 | 10.128.34.0/24 | 10.128.34.1 | 0 | NULL | NULL | cc48743c-e44c-48ee-b29c-5ceb53f60fbd | 9493 | NULL |
| 755fea49ab0643d0b731073e87e739e0 | 8f4a9baf-ad79-4730-9449-671706f24d7a | external_subnet | 004f76a6-3867-46f4-8718-5498634945a3 | 4 | 10.0.0.0/24 | 10.0.0.1 | 0 | NULL | NULL | NULL | 27 | NULL |
+----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+

This is only happening rarely, not 100% of the time.

See original description
Rodolfo Alonso (rodolfo-alonso-hernandez)
Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
Akihiro Motoki (amotoki)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/677221

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/677221
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4b6c2246c09eccbb193b4a59a8dfee9e24f97bed
Submitter: Zuul
Branch: master

commit 4b6c2246c09eccbb193b4a59a8dfee9e24f97bed
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Aug 19 13:09:20 2019 +0000

    Force "network_id" in "subnet" DB registers

    The "subnet" OVO does not allow to have an empty (None) "network_id" but
    the DB "subnet" table allows to have this parameter empty (NULL) in the
    database. In order to avoid any problem like the one described in the
    bug, this patch ensures the database "subnet" register does always have
    a "network_id" value and if a "network" register is being deleted, all
    related "subnet" registers are checked first.

    Change-Id: Iad210f0585b4201fdb87187b44a9b42267b58db4
    Closes-Bug: #1839658

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 15.0.0.0b1

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Here we are modifying the DB. Do you think this change can be ported to previous versions? I would prefer not to cherry-pick it.

Slawek Kaplonski (slaweq)
tags: removed: neutron-proactive-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.