AttributeError: 'Subnet' object has no attribute '_obj_network_id'

Bug #1784342 reported by Tobias Urdin on 2018-07-30
38
This bug affects 7 people
Affects Status Importance Assigned to Milestone
neutron
Undecided
Unassigned
neutron (Ubuntu)
Undecided
Unassigned

Bug Description

Running rally caused subnets to be created without a network_id causing this AttributeError.

OpenStack Queens RDO packages
[root@controller1 ~]# rpm -qa | grep -i neutron
python-neutron-12.0.2-1.el7.noarch
openstack-neutron-12.0.2-1.el7.noarch
python2-neutron-dynamic-routing-12.0.1-1.el7.noarch
python2-neutron-lib-1.13.0-1.el7.noarch
openstack-neutron-dynamic-routing-common-12.0.1-1.el7.noarch
python2-neutronclient-6.7.0-1.el7.noarch
openstack-neutron-bgp-dragent-12.0.1-1.el7.noarch
openstack-neutron-common-12.0.2-1.el7.noarch
openstack-neutron-ml2-12.0.2-1.el7.noarch

MariaDB [neutron]> select project_id, id, name, network_id, cidr from subnets where network_id is null;

+----------------------------------+--------------------------------------+---------------------------+------------+-------------+

| project_id | id | name | network_id | cidr |

+----------------------------------+--------------------------------------+---------------------------+------------+-------------+

| b80468629bc5410ca2c53a7cfbf002b3 | 7a23c72b-3df8-4641-a494-af7642563c8e | s_rally_1e4bebf1_1s3IN6mo | NULL | 1.9.13.0/24 |

| b80468629bc5410ca2c53a7cfbf002b3 | f7a57946-4814-477a-9649-cc475fb4e7b2 | s_rally_1e4bebf1_qWSFSMs9 | NULL | 1.5.20.0/24 |

+----------------------------------+--------------------------------------+---------------------------+------------+-------------+

2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation [req-c921b9fb-499b-41c1-9103-93e71a70820c b6b96932bbef41fdbf957c2dc01776aa 050c556faa5944a8953126c867313770 - default default] GET failed.: AttributeError: 'Subnet' object has no attribute '_obj_network_id'
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 678, in __call__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 569, in invoke_controller
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 131, in index
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return self.get(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 141, in get
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation **query_params)}
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 161, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1046, in get_subnets
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation for subnet_object in subnet_objs
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py", line 134, in _make_subnet_dict
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation 'network_id': subnet['network_id'],
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 774, in __getitem__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return getattr(self, name)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 68, in getter
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return getattr(self, attrname)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation AttributeError: 'Subnet' object has no attribute '_obj_network_id'
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation
2018-07-30 10:35:13.355 42618 INFO neutron.wsgi [req-c921b9fb-499b-41c1-9103-93e71a70820c b6b96932bbef41fdbf957c2dc01776aa 050c556faa5944a8953126c867313770 - default default] 172.20.104.11 "GET /v2.0/subnets HTTP/1.1" status: 500 len: 368 time: 1.3515501

will update bug if i find anything else in log files when rally tried creating these

description: updated
YAMAMOTO Takashi (yamamoto) wrote :

what kind of rally tasks did you run?

Changed in neutron:
status: New → Incomplete
Tobias Urdin (tobias-urdin) wrote :

It was neutron CRUD test. NeutronNetworks - create, update, list and delete network, subnets and ports in burst

YAMAMOTO Takashi (yamamoto) wrote :

rally-jobs/neutron.yaml in rally-openstack repo, you mean?

Tobias Urdin (tobias-urdin) wrote :

I'm not sure, I was not the one running the tests, I think this was caused by a MySQL timeout issue on our side however unless the creation of a subnet and the population of network_id is separat, or if the rally tests removes a subnet from a network and adds it to another then it's probably not an issue in the Neutron side.

Nate Johnston (nate-johnston) wrote :

I have not seen this kind of an issue before. Do you see these kinds of results consistently - in other words, if you run the rally tests 10 times, it experiences this issue ten out of ten of the attempts, even if the environment has been restacked in the meantime? We certainly run rally tests in the gate using pecan_wsgi on the stable/queens branch and they succeed; [1] is an example of such a run.

[1] http://logs.openstack.org/07/584307/1/check/neutron-rally-neutron/13b0e0c/

Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
Drew Freiberger (afreiberger) wrote :

We have had an incident of a network being deleted, but the recursive subnet deletions failing under load happening in a production Xenial-Queens cloud.

Can this somehow be mitigated by api update to require additional user intervention in case of existing subnets?

Changed in neutron:
status: Expired → Confirmed
Xav Paice (xavpaice) wrote :

Seen again in a Bionic-rocky cloud. One subnet was in the table without a matching network id. Deleted that row, and things worked again.

FWIW, this broke 'openstack subnet list'.

Xav Paice (xavpaice) wrote :

subscribed field-high, added Ubuntu Neutron package, since this has occurred in multiple production sites.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in neutron (Ubuntu):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers