AttributeError: 'Subnet' object has no attribute '_obj_network_id'

[Tobias Urdin]

Running rally caused subnets to be created without a network_id causing this AttributeError.

OpenStack Queens RDO packages
[root@controller1 ~]# rpm -qa | grep -i neutron
python-neutron-12.0.2-1.el7.noarch
openstack-neutron-12.0.2-1.el7.noarch
python2-neutron-dynamic-routing-12.0.1-1.el7.noarch
python2-neutron-lib-1.13.0-1.el7.noarch
openstack-neutron-dynamic-routing-common-12.0.1-1.el7.noarch
python2-neutronclient-6.7.0-1.el7.noarch
openstack-neutron-bgp-dragent-12.0.1-1.el7.noarch
openstack-neutron-common-12.0.2-1.el7.noarch
openstack-neutron-ml2-12.0.2-1.el7.noarch

MariaDB [neutron]> select project_id, id, name, network_id, cidr from subnets where network_id is null;

+----------------------------------+--------------------------------------+---------------------------+------------+-------------+

| project_id | id | name | network_id | cidr |

+----------------------------------+--------------------------------------+---------------------------+------------+-------------+

| b80468629bc5410ca2c53a7cfbf002b3 | 7a23c72b-3df8-4641-a494-af7642563c8e | s_rally_1e4bebf1_1s3IN6mo | NULL | 1.9.13.0/24 |

| b80468629bc5410ca2c53a7cfbf002b3 | f7a57946-4814-477a-9649-cc475fb4e7b2 | s_rally_1e4bebf1_qWSFSMs9 | NULL | 1.5.20.0/24 |

+----------------------------------+--------------------------------------+---------------------------+------------+-------------+

2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation [req-c921b9fb-499b-41c1-9103-93e71a70820c b6b96932bbef41fdbf957c2dc01776aa 050c556faa5944a8953126c867313770 - default default] GET failed.: AttributeError: 'Subnet' object has no attribute '_obj_network_id'
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 678, in __call__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 569, in invoke_controller
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 131, in index
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return self.get(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 141, in get
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation **query_params)}
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 161, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1046, in get_subnets
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation for subnet_object in subnet_objs
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py", line 134, in _make_subnet_dict
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation 'network_id': subnet['network_id'],
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 774, in __getitem__
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return getattr(self, name)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 68, in getter
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation return getattr(self, attrname)
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation AttributeError: 'Subnet' object has no attribute '_obj_network_id'
2018-07-30 10:35:13.351 42618 ERROR neutron.pecan_wsgi.hooks.translation
2018-07-30 10:35:13.355 42618 INFO neutron.wsgi [req-c921b9fb-499b-41c1-9103-93e71a70820c b6b96932bbef41fdbf957c2dc01776aa 050c556faa5944a8953126c867313770 - default default] 172.20.104.11 "GET /v2.0/subnets HTTP/1.1" status: 500 len: 368 time: 1.3515501

will update bug if i find anything else in log files when rally tried creating these

[YAMAMOTO Takashi (yamamoto)]

what kind of rally tasks did you run?

[Tobias Urdin (tobias-urdin)]

It was neutron CRUD test. NeutronNetworks - create, update, list and delete network, subnets and ports in burst

[YAMAMOTO Takashi (yamamoto)]

rally-jobs/neutron.yaml in rally-openstack repo, you mean?

[Tobias Urdin (tobias-urdin)]

I'm not sure, I was not the one running the tests, I think this was caused by a MySQL timeout issue on our side however unless the creation of a subnet and the population of network_id is separat, or if the rally tests removes a subnet from a network and adds it to another then it's probably not an issue in the Neutron side.

[Nate Johnston (nate-johnston)]

I have not seen this kind of an issue before. Do you see these kinds of results consistently - in other words, if you run the rally tests 10 times, it experiences this issue ten out of ten of the attempts, even if the environment has been restacked in the meantime? We certainly run rally tests in the gate using pecan_wsgi on the stable/queens branch and they succeed; [1] is an example of such a run.

[1] http://logs.openstack.org/07/584307/1/check/neutron-rally-neutron/13b0e0c/

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Drew Freiberger (afreiberger)]

We have had an incident of a network being deleted, but the recursive subnet deletions failing under load happening in a production Xenial-Queens cloud.

Can this somehow be mitigated by api update to require additional user intervention in case of existing subnets?

[Xav Paice (xavpaice)]

Seen again in a Bionic-rocky cloud. One subnet was in the table without a matching network id. Deleted that row, and things worked again.

FWIW, this broke 'openstack subnet list'.

[Xav Paice (xavpaice)]

subscribed field-high, added Ubuntu Neutron package, since this has occurred in multiple production sites.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Page (james-page)]

I had a dig around in the database - the subnets table has a foreign key reference to the networks table:

CONSTRAINT `subnets_ibfk_1` FOREIGN KEY (`network_id`) REFERENCES `networks` (`id`),

which I believe would prevent the deletion of the network from the networks table until the subnet was deleted?

[James Page (james-page)]

Full create table command:

CREATE TABLE `subnets` (
  `project_id` varchar(255) DEFAULT NULL,
  `id` varchar(36) NOT NULL,
  `name` varchar(255) DEFAULT NULL,
  `network_id` varchar(36) NOT NULL,
  `ip_version` int(11) NOT NULL,
  `cidr` varchar(64) NOT NULL,
  `gateway_ip` varchar(64) DEFAULT NULL,
  `enable_dhcp` tinyint(1) DEFAULT NULL,
  `ipv6_ra_mode` enum('slaac','dhcpv6-stateful','dhcpv6-stateless') DEFAULT NULL,
  `ipv6_address_mode` enum('slaac','dhcpv6-stateful','dhcpv6-stateless') DEFAULT NULL,
  `subnetpool_id` varchar(36) DEFAULT NULL,
  `standard_attr_id` bigint(20) NOT NULL,
  `segment_id` varchar(36) DEFAULT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `uniq_subnets0standard_attr_id` (`standard_attr_id`),
  KEY `network_id` (`network_id`),
  KEY `ix_subnets_subnetpool_id` (`subnetpool_id`),
  KEY `segment_id` (`segment_id`),
  KEY `ix_subnets_project_id` (`project_id`),
  CONSTRAINT `subnets_ibfk_1` FOREIGN KEY (`network_id`) REFERENCES `networks` (`id`),
  CONSTRAINT `subnets_ibfk_2` FOREIGN KEY (`standard_attr_id`) REFERENCES `standardattributes` (`id`) ON DELETE CASCADE,
  CONSTRAINT `subnets_ibfk_3` FOREIGN KEY (`segment_id`) REFERENCES `networksegments` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

[James Page (james-page)]

Ah - this behaviour was enforced @ train

see bug 1839658

[uchenily (uchenily)]

Hi, James, do you know how to reproduce this error? I think the foreign-key will prevent the deletion from the networks table until the subnet was deleted, so it seems that the field network_id will not be null in any time. `https://review.opendev.org/#/c/677221/` this patch force `network_id` not to be null, but how it works?

[Christian Sarrasin (sxc731)]

Anyone who comes across this issue, resulting in denial of service from neutron-server with the exceptions mentioned in the OP, it's reasonably painless to workaround; simply delete the subnet(s) with NULL network_id:

delete from neutron.subnets where network_id is null;

No need to restart neutron-server afterwards. Clearly this is just a workaround...