neutron

[ovn] Tempest tests failing while creating security group driver with KeyError: 'remote_address_group_id'

Bug #1906500 reported by yatin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Alfredo Moralejo

Bug Description

This is failing post https://review.opendev.org/c/openstack/neutron/+/751110, detected in a packstack job, it fails with below Traceback:-

2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource Traceback (most recent call last):
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/api/v2/resource.py", line 98, in resource
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource result = method(request=request, **args)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 437, in create
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource self.force_reraise()
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource raise value
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource self.force_reraise()
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource raise value
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource self.force_reraise()
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource raise value
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 547, in _create
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource objs = do_create(body, bulk=True)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 543, in do_create
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource request.context, reservation.reservation_id)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource self.force_reraise()
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource raise value
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 536, in do_create
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return obj_creator(request.context, **kwargs)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/db/securitygroups_rpc_base.py", line 67, in create_security_group_rule_bulk
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource context, security_group_rules)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return method(*args, **kwargs)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource self.force_reraise()
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource raise value
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource self.force_reraise()
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource raise value
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource self.force_reraise()
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource raise value
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/db/securitygroups_db.py", line 387, in create_security_group_rule_bulk_native
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource context, security_group_rules)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/db/securitygroups_db.py", line 666, in _validate_security_group_rules
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource self._validate_security_group_rule(context, rule)
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/db/securitygroups_db.py", line 640, in _validate_security_group_rule
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource remote_address_group_id = rule['remote_address_group_id']
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource KeyError: 'remote_address_group_id'
2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource

Example logs:-
Example:- https://logserver.rdoproject.org/ci.centos.org/weirdo-generic-packstack-scenario001/11909/weirdo-project/logs/latest/tempest.log.txt.gz
https://logserver.rdoproject.org/ci.centos.org/weirdo-generic-packstack-scenario001/11909/weirdo-project/logs/neutron/server.log.txt.gz

The issue didn't got detected in neutron devstack jobs running with ovn as extension security-groups-remote-address-group got loaded via 'logger' mechanism driver. security-groups-remote-address-group needs to be loaded with ovn driver too to get security group creation working

Revision history for this message
yatin (yatinkarel) wrote :

Fix is proposed:- https://review.opendev.org/c/openstack/neutron/+/765101

Changed in neutron:
status: New → In Progress
Bernard Cafarelli (bcafarel)
tags: added: ovn
Changed in neutron:
assignee: nobody → Alfredo Moralejo (amoralej)
importance: Undecided → High
Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.0.0.0rc1

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

Brian Haley (brian-haley)
Changed in neutron:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.