neutron

Designate integration - internal server error in Neutron

Bug #1891309 reported by Slawek Kaplonski on 2020-08-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	Dr. Jens Harbott

Bug Description

I found in the neutron-tempest-plugin-designate-scenario job that test neutron_tempest_plugin.scenario.test_dns_integration.DNSIntegrationTests.test_fip failed due to internal server error in Neutron server.

Failed job: https://fc098855e3982674115d-9f073b45b43178f46f80f0c5ee75fe29.ssl.cf5.rackcdn.com/640258/26/check/neutron-tempest-plugin-designate-scenario/db9e07e/testr_results.html

In neutron server logs I can see:

Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource [None req-b440f68b-a3e2-4da5-8e2a-dfbb1331855b tempest-DNSIntegrationTests-1379368462 tempest-DNSIntegrationTests-1379368462] create failed: No details.: designateclient.exceptions.Conflict: Duplicate Zone
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource Traceback (most recent call last):
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/services/externaldns/drivers/designate/driver.py", line 124, in create_record_set
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource [recordset_name])
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/designateclient/v2/recordsets.py", line 46, in create
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource name, zone_info = self._canonicalize_record_name(zone, name)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/designateclient/v2/recordsets.py", line 30, in _canonicalize_record_name
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource zone_info = self.client.zones.get(zone)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/designateclient/v2/zones.py", line 54, in get
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource zone = v2_utils.resolve_by_name(self.list, zone)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/designateclient/v2/utils.py", line 40, in resolve_by_name
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource raise exceptions.NotFound("Name %s didn't resolve" % name)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource designateclient.exceptions.NotFound: Name 5.24.172.in-addr.arpa. didn't resolve
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource During handling of the above exception, another exception occurred:
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource Traceback (most recent call last):
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource result = method(request=request, **args)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 437, in create
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource self.force_reraise()
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource raise value
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 154, in wrapper
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource self.force_reraise()
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource raise value
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 142, in wrapper
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource self.force_reraise()
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource raise value
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 556, in _create
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource obj = do_create(body)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 538, in do_create
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource request.context, reservation.reservation_id)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource self.force_reraise()
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource raise value
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 531, in do_create
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource return obj_creator(request.context, **kwargs)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/services/l3_router/l3_router_plugin.py", line 185, in create_floatingip
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource initial_status=n_const.FLOATINGIP_STATUS_DOWN)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 233, in wrapped
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource return method(*args, **kwargs)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource self.force_reraise()
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource raise value
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 154, in wrapper
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource self.force_reraise()
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource raise value
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 142, in wrapper
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource self.force_reraise()
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource raise value
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 1141, in create_floatingip
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource context, floatingip, initial_status)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/l3_db.py", line 1377, in _create_floatingip
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource dns_data)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/dns_db.py", line 117, in _process_dns_floatingip_create_postcommit
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource [floatingip_data['floating_ip_address']])
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/dns_db.py", line 247, in _add_ips_to_external_dns_service
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource ips)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/services/externaldns/drivers/designate/driver.py", line 128, in create_record_set
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource description=in_addr_zone_description)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/designateclient/v2/zones.py", line 46, in create
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource return self._post('/zones', data=data)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/designateclient/client.py", line 58, in _post
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource resp, body = self.client.session.post(url, **kwargs)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/keystoneauth1/adapter.py", line 401, in post
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource return self.request(url, 'POST', **kwargs)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/designateclient/v2/client.py", line 97, in request
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource raise exceptions.Conflict(**response_payload)
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource designateclient.exceptions.Conflict: Duplicate Zone
Aug 10 08:16:57.234512 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: ERROR neutron.api.v2.resource
Aug 10 08:16:57.245005 ubuntu-bionic-rax-iad-0019178926 neutron-server[12604]: INFO neutron.wsgi [None req-b440f68b-a3e2-4da5-8e2a-dfbb1331855b tempest-DNSIntegrationTests-1379368462 tempest-DNSIntegrationTests-1379368462] 10.209.97.155 "POST /v2.0/floatingips HTTP/1.1" status: 500 len: 363 time: 3.5691445

Tags:

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2020-08-12:

Another example of same issue: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_01a/742363/4/check/neutron-tempest-plugin-designate-scenario/01ae4ed/testr_results.html - so it is reproducible, at least in the gate.

Revision history for this message

Dr. Jens Harbott (j-harbott) wrote on 2020-08-18:

It seems that this is a race happening when two PTRs are being created at (almost) the same time and the needed PTR zone doesn't exist yet. Both API processes then check whether the zone exists, see that it isn't there, try to create it, but obviously only one of them succeeds.

Not sure how to solve it, one option might be to ignore the possible error on zone creation and try to go on with creating the recordset anyway.

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2020-08-18:

Thx Jens for checking that. Your proposal seems reasonable for me. Do You want to propose a patch to address this issue or do You want me to do that?

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-08-19: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/746910

Changed in neutron:
assignee:	nobody → Dr. Jens Harbott (j-harbott)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-05: Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/746910
Committed: https://opendev.org/openstack/neutron/commit/ca1c6fd69bda5f392712a8a966bf4f9b85ecb57a
Submitter: "Zuul (22348)"
Branch: master

commit ca1c6fd69bda5f392712a8a966bf4f9b85ecb57a
Author: Jens Harbott <email address hidden>
Date: Wed Aug 19 14:39:35 2020 +0200

designate: allow PTR zone creation to fail

    When multiple PTR records are created at the same time, Neutron may try
    to create the same zone multiple times, resulting in a conflict. Ignore
    the resulting error instead and try to create the record anyway.

Change-Id: I59b0f99463ab00743f19016a890561916df900ed
Closes-Bug: 1891309

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-05: Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/789659

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-05: Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/789660

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-05: Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/789661

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-05: Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/789662

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-05: Fix proposed to neutron (stable/stein)

#10

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/789663

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-05: Fix proposed to neutron (stable/rocky)

#11

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/789664

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-05: Fix proposed to neutron (stable/queens)

#12

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/789665

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-05: Related fix proposed to neutron-tempest-plugin (master)

#13

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/789709

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-06: Fix merged to neutron (stable/train)

#14

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789662
Committed: https://opendev.org/openstack/neutron/commit/12da41a48fc3cd884eb4af5c549c8c2bc9a0846a
Submitter: "Zuul (22348)"
Branch: stable/train

commit 12da41a48fc3cd884eb4af5c549c8c2bc9a0846a
Author: Jens Harbott <email address hidden>
Date: Wed Aug 19 14:39:35 2020 +0200

designate: allow PTR zone creation to fail

    Change-Id: I59b0f99463ab00743f19016a890561916df900ed
    Closes-Bug: 1891309
    (cherry picked from commit ca1c6fd69bda5f392712a8a966bf4f9b85ecb57a)

tags:

added: in-stable-train

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-06: Fix merged to neutron (stable/victoria)

#15

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789660
Committed: https://opendev.org/openstack/neutron/commit/b34d23762e236b7e6a9b85025da62503b76463a1
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit b34d23762e236b7e6a9b85025da62503b76463a1
Author: Jens Harbott <email address hidden>
Date: Wed Aug 19 14:39:35 2020 +0200

designate: allow PTR zone creation to fail

    Change-Id: I59b0f99463ab00743f19016a890561916df900ed
    Closes-Bug: 1891309
    (cherry picked from commit ca1c6fd69bda5f392712a8a966bf4f9b85ecb57a)

tags:

added: in-stable-victoria

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-07: Fix merged to neutron (stable/ussuri)

#16

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789661
Committed: https://opendev.org/openstack/neutron/commit/bfb42dfa02e0da488cfff1dff4fc7449b95619b9
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit bfb42dfa02e0da488cfff1dff4fc7449b95619b9
Author: Jens Harbott <email address hidden>
Date: Wed Aug 19 14:39:35 2020 +0200

designate: allow PTR zone creation to fail

    Change-Id: I59b0f99463ab00743f19016a890561916df900ed
    Closes-Bug: 1891309
    (cherry picked from commit ca1c6fd69bda5f392712a8a966bf4f9b85ecb57a)

tags:

added: in-stable-ussuri

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-10: Fix included in openstack/neutron 15.3.4

#17

This issue was fixed in the openstack/neutron 15.3.4 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-10: Fix included in openstack/neutron 16.3.2

#18

This issue was fixed in the openstack/neutron 16.3.2 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-10: Fix included in openstack/neutron 17.1.2

#19

This issue was fixed in the openstack/neutron 17.1.2 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-14: Fix merged to neutron (stable/wallaby)

#20

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789659
Committed: https://opendev.org/openstack/neutron/commit/87ad1bc89b9f16c5932e5a657db21bb8b1441e38
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 87ad1bc89b9f16c5932e5a657db21bb8b1441e38
Author: Jens Harbott <email address hidden>
Date: Wed Aug 19 14:39:35 2020 +0200

designate: allow PTR zone creation to fail

    Change-Id: I59b0f99463ab00743f19016a890561916df900ed
    Closes-Bug: 1891309
    (cherry picked from commit ca1c6fd69bda5f392712a8a966bf4f9b85ecb57a)

tags:

added: in-stable-wallaby

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-01: Fix merged to neutron (stable/stein)

#21

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789663
Committed: https://opendev.org/openstack/neutron/commit/3416706389970d7ff4e03297d3e422313eaaad68
Submitter: "Zuul (22348)"
Branch: stable/stein

commit 3416706389970d7ff4e03297d3e422313eaaad68
Author: Jens Harbott <email address hidden>
Date: Wed Aug 19 14:39:35 2020 +0200

designate: allow PTR zone creation to fail

    Change-Id: I59b0f99463ab00743f19016a890561916df900ed
    Closes-Bug: 1891309
    (cherry picked from commit ca1c6fd69bda5f392712a8a966bf4f9b85ecb57a)

tags:

added: in-stable-stein

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-07: Fix merged to neutron (stable/rocky)

#22

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789664
Committed: https://opendev.org/openstack/neutron/commit/8977e2e7a49804b0464df9da13c6838362b0bef1
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit 8977e2e7a49804b0464df9da13c6838362b0bef1
Author: Jens Harbott <email address hidden>
Date: Wed Aug 19 14:39:35 2020 +0200

designate: allow PTR zone creation to fail

    Change-Id: I59b0f99463ab00743f19016a890561916df900ed
    Closes-Bug: 1891309
    (cherry picked from commit ca1c6fd69bda5f392712a8a966bf4f9b85ecb57a)

tags:

added: in-stable-rocky

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-07: Fix merged to neutron (stable/queens)

#23

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789665
Committed: https://opendev.org/openstack/neutron/commit/e9b7c64802dc10906b1a056f764d8328a69ef3ee
Submitter: "Zuul (22348)"
Branch: stable/queens

commit e9b7c64802dc10906b1a056f764d8328a69ef3ee
Author: Jens Harbott <email address hidden>
Date: Wed Aug 19 14:39:35 2020 +0200

designate: allow PTR zone creation to fail

    Change-Id: I59b0f99463ab00743f19016a890561916df900ed
    Closes-Bug: 1891309
    (cherry picked from commit ca1c6fd69bda5f392712a8a966bf4f9b85ecb57a)

tags:

added: in-stable-queens

Bernard Cafarelli (bcafarel) on 2021-06-11

tags:

added: neutron-proactive-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-06-24: Related fix merged to neutron-tempest-plugin (master)

#24

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/789709
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/4a91784c02b7799241af2c01b9964442c65cf7af
Submitter: "Zuul (22348)"
Branch: master

commit 4a91784c02b7799241af2c01b9964442c65cf7af
Author: Slawek Kaplonski <email address hidden>
Date: Wed May 5 08:44:51 2021 +0200

Move back designate scenario job to the check queue

    Due to bug [1] designate integration job was moved to
    the experimental queue some time ago but now as [1] is fixed,
    we can make it again running in the check queue and make it voting.

[1] https://launchpad.net/bugs/1891309

    Depends-On: https://review.opendev.org/c/openstack/neutron/+/789659
    Depends-On: https://review.opendev.org/c/openstack/neutron/+/789660
    Depends-On: https://review.opendev.org/c/openstack/neutron/+/789661
    Depends-On: https://review.opendev.org/c/openstack/neutron/+/789662
    Depends-On: https://review.opendev.org/c/openstack/neutron/+/789663
    Depends-On: https://review.opendev.org/c/openstack/neutron/+/789664
    Depends-On: https://review.opendev.org/c/openstack/neutron/+/789665