Documentation for dns integration needs improvement

[Albert Mikaelyan]

The documentation of neutron dns integration needs some improvement in order to avoid common pitfall during deployment.

[ORIGINAL DESCRIPTION]
The problem:

Upon instance/port deletion the following error is received and the instance/port comes into ERROR state. The instance/port is deleted successfully after second retry:

2017-10-09 12:46:52.555 39624 ERROR neutron.callbacks.manager [req-70d6ae09-694a-4ba7-8189-f99159e71fc0 bc39ed40eefa4bd39e91ef35c5e48772 9e1b0975ef23425d9f519ff1b97cdef1 - - -] Callback neutron.plugins.ml2.extensions.dns_integration._delete_port_in_external_dns_service--9223363296916797971 raised Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-f5476d34-df91-41e8-be95-b481dc6d68f0)
2017-10-09 12:46:52.605 39617 INFO neutron.wsgi [req-4deac1db-6401-43ff-a7c9-ef7e26b3a24d 2cccfff294fc42a397be3c5202401037 5cc5d6cd841d4662b809cb883f4a0a8a - - -] 10.255.3.3 - - [09/Oct/2017 12:46:52] "GET /v2.0/ports.json?network_id=7e666b30-14d6-492c-893b-85cffa6a8e9f&device_owner=network%3Adhcp HTTP/1.1" 200 2437 0.071344
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource [req-70d6ae09-694a-4ba7-8189-f99159e71fc0 bc39ed40eefa4bd39e91ef35c5e48772 9e1b0975ef23425d9f519ff1b97cdef1 - - -] delete failed: No details.
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource Traceback (most recent call last):
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 93, in resource
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource result = method(request=request, **args)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/api/v2/base.py", line 562, in delete
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource return self._delete(request, id, **kwargs)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/db/api.py", line 95, in wrapped
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource self.force_reraise()
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource self.force_reraise()
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/db/api.py", line 131, in wrapped
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource traceback.format_exc())
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource self.force_reraise()
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/api/v2/base.py", line 584, in _delete
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource obj_deleter(request.context, id, **kwargs)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/common/utils.py", line 775, in inner
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource return f(self, context, *args, **kwargs)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/db/api.py", line 166, in wrapped
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource return method(*args, **kwargs)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/db/api.py", line 95, in wrapped
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource self.force_reraise()
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource self.force_reraise()
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/db/api.py", line 131, in wrapped
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource traceback.format_exc())
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource self.force_reraise()
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1507, in delete_port
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource self._pre_delete_port(context, id, l3_port_check)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource File "/openstack/venvs/neutron-15.1.9/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1501, in _pre_delete_port
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource raise e.errors[0].error
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-f5476d34-df91-41e8-be95-b481dc6d68f0)
2017-10-09 12:46:52.611 39624 ERROR neutron.api.v2.resource

My configuration:

I must say that although I'm still on Ocata I have followed the Pike guide, as Ocata guide is missing the needed configuration for keystone v3 (!). Although, Ocata is fully supporting the keystone v3 endpoint in [designate] section according to release notes.

[designate]

url = https://external-domain:9001/v2
auth_uri = https://external-domain:5000

admin_auth_url = http://internal-ip:35357/v3
admin_username = neutron
admin_password = pass
admin_tenant_name = service

region_name = RegionOne
domain_name = Default
project_domain_name = Default
user_domain_name = Default
project_name = service
username = neutron
password = pass

allow_reverse_dns_lookup = True
ipv4_ptr_zone_prefix_size = 24
ipv6_ptr_zone_prefix_size = 116
insecure = True

What I've tried:
1. Using designate credentials instead of neutron
2. Adding the 'domain_name' configuration
3. removing admin_* configurations - probably bad idea, but tried anyway.

How to reproduce:
1. Configure neutron to use integrate with designate as in the guide: https://docs.openstack.org/neutron/pike/admin/config-dns-int.html
2. Create instance or a port. Port is easier and faster:
openstack port create myport --dns-name myport --network selfservice1
3. The above error would be seen in neutron-server.log, however the command will finish successfully and dns and port would be created as they should.
4. Now try deleting the port:
openstack port delete myport
5. The above error would be seen again, but on delete method, and the command will *fail*.
6. Try deleting the port again. The port should be deleted successfully.

[Dr. Jens Harbott (j-harbott)]

You need so set "auth_type=password" in order to activate new style credentials in the [designate] section, and also "auth_url = http://internal-ip:35357/v3".

Still I agree that the deletion should always succeed despite the designate error.

[Miguel Lavalle (minsel)]

Two questions:

1) What is the value of the 'dns_domain' attribute in network 'selfservice1'?
2) Assuming that network has a valid value in attribute 'dns_domain', has the corresponding zone been created in Designate, before creating and deleting the port?

[Albert Mikaelyan (tahvok)]

It is working now!
Setting auth_type and auth_url as well as pointing the authentication to admin urls made it work!
Here's my final configuration:

[designate]

url = https://external-domain:9001/v2

admin_auth_url = http://internal-ip:35357/v3
admin_username = designate
admin_password = pass
admin_tenant_name = service

auth_type = password
auth_uri = http://internal-ip:35357
auth_url = http://internal-ip:35357/v3
region_name = RegionOne
project_domain_name = Default
user_domain_name = Default
project_name = service
username = designate
password = pass

allow_reverse_dns_lookup = True
ipv4_ptr_zone_prefix_size = 24
ipv6_ptr_zone_prefix_size = 116
insecure = True

Few notes:
1. It is not documented anywhere at all. I saw a bug open that some documentation is needed, so my bug could be used as what configurations are needed.
2. You might notice that I'm using 'designate' user instead of 'neutron' - this does not make any difference as both users are admins in my case, but I think it is the correct way, as neutron needs to access designate - so it should use designate credentials instead of neutron credentials - so I think this is documented wrongly as well.

One question:
Do I need to use the admin* configurations? What is their purpose? I saw in the code that their are omitted if auth_type is provided: https://github.com/openstack/neutron/blob/b70d83cf4ec8bc583d2db2546b175a072507a3f3/neutron/services/externaldns/drivers/designate/driver.py#L50

So do we still need these? Because the documentation example is showing that both should be used.

[Dr. Jens Harbott (j-harbott)]

No, the admin_* parameters are not needed anymore. I'll draft a docs update.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/541712

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/541712
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f305559292e6ecfd35740268e69b10cf99089fb2
Submitter: Zuul
Branch: master

commit f305559292e6ecfd35740268e69b10cf99089fb2
Author: Jens Harbott <email address hidden>
Date: Wed Feb 7 12:56:12 2018 +0000

    Update documentation for DNS integration

    - Split documentation for external DNS integration into a new document
    - Update configs to current standards
    - Remove use of old designate client

    Change-Id: I7a50ad72e35e2c01f874b872ddeff1aa8bfe3424
    Closes-Bug: 1722367
    Related-Bug: 1725630

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/546038

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546038
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=abb60c6175af435964028ce7c97bb4803aeab004
Submitter: Zuul
Branch: stable/queens

commit abb60c6175af435964028ce7c97bb4803aeab004
Author: Jens Harbott <email address hidden>
Date: Wed Feb 7 12:56:12 2018 +0000

    Update documentation for DNS integration

    - Split documentation for external DNS integration into a new document
    - Update configs to current standards
    - Remove use of old designate client

    Change-Id: I7a50ad72e35e2c01f874b872ddeff1aa8bfe3424
    Closes-Bug: 1722367
    Related-Bug: 1725630
    (cherry picked from commit f305559292e6ecfd35740268e69b10cf99089fb2)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0b1 development milestone.