Inproper configuration of DNS in kolla-ansible

[Michal Arbet]

Hi,

In kolla-ansible we are configuring DNS in bad way, we are not settings dns_domain in neutron which is REQUIRED for both internal/external DNS configuration ...

I suggest adding a user switch that will tell whether DNS is turned on or not + precheck that will check the dns_domain setup for neutron ..and write tests for designate..

Check logs below :

BUGGY >>>>>

root@controller0:~# cat /etc/kolla/neutron-server/neutron.conf | egrep 'dns_domain|external_dns_driver'
external_dns_driver = designate
dns_domain = openstacklocal

root@deploy:/home/kolla# openstack network show d2fa31a9-5fb4-49a2-8504-898a5d30d9c8 -f json -c name -c dns_domain -c id
{
  "dns_domain": "public.bobcat.ultimum.cloud.",
  "id": "d2fa31a9-5fb4-49a2-8504-898a5d30d9c8",
  "name": "public-network-1"
}

root@deploy:/home/kolla# openstack subnet show 128aaa74-4aab-4277-b23a-d15903ce35ab -f json -c id -c name -c network_id -c dns_publish_fixed_ip
{
  "dns_publish_fixed_ip": true,
  "id": "128aaa74-4aab-4277-b23a-d15903ce35ab",
  "name": "public-subnet-1",
  "network_id": "d2fa31a9-5fb4-49a2-8504-898a5d30d9c8"
}

(kolla-ansible)[root]# openstack port list --network d2fa31a9-5fb4-49a2-8504-898a5d30d9c8 | egrep -v '185.21.196.205|185.21.196.222' # Except dhcp ports
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+
| b024c8c8-f8a0-4697-817f-da91adf0a8a9 | | fa:16:3e:3f:35:07 | ip_address='185.21.196.120', subnet_id='128aaa74-4aab-4277-b23a-d15903ce35ab' | ACTIVE |
| fa86d086-b90d-459d-af9a-7d46b2a92488 | | fa:16:3e:a9:09:b8 | ip_address='185.21.196.66', subnet_id='128aaa74-4aab-4277-b23a-d15903ce35ab' | ACTIVE |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+

DNSmasq - fqdns created with openstacklocal. which is default and kolla don't set

(neutron-dhcp-agent)[root@controller0 /]# cat /var/lib/neutron/kolla/dhcp/d2fa31a9-5fb4-49a2-8504-898a5d30d9c8/host
fa:16:3e:3f:35:07,host-185-21-196-120.openstacklocal.,185.21.196.120
fa:16:3e:a9:09:b8,host-185-21-196-66.openstacklocal.,185.21.196.66

root@deploy:/home/kolla# openstack zone list
+--------------------------------------+------------------------------+---------+------------+--------+--------+
| id | name | type | serial | status | action |
+--------------------------------------+------------------------------+---------+------------+--------+--------+
| 778823c7-3a76-42fd-ab4e-acbf68df9499 | public.bobcat.ultimum.cloud. | PRIMARY | 1705401712 | ACTIVE | NONE |
| 4fd09ee4-104f-46a0-a345-48fb36036005 | test.bobcat.ultimum.cloud. | PRIMARY | 1705340498 | ACTIVE | NONE |
+--------------------------------------+------------------------------+---------+------------+--------+--------+

Records in zone :

root@deploy:/home/kolla# openstack recordset list 778823c7-3a76-42fd-ab4e-acbf68df9499
+--------------------------------------+------------------------------+------+----------------------------------------------------------------------------------+--------+--------+
| id | name | type | records | status | action |
+--------------------------------------+------------------------------+------+----------------------------------------------------------------------------------+--------+--------+
| f4e03a03-f6a0-497e-8182-e5f1268f5e0a | public.bobcat.ultimum.cloud. | SOA | ns.bobcat.ultimum.cloud. michal.arbet.ultimum.io. 1705401712 3502 600 86400 3600 | ACTIVE | NONE |
| cf879a02-13a7-406c-ad4e-6dfa542ed49b | public.bobcat.ultimum.cloud. | NS | ns.bobcat.ultimum.cloud. | ACTIVE | NONE |
+--------------------------------------+------------------------------+------+----------------------------------------------------------------------------------+--------+--------+

Create server (DNS record creation should be automatic)

root@deploy:/home/kolla# openstack server create --flavor 1 --image 3f738cfd-7b51-4fad-b0fd-2feebe0b4788 --network d2fa31a9-5fb4-49a2-8504-898a5d30d9c8 kevko-test
+-----------------------------+-----------------------------------------------+
| Field | Value |
+-----------------------------+-----------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | 8H7S9d6Qtau6 |
| config_drive | |
| created | 2024-01-16T10:56:04Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | 81ab249b-a151-43b6-a294-4b68c759acf8 |
| image | cirros (3f738cfd-7b51-4fad-b0fd-2feebe0b4788) |
| key_name | None |
| name | kevko-test |
| progress | 0 |
| project_id | df3687b4123b46ac91e925f900b941e8 |
| properties | |
| security_groups | name='default' |
| status | BUILD |
| updated | 2024-01-16T10:56:05Z |
| user_id | 3f0eb0e10d3a4420a2d3b6c430785862 |
| volumes_attached | |
+-----------------------------+-----------------------------------------------+

DNS record not created :

root@deploy:/home/kolla# openstack recordset list 778823c7-3a76-42fd-ab4e-acbf68df9499
+--------------------------------------+------------------------------+------+----------------------------------------------------------------------------------+--------+--------+
| id | name | type | records | status | action |
+--------------------------------------+------------------------------+------+----------------------------------------------------------------------------------+--------+--------+
| f4e03a03-f6a0-497e-8182-e5f1268f5e0a | public.bobcat.ultimum.cloud. | SOA | ns.bobcat.ultimum.cloud. michal.arbet.ultimum.io. 1705401712 3502 600 86400 3600 | ACTIVE | NONE |
| cf879a02-13a7-406c-ad4e-6dfa542ed49b | public.bobcat.ultimum.cloud. | NS | ns.bobcat.ultimum.cloud. | ACTIVE | NONE |
+--------------------------------------+------------------------------+------+----------------------------------------------------------------------------------+--------+--------+

DNSmasq :

(neutron-dhcp-agent)[root@controller0 /]# cat /var/lib/neutron/kolla/dhcp/d2fa31a9-5fb4-49a2-8504-898a5d30d9c8/host
fa:16:3e:3f:35:07,host-185-21-196-120.openstacklocal.,185.21.196.120
fa:16:3e:a9:09:b8,host-185-21-196-66.openstacklocal.,185.21.196.66
fa:16:3e:c1:77:3f,host-185-21-196-88.openstacklocal.,185.21.196.88

FIXED >>>>

So, designate integration not working in kolla out of the box. As per designate documentation there is need to sed dns_domain in neutron to designate integration start to WORK !

Let's reconfigure dns_domain in neutron services because it need to be different as it is in default

kolla-ansible -i /etc/kolla/inventory reconfigure -t neutron --diff
.
.
.
.
--- before: /etc/kolla/neutron-server/neutron.conf
+++ after: /root/.ansible/tmp/ansible-local-63476ldgpp6da/tmp7_7wj9ha/source
@@ -18,7 +18,7 @@
 transport_url = rabbit://openstack:REPLACED@192.168.205.10:5672,openstack:REPLACED@192.168.205.11:5672,openstack:REPLACED@192.168.205.12:5672//
 external_dns_driver = designate
 ipam_driver = internal
-dns_domain = openstacklocal
+dns_domain = openstack.local.

Soooo , DNSmasq regenerated :

(neutron-dhcp-agent)[root@controller0 /]# cat /var/lib/neutron/kolla/dhcp/d2fa31a9-5fb4-49a2-8504-898a5d30d9c8/host
fa:16:3e:3f:35:07,host-185-21-196-120.openstack.local.,185.21.196.120
fa:16:3e:c1:77:3f,host-185-21-196-88.openstack.local.,185.21.196.88
fa:16:3e:a9:09:b8,host-185-21-196-66.openstack.local.,185.21.196.66

Let's remove old server we created before as this setup didn't work:

openstack server delete 81ab249b-a151-43b6-a294-4b68c759acf8

And create new one when it's finally working as it could :

openstack server create --flavor 1 --image 3f738cfd-7b51-4fad-b0fd-2feebe0b4788 --network d2fa31a9-5fb4-49a2-8504-898a5d30d9c8 kevko-test-dns-fixed

(kolla-ansible)[root]# openstack server create --flavor 1 --image 3f738cfd-7b51-4fad-b0fd-2feebe0b4788 --network d2fa31a9-5fb4-49a2-8504-898a5d30d9c8 kevko-test-dns-fixed
+-----------------------------+-----------------------------------------------+
| Field | Value |
+-----------------------------+-----------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | yDCYD3Ggqge6 |
| config_drive | |
| created | 2024-01-16T11:08:10Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | 778490cf-8c2f-42ed-83cd-812fbe4349f2 |
| image | cirros (3f738cfd-7b51-4fad-b0fd-2feebe0b4788) |
| key_name | None |
| name | kevko-test-dns-fixed |
| progress | 0 |
| project_id | df3687b4123b46ac91e925f900b941e8 |
| properties | |
| security_groups | name='default' |
| status | BUILD |
| updated | 2024-01-16T11:08:10Z |
| user_id | 3f0eb0e10d3a4420a2d3b6c430785862 |
| volumes_attached | |
+-----------------------------+-----------------------------------------------+

(kolla-ansible)[root]# openstack recordset list 778823c7-3a76-42fd-ab4e-acbf68df9499
+--------------------------------------+---------------------------------------------------+------+----------------------------------------------------------------------------------+--------+--------+
| id | name | type | records | status | action |
+--------------------------------------+---------------------------------------------------+------+----------------------------------------------------------------------------------+--------+--------+
| f4e03a03-f6a0-497e-8182-e5f1268f5e0a | public.bobcat.ultimum.cloud. | SOA | ns.bobcat.ultimum.cloud. michal.arbet.ultimum.io. 1705403302 3502 600 86400 3600 | ACTIVE | NONE |
| cf879a02-13a7-406c-ad4e-6dfa542ed49b | public.bobcat.ultimum.cloud. | NS | ns.bobcat.ultimum.cloud. | ACTIVE | NONE |
| 37a4ff07-0c2f-4477-8464-f4a67b88526b | kevko-test-dns-fixed.public.bobcat.ultimum.cloud. | A | 185.21.196.88 | ACTIVE | NONE |
+--------------------------------------+---------------------------------------------------+------+----------------------------------------------------------------------------------+--------+--------+

DNSmasq also fixed

(neutron-dhcp-agent)[root@controller0 /]# cat /var/lib/neutron/kolla/dhcp/d2fa31a9-5fb4-49a2-8504-898a5d30d9c8/host
fa:16:3e:3f:35:07,host-185-21-196-120.openstack.local.,185.21.196.120
fa:16:3e:a9:09:b8,host-185-21-196-66.openstack.local.,185.21.196.66
fa:16:3e:ed:de:cd,kevko-test-dns-fixed.public.bobcat.ultimum.cloud.,185.21.196.88

External DNS also ok :

root@controller0:~# dig kevko-test-dns-fixed.public.bobcat.ultimum.cloud @ns.public.bobcat.ultimum.cloud

; <<>> DiG 9.18.12-0ubuntu0.22.04.3-Ubuntu <<>> kevko-test-dns-fixed.public.bobcat.ultimum.cloud @ns.public.bobcat.ultimum.cloud
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27840
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 305a750bd4d29d390100000065a66459f3ba2d6610b57c84 (good)
;; QUESTION SECTION:
;kevko-test-dns-fixed.public.bobcat.ultimum.cloud. IN A

;; ANSWER SECTION:
kevko-test-dns-fixed.public.bobcat.ultimum.cloud. 300 IN A 185.21.196.88

;; Query time: 0 msec
;; SERVER: 192.168.205.10#53(ns.public.bobcat.ultimum.cloud) (UDP)
;; WHEN: Tue Jan 16 11:11:21 UTC 2024
;; MSG SIZE rcvd: 121

Everything working like a charm :

(kolla-ansible)[root]# openstack server delete 778490cf-8c2f-42ed-83cd-812fbe4349f2
(kolla-ansible)[root]# openstack recordset list 778823c7-3a76-42fd-ab4e-acbf68df9499
+--------------------------------------+------------------------------+------+----------------------------------------------------------------------------------+--------+--------+
| id | name | type | records | status | action |
+--------------------------------------+------------------------------+------+----------------------------------------------------------------------------------+--------+--------+
| f4e03a03-f6a0-497e-8182-e5f1268f5e0a | public.bobcat.ultimum.cloud. | SOA | ns.bobcat.ultimum.cloud. michal.arbet.ultimum.io. 1705403302 3502 600 86400 3600 | ACTIVE | NONE |
| cf879a02-13a7-406c-ad4e-6dfa542ed49b | public.bobcat.ultimum.cloud. | NS | ns.bobcat.ultimum.cloud. | ACTIVE | NONE |
+--------------------------------------+------------------------------+------+----------------------------------------------------------------------------------+--------+--------+

[Michal Arbet (michalarbet)]

Designate documentation where it is specified that it's requirement for both internal/external DNS setup

https://docs.openstack.org/neutron/latest/admin/config-dns-int.html

INTERNAL :

"Edit the /etc/neutron/neutron.conf file and assign a value different to openstacklocal (its default value) to the dns_domain parameter in the [default] section. As an example:
"

EXTERNAL :

The enablement of this functionality is prerequisite for the enablement of the Networking service integration with an external DNS service, which is described in detail in DNS Integration with an External Service.

[Sven Kieske (s-kieske)]

the culprit seems to be this code:

https://github.com/openstack/kolla-ansible/blob/77c18fa615cc592976ae65a52c0198a14e054876/ansible/roles/neutron/templates/neutron.conf.j2#L65

```
{% if enable_designate | bool %}
{% if designate_ns_record is string %}
dns_domain = {{ designate_ns_record }}.
{% endif %}
external_dns_driver = designate
```

so we only set the dns_domain is designate_ns_record is a string, which I understand is currently a tangled up mess, because we are using a single variable for multiple things, that is "designate_ns_record" is used as a string for dns_domain but as a list(?) for other purposes.

So I guess this is, strictly speaking, a bug, because at least the code should look like something like this (made up example):

```

{% if enable_designate | bool %}
dns_domain = {{ new_made_up_variable_containing_a_single_valid_domain }}.
external_dns_driver = designate
```

[Michal Arbet (michalarbet)]

Yes ! Of course ! However, in the event of a repair, I would modify the concept a little so that it makes more sense and fix the documentation and write tests.

Btw, designate_ns_records was changed by Michal Nasiadka https://review.opendev.org/c/openstack/kolla-ansible/+/802304 and this is wrong , because this commit effectively turned off designate integration as dns_domain is not filled and therefore REQUIREMENT is not fullfilled.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/905502
Committed: https://opendev.org/openstack/kolla-ansible/commit/66c4f72c509e3746f090d8b48888a9c42d37705b
Submitter: "Zuul (22348)"
Branch: master

commit 66c4f72c509e3746f090d8b48888a9c42d37705b
Author: Michal Arbet <email address hidden>
Date: Fri Jan 12 18:05:38 2024 +0100

    Enable instance usage audit only when ceilometer is enabled

    This patch disables periodic compute.instance.exists
    notifications when designate is enabled.

    Related-Bug: #2049503
    Change-Id: I39fe2db9182de23c1df814d911eec15e86317702

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/906897

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/906898

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/906899

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/906900

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/907221

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Dr. Jens Harbott <email address hidden>" on branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/906900
Reason: please repropose against unmaintained/yoga when that is created

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Dr. Jens Harbott <email address hidden>" on branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/907221
Reason: please repropose against unmaintained/yoga when that is created

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/905852
Committed: https://opendev.org/openstack/kolla-ansible/commit/6f847610b5e31cab912de2c83709e68e8d771125
Submitter: "Zuul (22348)"
Branch: master

commit 6f847610b5e31cab912de2c83709e68e8d771125
Author: Michal Arbet <email address hidden>
Date: Wed Jan 17 12:02:52 2024 +0100

    Fix neutron DNS integration

    This patch basically does a simple thing, on the basis
    of a variable neutron_dns_integration it enables/disables
    DNS integration.

    There is also precheck added which checks whether dns_domain
    in neutron.conf has a non-default value if DNS integration is
    enabled as this is requirement.

    [1] https://docs.openstack.org/neutron/latest/admin/config-dns-int.html
    [2] https://docs.openstack.org/neutron/latest/admin/config-dns-int-ext-serv.html#config-dns-int-ext-serv

    Closes-Bug: #2049503

    Change-Id: I90f0f8dcec6fa0112179f050d96e9d9db5956cf8

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/906899
Committed: https://opendev.org/openstack/kolla-ansible/commit/673eda91a431c41bb69b42050aac946cc04a604d
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 673eda91a431c41bb69b42050aac946cc04a604d
Author: Michal Arbet <email address hidden>
Date: Fri Jan 12 18:05:38 2024 +0100

    Enable instance usage audit only when ceilometer is enabled

    This patch disables periodic compute.instance.exists
    notifications when designate is enabled.

    Related-Bug: #2049503
    Change-Id: I39fe2db9182de23c1df814d911eec15e86317702
    (cherry picked from commit 66c4f72c509e3746f090d8b48888a9c42d37705b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/906897
Committed: https://opendev.org/openstack/kolla-ansible/commit/0fb1067e1a165ebada122465d01f17d73af80a73
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 0fb1067e1a165ebada122465d01f17d73af80a73
Author: Michal Arbet <email address hidden>
Date: Fri Jan 12 18:05:38 2024 +0100

    Enable instance usage audit only when ceilometer is enabled

    This patch disables periodic compute.instance.exists
    notifications when designate is enabled.

    Related-Bug: #2049503
    Change-Id: I39fe2db9182de23c1df814d911eec15e86317702
    (cherry picked from commit 66c4f72c509e3746f090d8b48888a9c42d37705b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/906898
Committed: https://opendev.org/openstack/kolla-ansible/commit/967aa7a2942421d03d0b972e6db06bbe93231e00
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 967aa7a2942421d03d0b972e6db06bbe93231e00
Author: Michal Arbet <email address hidden>
Date: Fri Jan 12 18:05:38 2024 +0100

    Enable instance usage audit only when ceilometer is enabled

    This patch disables periodic compute.instance.exists
    notifications when designate is enabled.

    Related-Bug: #2049503
    Change-Id: I39fe2db9182de23c1df814d911eec15e86317702
    (cherry picked from commit 66c4f72c509e3746f090d8b48888a9c42d37705b)