FQDN inside guest VM is not the same as dns_assignment on network port

[Walid Moghrabi]

Hi,

Here is what we're trying to achieve :
- create a tenant associated to a provider network (vlan based, no FIP)
- create a dns zone associated to this tenant with Designate (basically, <tenant>.<basedomain>)
- create instances within this tenant which will have recordsets created automatically in the DNS zone in the form <instance>.<tenant>.<basedomain>
- the FQDN inside the VM instance should be what is set in the DNS zone associated to the tenant (so <instance>.<tenant>.<basedomain>)

We're running Charmed Openstack on the yoga/stable channel on Ubuntu 20.04.
We're also using OVN.

Basically, everything that I explained is working and you'll see in the example below that even the instance's network port has the good dns_assignment attribute value with a fqdn in the form <instance>.<tenant>.<basedomain> *but* inside the guest instance, there is no way to get the proper FQDN, neither from "hostname/hostnamectl" nor from the Openstack metadata.
We're always getting the <instance>.<basedomain> formed FQDN.
We don't even see any reference to it inside the metadata, the closest we get is the project_id but not the name.

Here are all the informations I could gathered to describe our issue :
===========================================================================================================
##########################################################################################
### OPENSTACK SIDE
##########################################################################################

ubuntu@testcloud1:~$ openstack project show test-terraform
+-------------+---------------------------------------------------------------------------------------------+
| Field | Value |
+-------------+---------------------------------------------------------------------------------------------+
| description | tenant created by terraform for test-terraform |
| domain_id | a68af68161ee4515a5a3d76095cd4d94 |
| enabled | True |
| id | 5d3f1ee6a825498696ce1a30f34bb119 |
| is_domain | False |
| name | test-terraform |
| options | {} |
| parent_id | a68af68161ee4515a5a3d76095cd4d94 |
| tags | ['createdBy=terraform', '<email address hidden>', 'ownerName=cloud', 'zone=b2b'] |
+-------------+---------------------------------------------------------------------------------------------+

ubuntu@testcloud1:~$ openstack network show 04d4287b-b338-460c-af5c-552043dbace2
+---------------------------+---------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------------+---------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | |
| created_at | 2023-02-09T15:49:39Z |
| description | |
| dns_domain | test-terraform.ops.testcloud.fr. |
| id | 04d4287b-b338-460c-af5c-552043dbace2 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | None |
| is_vlan_transparent | None |
| mtu | 9000 |
| name | test-terraform-default |
| port_security_enabled | True |
| project_id | 5d3f1ee6a825498696ce1a30f34bb119 |
| provider:network_type | vlan |
| provider:physical_network | dcfabric |
| provider:segmentation_id | 1012 |
| qos_policy_id | None |
| revision_number | 3 |
| router:external | Internal |
| segments | None |
| shared | False |
| status | ACTIVE |
| subnets | 50d94a98-b288-4ed6-a301-7f1e2a18227b |
| tags | aciEpg, createdBy=terraform, default=true, <email address hidden>, ownerName=cloud, zone=b2b |
| updated_at | 2023-02-09T15:49:46Z |
+---------------------------+---------------------------------------------------------------------------------------------------------+

ubuntu@testcloud1:~$ openstack subnet show 50d94a98-b288-4ed6-a301-7f1e2a18227b
+----------------------+---------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+---------------------------------------------------------------------------------------------------------+
| allocation_pools | 10.27.192.194-10.27.192.254 |
| cidr | 10.27.192.0/19 |
| created_at | 2023-02-09T15:49:46Z |
| description | |
| dns_nameservers | 10.53.96.31, 10.53.96.32 |
| dns_publish_fixed_ip | None |
| enable_dhcp | True |
| gateway_ip | 10.27.192.1 |
| host_routes | |
| id | 50d94a98-b288-4ed6-a301-7f1e2a18227b |
| ip_version | 4 |
| ipv6_address_mode | None |
| ipv6_ra_mode | None |
| name | test-terraform-28 |
| network_id | 04d4287b-b338-460c-af5c-552043dbace2 |
| project_id | 5d3f1ee6a825498696ce1a30f34bb119 |
| revision_number | 1 |
| segment_id | None |
| service_types | |
| subnetpool_id | None |
| tags | aciEpg, createdBy=terraform, default=true, <email address hidden>, ownerName=cloud, zone=b2b |
| updated_at | 2023-02-09T15:49:52Z |
+----------------------+---------------------------------------------------------------------------------------------------------+

ubuntu@testcloud1:~$ openstack zone show 10d2152f-3b49-4f29-8027-b0c7e79a497a --all-projects
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| action | NONE |
| attributes | |
| created_at | 2023-02-09T15:49:39.000000 |
| description | default tenant zone |
| email | <email address hidden> |
| id | 10d2152f-3b49-4f29-8027-b0c7e79a497a |
| masters | |
| name | test-terraform.ops.testcloud.fr. |
| pool_id | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id | 5d3f1ee6a825498696ce1a30f34bb119 |
| serial | 1679402949 |
| status | ACTIVE |
| transferred_at | 2023-03-21T12:49:31.000000 |
| ttl | 3600 |
| type | PRIMARY |
| updated_at | 2023-03-21T12:50:14.000000 |
| version | 1229096 |
+----------------+--------------------------------------+

ubuntu@testcloud1:~$ openstack recordset list 10d2152f-3b49-4f29-8027-b0c7e79a497a --all-projects
+--------------------------------------+----------------------------------+-------------------------------------------------+------+---------------------------------------------------+--------+--------+
| id | project_id | name | type | records | status | action |
+--------------------------------------+----------------------------------+-------------------------------------------------+------+---------------------------------------------------+--------+--------+
| da21623f-f392-4d3d-8d1f-c78388fad31e | 5d3f1ee6a825498696ce1a30f34bb119 | test-terraform.ops.testcloud.fr. | SOA | ns2.designate.testcloud.fr. | ACTIVE | NONE |
| | | | | support.testcloud.fr. 1679403279 3547 600 86400 | | |
| | | | | 3600 | | |
| f85010e8-ceac-479d-b08b-7a10e8f5fa82 | 5d3f1ee6a825498696ce1a30f34bb119 | test-terraform.ops.testcloud.fr. | NS | ns3.designate.testcloud.fr. | ACTIVE | NONE |
| | | | | ns2.designate.testcloud.fr. | | |
| | | | | ns1.designate.testcloud.fr. | | |
| f37ab515-da5c-49dd-a44e-b4667ab72532 | 5d3f1ee6a825498696ce1a30f34bb119 | toto.test-terraform.ops.testcloud.fr. | A | 10.27.192.218 | ACTIVE | NONE |
| aebd9079-e88f-4a7b-b2a9-682f84e5b63e | 5d3f1ee6a825498696ce1a30f34bb119 | test2-cm-az1.test-terraform.ops.testcloud.fr. | A | 10.27.192.230 | ACTIVE | NONE |
| d92d84b6-47f8-4178-a1ad-db54031aab62 | 5d3f1ee6a825498696ce1a30f34bb119 | test1-cm-az1.test-terraform.ops.testcloud.fr. | A | 10.27.192.197 | ACTIVE | NONE |
| bfe15a2a-031b-4b6b-9487-af689ff25bf2 | 5d3f1ee6a825498696ce1a30f34bb119 | test-volumes.test-terraform.ops.testcloud.fr. | A | 10.27.192.228 | ACTIVE | NONE |
+--------------------------------------+----------------------------------+-------------------------------------------------+------+---------------------------------------------------+--------+--------+

ubuntu@testcloud1:~$ openstack port show d4b429b1-f4a8-4b28-a8dd-97a4e7650dc2
+-------------------------+-------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+-------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | dlpe2v23513.maas.testcloud.fr |
| binding_profile | |
| binding_vif_details | bound_drivers.0='ovn', connectivity='l2', port_filter='True' |
| binding_vif_type | ovs |
| binding_vnic_type | normal |
| created_at | 2023-03-20T16:58:18Z |
| data_plane_status | None |
| description | |
| device_id | b1c38ba2-6543-4d7a-87b3-bd0e78f88302 |
| device_owner | compute:b2b |
| device_profile | None |
| dns_assignment | fqdn='test-volumes.test-terraform.ops.testcloud.fr.', hostname='test-volumes', ip_address='10.27.192.228' |
| dns_domain | |
| dns_name | test-volumes |
| extra_dhcp_opts | |
| fixed_ips | ip_address='10.27.192.228', subnet_id='50d94a98-b288-4ed6-a301-7f1e2a18227b' |
| id | d4b429b1-f4a8-4b28-a8dd-97a4e7650dc2 |
| ip_allocation | immediate |
| mac_address | fa:16:3e:00:df:c1 |
| name | |
| network_id | 04d4287b-b338-460c-af5c-552043dbace2 |
| numa_affinity_policy | None |
| port_security_enabled | True |
| project_id | 5d3f1ee6a825498696ce1a30f34bb119 |
| propagate_uplink_status | None |
| qos_network_policy_id | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 5 |
| security_group_ids | e364d92f-7f88-46b1-bd9a-8d2fe5753c59 |
| status | ACTIVE |
| tags | |
| trunk_details | None |
| updated_at | 2023-03-20T16:58:25Z |
+-------------------------+-------------------------------------------------------------------------------------------------------------+

##########################################################################################
### GUEST INSTANCE SIDE
##########################################################################################
[root@test-volumes ~]# hostname -f
test-volumes

[root@test-volumes ~]# hostnamectl status
 Static hostname: test-volumes
       Icon name: computer-vm
         Chassis: vm
      Machine ID: b1c38ba265434d7a87b3bd0e78f88302
         Boot ID: a584aef6e4474f4a9284592a2337c7ed
  Virtualization: kvm
Operating System: Ubuntu 22.04.1 LTS
          Kernel: Linux 5.15.0-48-generic
    Architecture: x86-64
 Hardware Vendor: OpenStack Foundation
  Hardware Model: OpenStack Nova

[root@test-volumes ~]# curl -s http://169.254.169.254/openstack/latest/meta_data.json | jq
{
  "uuid": "b1c38ba2-6543-4d7a-87b3-bd0e78f88302",
  "hostname": "test-volumes.ops.testcloud.fr.",
  "name": "test-volumes",
  "launch_index": 0,
  "availability_zone": "b2b",
  "random_seed": "5+WkDF0IpCad2bvyOxLvHiitSkEaNhOmPGWauZ3se0m1lwVucDUbzHShEc74k7doDxvnqHUX7hGtU9qVLFWqqXkt+N2JSIkqgYBVxIcyhZh8p4cZW2YYZDpO06SxlMC+7xdwmEnkO9bba1kgOknJTUMxIdICWjcITG6Xa7V/kgWi0Jc/eKsDJ/T3UOH+G+DM1WKLf7gjk2mkHCFqUXq949wPRv1aKRk9ZqNHLZqBT5VxPxwqGYSwYhNHRghhyC7pTlxs8u2jqCgZE6v8ZTQ+oh40F/a8KhAo+v67qeJ0rFGsC/Q+ijnD7H/wnJUwjXxGJ3Srf7Ajl1gIwWVMnBwGOO82ubll0ite734v+4Onlyf3KWvqNtotCqqsnAB6oB+nJMmn5QmjJZW7+lq8tRoC+8mwMbnIw/77gx6tf0SvrypAapiOF4rm+vGX56N3XiJStrdsMg4zYYcFgtgSDYHw9kA01Hz2pUGnlbXGv7q0ZfC3yd2vYAXZH6Y1HAtk1x0VyqtSypJ4PH039so3GV2vwQOexsLmGdu/IDfiAsGQEbD39RHRDVQIBKxUtbc3XrJHRZERePIGzA9S3jSydJrU4Nbj2+leU2WQlzelRnuu/8usiCNrR36R+DOc3e9yFh+Unx4/kXLqc8M5o49npdDdVPjvraMwB3fH6t/7DJM/M0E=",
  "project_id": "5d3f1ee6a825498696ce1a30f34bb119",
  "devices": [],
  "dedicated_cpus": []
}

[root@test-volumes ~]# curl -s http://169.254.169.254/openstack/latest/network_data.json | jq
{
  "links": [
    {
      "id": "tapd4b429b1-f4",
      "vif_id": "d4b429b1-f4a8-4b28-a8dd-97a4e7650dc2",
      "type": "ovs",
      "mtu": 9000,
      "ethernet_mac_address": "fa:16:3e:00:df:c1"
    }
  ],
  "networks": [
    {
      "id": "network0",
      "type": "ipv4_dhcp",
      "link": "tapd4b429b1-f4",
      "network_id": "04d4287b-b338-460c-af5c-552043dbace2"
    }
  ],
  "services": [
    {
      "type": "dns",
      "address": "10.53.96.31"
    },
    {
      "type": "dns",
      "address": "10.53.96.32"
    }
  ]
}

[Walid Moghrabi (walid-fdj)]

Might be easier to read in a file ...

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Walid:

According to the information provided, Neutron is sending the correct information to Nova. The server host name should be retrieved from port.dns_assignment.fqdn, as in [1]. Actually this Nova patch was merged in Yoga. If the metadata retrieve doesn't have this information, you should ask this question to Nova folks. Before that, please check you have [1] in your deployment.

Regards.

[1]https://review.opendev.org/c/openstack/nova/+/778549/7/nova/api/openstack/compute/schemas/servers.py#b362

[Balazs Gibizer (balazs-gibizer)]

I think this is related to
* https://specs.openstack.org/openstack/nova-specs/specs/xena/implemented/configurable-instance-hostnames.html
* https://specs.openstack.org/openstack/nova-specs/specs/2023.1/implemented/fqdn-in-hostname.html

Could you check if the above docs help?

[Dr. Jens Harbott (j-harbott)]

It seems the server was created with terraform, can you share the definition that was used? Can you check whether the issue also exists when using the "openstack server create" CLI command? Please also attach the output of "openstack server show" in both cases.

[Walid Moghrabi (walid-fdj)]

Yes, it was created with Terraform however, creating a server through the CLI or Horizon is even worst.
The fqdn set at the port level only account for the cluster global domain ("test.ops.testcloud.fr") and not the tenant based fqdn I expect ("test.tenant.ops.testcloud.fr").
Because of this, reverse DNS is not working and things like "sudo" or "hostname -f" are very slow because those commands wait until the reverse times out.

"hostname -A" works as expected, it provides the FQDN with no delay so I don't know exactly where the difference is.
However, in the case of Terraform, I can "forge" the FQDN at the port level and thus, I retrieve the good FQDN inside the instance through "hostname -A" but this is a trick and it doesn't work if I spawn an instance through CLI/Horizon though.

It looks like it just d'ont care about Designate and the zone assigned to the tenant and only care about the global domain which is cluster wide which is definitely not what I want.

[Felipe Alencastro (falencastro)]

Nova metadata gets the domain from the dhcp_domain parameter in nova.conf, on charmed openstack the value of juju config neutron-api dns-domain, gets passed to dhcp_domain in nova.conf.

Here's the code on yoga which generates the hostname for the nova metadata:

    def _get_hostname(self):
        # TODO(stephenfin): At some point in the future, we may wish to
        # retrieve this information from neutron.
        if CONF.api.dhcp_domain:
            return '.'.join([self.instance.hostname, CONF.api.dhcp_domain])

        return self.instance.hostname

Maybe that's a RFE to make nova metadata receive the hostname from the neutron dns_assignment attribute.

[Felipe Alencastro (falencastro)]

We see the same behavior on our clouds (charmed yoga) with neutron+dns integration[1] set up, dhcp_domain in nova is set to the charm default-value of openstack.example.

[felipe_alencastro@alencastro ~]$ openstack network create --dns-domain test.example.org. example-net

[felipe_alencastro@alencastro ~]$ openstack network show example-net -c dns_domain
+------------+-------------------+
| Field | Value |
+------------+-------------------+
| dns_domain | test.example.org. |
+------------+-------------------+

[felipe_alencastro@alencastro ~]$ openstack server create --flavor m1.xsmall --image rocky8 --network example-net --security-group test --key-name test-key rocky8ci

[rocky@rocky8ci ~]$ hostnamectl
   Static hostname: rocky8ci.openstack.example
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 00d5403cf95c4216935ffbcbab997f6d
           Boot ID: 7963f8a0e98c40788e518383a77e0417
    Virtualization: kvm
  Operating System: Rocky Linux 8.8 (Green Obsidian)
       CPE OS Name: cpe:/o:rocky:rocky:8:GA
            Kernel: Linux 4.18.0-477.27.1.el8_8.x86_64
      Architecture: x86-64

[rocky@rocky8ci ~]$ hostname -A <-------------- hostname -A does a PTR query and receives the proper fqdn from designate resolvers.
rocky8ci.test.example.org.

[rocky@rocky8ci ~]$ curl -s http://169.254.169.254/openstack/latest/meta_data.json |jq -r .hostname
rocky8ci.openstack.example.

[1] https://docs.openstack.org/designate/latest/user/neutron-integration.html