Nova metadata service returns wrong hostname after neutron-api dns-domain config is set

Bug #1805645 reported by Pedro Guimarães on 2018-11-28
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack neutron-api charm
Medium
David Ames
OpenStack neutron-gateway charm
High
David Ames
OpenStack nova-cloud-controller charm
Medium
David Ames

Bug Description

We are facing an issue to configure metadata service correctly. Neutron API charm's dns-domain was configured for an specific value but nova metadata service continues to return original "NAME_OF_SERVER.novalocal." as hostname.

dns-domain seems to correctly configure DHCP domain name.

curl http://169.254.169.254/openstack/latest/meta_data.json returns:
{"random_seed": "redacted", "type": "ssh", "name": "dnstest"}], "hostname": "tdnsserver.novalocal", "launch_index": 0, "devices": [], "public_keys": {"dnstest": "redacted"}, "project_id": "redacted", "name": "tdnsserver"}

While /var/lib/dhclient/dhclient.leases contains the correct domain name.

However, not all types of Guest OS follow DHCP's definition of domain. RHEL-based guest OS, for example, set their hostnames to the value present on metadata service despite DHCP's original configuration. Although one can work-around this by using user_data flag on "openstack server create" and setting "preserve_hostname: true", there are situations where this is not feasible (e.g. when OpenStack is under management of an orchestrator).

We have a request to setup Designate to register all machines that are connected to a self-service network. That deployment is using BGP to advertise that SSN.

On a particular customer, we have the following configuration set:
Neutron-api: https://pastebin.canonical.com/p/vPhfkCbdw6/
Please, check that vni_ranges jump on one value, that was necessary to make the SSN available to Designate, as described on: https://docs.openstack.org/mitaka/networking-guide/config-dns-int.html

Nova-cloud-controller: https://pastebin.canonical.com/p/yThtgY95BT/
Neutron-Gateway: https://pastebin.canonical.com/p/q97hqfgMCP/
Neutron-Openvswitch: https://pastebin.canonical.com/p/MPJ3zKSYxK/

Pedro Guimarães (pguimaraes) wrote :

Marking as Field Critical as I'm facing a deployment on a customer that is exactly on the Guest OS/orchestrator situation described above.

James Page (james-page) wrote :

This is the code that generates the hostname in the nova metadata:

    def _get_hostname(self):
        return "%s%s%s" % (self.instance.hostname,
                           '.' if CONF.dhcp_domain else '',
                           CONF.dhcp_domain)

This uses a deprecated dhcp_domain option that was part of nova-network.

James Page (james-page) wrote :

You might try setting this option via config-flags to match the dns-domain value in neutron-api - that would at least make the metadata consistent with the dns domain information being created and handed out via dhcp.

description: updated
Ryan Beisner (1chb1n) wrote :

Upstream reference, which appears to be a dup/invalid stance: https://bugs.launchpad.net/nova/+bug/1694417

Ryan Beisner (1chb1n) wrote :

See also, the behavior difference in consuming the data: https://bugs.launchpad.net/cloud-init/+bug/1649802

Changed in charm-neutron-gateway:
importance: Undecided → High
assignee: nobody → David Ames (thedac)
milestone: none → 19.04
David Ames (thedac) on 2018-11-28
Changed in charm-neutron-gateway:
status: New → Triaged
Pedro Guimarães (pguimaraes) wrote :

@james-page
I have:
$ juju config neutron-api config-flags
dhcp_domain=RIGHT_URL.
$ juju config nova-cloud-controller config-flags
dhcp_domain=RIGHT_URL.,dns_domain=RIGHT_URL.

Still receiving *.novalocal from metadata server when executing

curl http://169.254.169.254/openstack/latest/meta_data.json

David Ames (thedac) wrote :

Confirmed setting dhcp_domain on the neutron-gateway's nova.conf allows VMs booted after the configuration change to be able to retrieve the correct domain for its hostname from the nova-metadata-api.

Check metadata hostname with this curl statement:
curl http://169.254.169.254/latest/meta-data/hostname

PR on the way.

Fix proposed to branch: master
Review: https://review.openstack.org/620668

Changed in charm-neutron-gateway:
status: Triaged → In Progress
David Ames (thedac) wrote :

Pedro,

The dhcp_domain needs to be set on the neutron-gateway at Queens because the nova-metadata-api runs there.

Please test the following which will set the domain to the value neutron-api's dns-domain is set to.
cs:~thedac/neutron-gateway-2
https://review.openstack.org/#/c/620668/

This test will show what is configured on the nova-metadata-api:
curl http://169.254.169.254/latest/meta-data/hostname

David Ames (thedac) wrote :

For completeness, I am adding neutron-api and nova-cloud-controller for OpenStack releases >= Rocky where we now run nova-metadata-api on the nova-cloud-controller.

Note: these are lower priority and do not affect the environment which caused the field escalation. This PR solves the escalation: https://review.openstack.org/#/c/620668/

Changed in charm-neutron-api:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → David Ames (thedac)
milestone: none → 19.04
Changed in charm-nova-cloud-controller:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → David Ames (thedac)
milestone: none → 19.04

Fix proposed to branch: master
Review: https://review.openstack.org/620678

Changed in charm-neutron-api:
status: Triaged → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/620683

Changed in charm-nova-cloud-controller:
status: Triaged → In Progress
Vern Hart (vhart) wrote :

I have forked the charm we had deployed (version 252) and applied thedac's patch to the nova.conf template for mitaka (there was no template/newton/nova.conf in 252). Without any other configuration, I launched a RHEL 7.5 instance and it received the correct hostname. I also verified the expected behavior is still consistent on an Ubuntu 18.04 instance.

In both cases, the meta-data has the correct hostname with the configured domain (and not novalocal).

Reviewed: https://review.openstack.org/620668
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=684a93515d7a18d51e1faf9fb43c829f07abce52
Submitter: Zuul
Branch: master

commit 684a93515d7a18d51e1faf9fb43c829f07abce52
Author: David Ames <email address hidden>
Date: Wed Nov 28 09:55:50 2018 -0800

    Set dhcp_domain for nova-metadata-api

    For guest operating systems that rely on nova metadata rather than dhcp
    offers for host dns domain settings it is necessary to set dhcp_domain
    in the nova.conf.

    Change-Id: If7de988ddcd8817d02b261bea601d6922275890c
    Partial-Bug: #1805645

Reviewed: https://review.openstack.org/620678
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=f90eef2c4be311012509284ec2f6530c60192311
Submitter: Zuul
Branch: master

commit f90eef2c4be311012509284ec2f6530c60192311
Author: David Ames <email address hidden>
Date: Wed Nov 28 10:40:18 2018 -0800

    Pass dns-domain over neutron-api relation

    In order to have nova-metadata-api use the same dns-domain as neutron
    when nova-metadata-api runs on nova-cloud-controller we must pass the
    dns-domain on the neutron-api relation.

    Change-Id: I14fe163a445b6dcffccced762041942de7d6b41d
    Partial-Bug: #1805645

Reviewed: https://review.openstack.org/620835
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=ee0378463808105f7200a58746eccb9d48ab58ac
Submitter: Zuul
Branch: stable/18.11

commit ee0378463808105f7200a58746eccb9d48ab58ac
Author: David Ames <email address hidden>
Date: Wed Nov 28 09:55:50 2018 -0800

    Set dhcp_domain for nova-metadata-api

    For guest operating systems that rely on nova metadata rather than dhcp
    offers for host dns domain settings it is necessary to set dhcp_domain
    in the nova.conf.

    Change-Id: If7de988ddcd8817d02b261bea601d6922275890c
    Partial-Bug: #1805645

David Ames (thedac) on 2018-11-29
Changed in charm-neutron-gateway:
status: In Progress → Fix Released
Changed in charm-neutron-api:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/620683
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=e245c0bad4c167847f79fa8c1b3099146cdaec79
Submitter: Zuul
Branch: master

commit e245c0bad4c167847f79fa8c1b3099146cdaec79
Author: David Ames <email address hidden>
Date: Wed Nov 28 10:53:47 2018 -0800

    Receive dns-domain over neutron-api relation

    In order to have nova-metadata-api use the same dns-domain as neutron
    when nova-metadata-api runs on nova-cloud-controller we must recieve
    the dns-domain on the neutron-api relation.

    Change-Id: I22c88a6fea87c56efdb5d93474a5ec7d2a6432d0
    Partial-Bug: #1805645

David Ames (thedac) on 2018-11-30
Changed in charm-nova-cloud-controller:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/621522
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=e50f3ed21059d5f0f0ca0bc02870bf80f7bed677
Submitter: Zuul
Branch: stable/18.11

commit e50f3ed21059d5f0f0ca0bc02870bf80f7bed677
Author: David Ames <email address hidden>
Date: Wed Nov 28 10:53:47 2018 -0800

    Receive dns-domain over neutron-api relation

    In order to have nova-metadata-api use the same dns-domain as neutron
    when nova-metadata-api runs on nova-cloud-controller we must recieve
    the dns-domain on the neutron-api relation.

    Change-Id: I22c88a6fea87c56efdb5d93474a5ec7d2a6432d0
    Partial-Bug: #1805645
    (cherry picked from commit e245c0bad4c167847f79fa8c1b3099146cdaec79)

Reviewed: https://review.openstack.org/621524
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=04d60143a0c6011c4bba5e997e2d9205bb8f0cf6
Submitter: Zuul
Branch: stable/18.11

commit 04d60143a0c6011c4bba5e997e2d9205bb8f0cf6
Author: David Ames <email address hidden>
Date: Wed Nov 28 10:40:18 2018 -0800

    Pass dns-domain over neutron-api relation

    In order to have nova-metadata-api use the same dns-domain as neutron
    when nova-metadata-api runs on nova-cloud-controller we must pass the
    dns-domain on the neutron-api relation.

    Change-Id: I14fe163a445b6dcffccced762041942de7d6b41d
    Partial-Bug: #1805645
    (cherry picked from commit f90eef2c4be311012509284ec2f6530c60192311)

James Page (james-page) on 2018-12-06
Changed in charm-neutron-api:
status: Fix Committed → Fix Released
Changed in charm-nova-cloud-controller:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers