Nova metadata service returns wrong hostname after neutron-api dns-domain config is set

Bug #1805645 reported by Pedro Guimarães
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Neutron API Charm
Fix Released
Medium
David Ames
OpenStack Neutron Gateway Charm
Fix Released
High
David Ames
OpenStack Nova Cloud Controller Charm
Fix Released
Medium
David Ames
OpenStack Nova Compute Charm
Fix Released
High
David Ames

Bug Description

We are facing an issue to configure metadata service correctly. Neutron API charm's dns-domain was configured for an specific value but nova metadata service continues to return original "NAME_OF_SERVER.novalocal." as hostname.

dns-domain seems to correctly configure DHCP domain name.

curl http://169.254.169.254/openstack/latest/meta_data.json returns:
{"random_seed": "redacted", "type": "ssh", "name": "dnstest"}], "hostname": "tdnsserver.novalocal", "launch_index": 0, "devices": [], "public_keys": {"dnstest": "redacted"}, "project_id": "redacted", "name": "tdnsserver"}

While /var/lib/dhclient/dhclient.leases contains the correct domain name.

However, not all types of Guest OS follow DHCP's definition of domain. RHEL-based guest OS, for example, set their hostnames to the value present on metadata service despite DHCP's original configuration. Although one can work-around this by using user_data flag on "openstack server create" and setting "preserve_hostname: true", there are situations where this is not feasible (e.g. when OpenStack is under management of an orchestrator).

We have a request to setup Designate to register all machines that are connected to a self-service network. That deployment is using BGP to advertise that SSN.

On a particular customer, we have the following configuration set:
Neutron-api: https://pastebin.canonical.com/p/vPhfkCbdw6/
Please, check that vni_ranges jump on one value, that was necessary to make the SSN available to Designate, as described on: https://docs.openstack.org/mitaka/networking-guide/config-dns-int.html

Nova-cloud-controller: https://pastebin.canonical.com/p/yThtgY95BT/
Neutron-Gateway: https://pastebin.canonical.com/p/q97hqfgMCP/
Neutron-Openvswitch: https://pastebin.canonical.com/p/MPJ3zKSYxK/

Revision history for this message
Pedro Guimarães (pguimaraes) wrote :

Marking as Field Critical as I'm facing a deployment on a customer that is exactly on the Guest OS/orchestrator situation described above.

Revision history for this message
James Page (james-page) wrote :

This is the code that generates the hostname in the nova metadata:

    def _get_hostname(self):
        return "%s%s%s" % (self.instance.hostname,
                           '.' if CONF.dhcp_domain else '',
                           CONF.dhcp_domain)

This uses a deprecated dhcp_domain option that was part of nova-network.

Revision history for this message
James Page (james-page) wrote :

You might try setting this option via config-flags to match the dns-domain value in neutron-api - that would at least make the metadata consistent with the dns domain information being created and handed out via dhcp.

description: updated
Revision history for this message
Ryan Beisner (1chb1n) wrote :

Upstream reference, which appears to be a dup/invalid stance: https://bugs.launchpad.net/nova/+bug/1694417

Revision history for this message
Ryan Beisner (1chb1n) wrote :

See also, the behavior difference in consuming the data: https://bugs.launchpad.net/cloud-init/+bug/1649802

Changed in charm-neutron-gateway:
importance: Undecided → High
assignee: nobody → David Ames (thedac)
milestone: none → 19.04
David Ames (thedac)
Changed in charm-neutron-gateway:
status: New → Triaged
Revision history for this message
Pedro Guimarães (pguimaraes) wrote :

@james-page
I have:
$ juju config neutron-api config-flags
dhcp_domain=RIGHT_URL.
$ juju config nova-cloud-controller config-flags
dhcp_domain=RIGHT_URL.,dns_domain=RIGHT_URL.

Still receiving *.novalocal from metadata server when executing

curl http://169.254.169.254/openstack/latest/meta_data.json

Revision history for this message
David Ames (thedac) wrote :

Confirmed setting dhcp_domain on the neutron-gateway's nova.conf allows VMs booted after the configuration change to be able to retrieve the correct domain for its hostname from the nova-metadata-api.

Check metadata hostname with this curl statement:
curl http://169.254.169.254/latest/meta-data/hostname

PR on the way.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-neutron-gateway (master)

Fix proposed to branch: master
Review: https://review.openstack.org/620668

Changed in charm-neutron-gateway:
status: Triaged → In Progress
Revision history for this message
David Ames (thedac) wrote :

Pedro,

The dhcp_domain needs to be set on the neutron-gateway at Queens because the nova-metadata-api runs there.

Please test the following which will set the domain to the value neutron-api's dns-domain is set to.
cs:~thedac/neutron-gateway-2
https://review.openstack.org/#/c/620668/

This test will show what is configured on the nova-metadata-api:
curl http://169.254.169.254/latest/meta-data/hostname

Revision history for this message
David Ames (thedac) wrote :

For completeness, I am adding neutron-api and nova-cloud-controller for OpenStack releases >= Rocky where we now run nova-metadata-api on the nova-cloud-controller.

Note: these are lower priority and do not affect the environment which caused the field escalation. This PR solves the escalation: https://review.openstack.org/#/c/620668/

Changed in charm-neutron-api:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → David Ames (thedac)
milestone: none → 19.04
Changed in charm-nova-cloud-controller:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → David Ames (thedac)
milestone: none → 19.04
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-neutron-api (master)

Fix proposed to branch: master
Review: https://review.openstack.org/620678

Changed in charm-neutron-api:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-cloud-controller (master)

Fix proposed to branch: master
Review: https://review.openstack.org/620683

Changed in charm-nova-cloud-controller:
status: Triaged → In Progress
Revision history for this message
Vern Hart (vern) wrote :

I have forked the charm we had deployed (version 252) and applied thedac's patch to the nova.conf template for mitaka (there was no template/newton/nova.conf in 252). Without any other configuration, I launched a RHEL 7.5 instance and it received the correct hostname. I also verified the expected behavior is still consistent on an Ubuntu 18.04 instance.

In both cases, the meta-data has the correct hostname with the configured domain (and not novalocal).

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-neutron-gateway (stable/18.11)

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/620835

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-gateway (master)

Reviewed: https://review.openstack.org/620668
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=684a93515d7a18d51e1faf9fb43c829f07abce52
Submitter: Zuul
Branch: master

commit 684a93515d7a18d51e1faf9fb43c829f07abce52
Author: David Ames <email address hidden>
Date: Wed Nov 28 09:55:50 2018 -0800

    Set dhcp_domain for nova-metadata-api

    For guest operating systems that rely on nova metadata rather than dhcp
    offers for host dns domain settings it is necessary to set dhcp_domain
    in the nova.conf.

    Change-Id: If7de988ddcd8817d02b261bea601d6922275890c
    Partial-Bug: #1805645

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-api (master)

Reviewed: https://review.openstack.org/620678
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=f90eef2c4be311012509284ec2f6530c60192311
Submitter: Zuul
Branch: master

commit f90eef2c4be311012509284ec2f6530c60192311
Author: David Ames <email address hidden>
Date: Wed Nov 28 10:40:18 2018 -0800

    Pass dns-domain over neutron-api relation

    In order to have nova-metadata-api use the same dns-domain as neutron
    when nova-metadata-api runs on nova-cloud-controller we must pass the
    dns-domain on the neutron-api relation.

    Change-Id: I14fe163a445b6dcffccced762041942de7d6b41d
    Partial-Bug: #1805645

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-gateway (stable/18.11)

Reviewed: https://review.openstack.org/620835
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=ee0378463808105f7200a58746eccb9d48ab58ac
Submitter: Zuul
Branch: stable/18.11

commit ee0378463808105f7200a58746eccb9d48ab58ac
Author: David Ames <email address hidden>
Date: Wed Nov 28 09:55:50 2018 -0800

    Set dhcp_domain for nova-metadata-api

    For guest operating systems that rely on nova metadata rather than dhcp
    offers for host dns domain settings it is necessary to set dhcp_domain
    in the nova.conf.

    Change-Id: If7de988ddcd8817d02b261bea601d6922275890c
    Partial-Bug: #1805645

David Ames (thedac)
Changed in charm-neutron-gateway:
status: In Progress → Fix Released
Changed in charm-neutron-api:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-cloud-controller (master)

Reviewed: https://review.openstack.org/620683
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=e245c0bad4c167847f79fa8c1b3099146cdaec79
Submitter: Zuul
Branch: master

commit e245c0bad4c167847f79fa8c1b3099146cdaec79
Author: David Ames <email address hidden>
Date: Wed Nov 28 10:53:47 2018 -0800

    Receive dns-domain over neutron-api relation

    In order to have nova-metadata-api use the same dns-domain as neutron
    when nova-metadata-api runs on nova-cloud-controller we must recieve
    the dns-domain on the neutron-api relation.

    Change-Id: I22c88a6fea87c56efdb5d93474a5ec7d2a6432d0
    Partial-Bug: #1805645

David Ames (thedac)
Changed in charm-nova-cloud-controller:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-cloud-controller (stable/18.11)

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/621522

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-neutron-api (stable/18.11)

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/621524

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-cloud-controller (stable/18.11)

Reviewed: https://review.openstack.org/621522
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=e50f3ed21059d5f0f0ca0bc02870bf80f7bed677
Submitter: Zuul
Branch: stable/18.11

commit e50f3ed21059d5f0f0ca0bc02870bf80f7bed677
Author: David Ames <email address hidden>
Date: Wed Nov 28 10:53:47 2018 -0800

    Receive dns-domain over neutron-api relation

    In order to have nova-metadata-api use the same dns-domain as neutron
    when nova-metadata-api runs on nova-cloud-controller we must recieve
    the dns-domain on the neutron-api relation.

    Change-Id: I22c88a6fea87c56efdb5d93474a5ec7d2a6432d0
    Partial-Bug: #1805645
    (cherry picked from commit e245c0bad4c167847f79fa8c1b3099146cdaec79)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-api (stable/18.11)

Reviewed: https://review.openstack.org/621524
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=04d60143a0c6011c4bba5e997e2d9205bb8f0cf6
Submitter: Zuul
Branch: stable/18.11

commit 04d60143a0c6011c4bba5e997e2d9205bb8f0cf6
Author: David Ames <email address hidden>
Date: Wed Nov 28 10:40:18 2018 -0800

    Pass dns-domain over neutron-api relation

    In order to have nova-metadata-api use the same dns-domain as neutron
    when nova-metadata-api runs on nova-cloud-controller we must pass the
    dns-domain on the neutron-api relation.

    Change-Id: I14fe163a445b6dcffccced762041942de7d6b41d
    Partial-Bug: #1805645
    (cherry picked from commit f90eef2c4be311012509284ec2f6530c60192311)

James Page (james-page)
Changed in charm-neutron-api:
status: Fix Committed → Fix Released
Changed in charm-nova-cloud-controller:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-cloud-controller (master)

Fix proposed to branch: master
Review: https://review.opendev.org/701598

Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

The fix released was only for Rocky+ and I was still hitting this exact same issue on a Queens/Bionic installation.

A new release that include the fix for Queens in being prepared.

Revision history for this message
David Ames (thedac) wrote :

There are two new problems.

1) The original fix was only applied to Rocky and above
2) The original fix only applied when a network node (neutron-gateway) was deployed. For distributed deployments (metadata running on nova-compute hosts) the problem still remains.

For 1, this is an update for nova-cloud-controller: https://review.opendev.org/701598

For 2, we need to pass the dns_domain information to nova-compute either directly from neutron-api or from nova-cloud-controller and update nova-compute templates similar to nova-cloud-controller.

Changed in charm-nova-compute:
status: New → In Progress
importance: Undecided → High
assignee: nobody → David Ames (thedac)
milestone: none → 20.02
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-cloud-controller (master)

Reviewed: https://review.opendev.org/701598
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=a8c76095ab7865973b1449e71ede2dbc5cd3a857
Submitter: Zuul
Branch: master

commit a8c76095ab7865973b1449e71ede2dbc5cd3a857
Author: David Ames <email address hidden>
Date: Wed Jan 8 11:31:10 2020 -0800

    Receive dns-domain over neutron-api relation

    In commit e245c0bad4c167847f79fa8c1b3099146cdaec79 we updated the Rocky
    template only. We have Queens deployments that also requires this
    functionality. Pike is the extant template to affect Queens.

    In order to have nova-metadata-api use the same dns-domain as neutron
    when nova-metadata-api runs on nova-cloud-controller we must recieve
    the dns-domain on the neutron-api relation.

    Change-Id: I9364cf57664695b08411cc3c6a101e132ea58cc7
    Partial-Bug: #1805645

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-cloud-controller (master)

Fix proposed to branch: master
Review: https://review.opendev.org/702055

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-compute (master)

Fix proposed to branch: master
Review: https://review.opendev.org/702056

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-compute (master)

Reviewed: https://review.opendev.org/702056
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=e39738355ee29a85f1f6f3f47f003c02a4b18aef
Submitter: Zuul
Branch: master

commit e39738355ee29a85f1f6f3f47f003c02a4b18aef
Author: David Ames <email address hidden>
Date: Fri Jan 10 15:40:03 2020 -0800

    Get DNS domain for nova metadata

    Get the DNS domain for the dhcp_domain setting for nova metadata over
    the cloud-compute relation from nova-cloud-controller.

    Fixes Bug #1805645 when using provider networks with no network node.

    Change-Id: Iaba14e6939ab5d488c6299ce5cb33e26f757cf95
    Partial-Bug: #1805645

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-cloud-controller (master)

Reviewed: https://review.opendev.org/702055
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=bc1bb8ffe73ddccd333740a566552bd77367bbf8
Submitter: Zuul
Branch: master

commit bc1bb8ffe73ddccd333740a566552bd77367bbf8
Author: David Ames <email address hidden>
Date: Wed Jan 8 11:31:10 2020 -0800

    Handle DNS domain for metadata

    Pass the dns-domain over the cloud-compute relation to nova-compute.

    Change-Id: I184e955488881e7329b1e4a1670261a88ec4f7fa
    Partial-Bug: #1805645

James Page (james-page)
Changed in charm-nova-compute:
status: In Progress → Fix Committed
Liam Young (gnuoy)
Changed in charm-nova-compute:
status: Fix Committed → Fix Released
Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

There is a special case which this fix did not cover, that is the "enable-local-dhcp-and-metadata" option on neutron-openvswitch (and no neutron-gateway).

In this case, the nova files that need to be updated with dhcp_domain option is the nova.conf on the compute nodes themselves (in this particular case, this config file is used not only by the nova-compute but also by the nova-metadata-api that is running on the compute nodes because of the local-metadata option).

So, in this case the workaround is to add config-flags="dhcp_domain=DOMAIN." on nova-compute and not on nova-cloud-controller.

I tested and it fixes the issue. But it would be nice for this to be incorporated on the charm.

Two options I see are:

- add a dns-domain (or dhcp-domain) to nova-compute charm
- improve the nova-compute charm to receive dns-domain over neutorn-api relation the same way as nova-cloud-controller is doing after the fix discussed on this bug report.

David Ames (thedac)
Changed in charm-nova-compute:
status: Fix Released → Triaged
David Ames (thedac)
Changed in charm-nova-compute:
status: Triaged → Fix Released
Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

Updates:

1) I noticed that indeed the bug was fixed in both nova-cloud-controller and nova-compute (apart from neutron-api), and therefore my last comment is wrong.
2) The fix was done in a later version, I was running old charms and I needed to upgrade to take advantage of the fix.

So, the cloud was installed with charms release 19.04, and therefore it contained:

nova-compute 309 (but the fix was only in 314)
neutron-api 281 (but the fix was only in 284)

Both new charms are part of charms release 20.02 if I understand it correctly.

So I upgraded both charms to the new versions. But it did not work as expected. The dhcp_domain option is not being added to the config. It still works if I manually insert it thru config-flags, but not automatically.

Is there any other charm that should be also upgraded for this to work?

Revision history for this message
David Ames (thedac) wrote :

> Is there any other charm that should be also upgraded for this to work?

Yes, as you suspected nova-cloud-controller also has a change that is needed.

So it is necessary to have the 20.02+ versions of nova-compute, nova-cloud-controller and neutron-api for a DVR topology to have the fix.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.