nova metadata service breaks on hostname

[admin0]

For more than a year, i have deployed multiple clusters using openstack-ansible.
This particular case is using FQDN for the internal and external endpoint.

[DEFAULT]
...

# Metadata
metadata_host = my-int.XXX.com
metadata_port = 8775
metadata_workers = 3
...

Recently, this breaks nova-metadata.

2017-08-25 21:19:02.928 18130 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-08-25 21:19:02.931 18130 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net'
2017-08-25 21:19:03.078 18130 CRITICAL nova [-] AddrFormatError: failed to detect a valid IP address from 'my-int.XXX.com'
2017-08-25 21:19:03.078 18130 ERROR nova Traceback (most recent call last):
2017-08-25 21:19:03.078 18130 ERROR nova File "/openstack/venvs/nova-15.1.8/bin/nova-api-metadata", line 11, in <module>
2017-08-25 21:19:03.078 18130 ERROR nova sys.exit(main())
2017-08-25 21:19:03.078 18130 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/nova/cmd/api_metadata.py", line 48, in main
2017-08-25 21:19:03.078 18130 ERROR nova server = service.WSGIService('metadata', use_ssl=should_use_ssl)
2017-08-25 21:19:03.078 18130 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/nova/service.py", line 309, in __init__
2017-08-25 21:19:03.078 18130 ERROR nova self.manager = self._get_manager()
2017-08-25 21:19:03.078 18130 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/nova/service.py", line 364, in _get_manager
2017-08-25 21:19:03.078 18130 ERROR nova return manager_class()
2017-08-25 21:19:03.078 18130 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/nova/api/manager.py", line 30, in __init__
2017-08-25 21:19:03.078 18130 ERROR nova self.network_driver.metadata_accept()
2017-08-25 21:19:03.078 18130 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/nova/network/linux_net.py", line 599, in metadata_accept
2017-08-25 21:19:03.078 18130 ERROR nova (CONF.metadata_port, _iptables_dest(CONF.metadata_host)))
2017-08-25 21:19:03.078 18130 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/nova/network/linux_net.py", line 588, in _iptables_dest
2017-08-25 21:19:03.078 18130 ERROR nova if ((netaddr.IPAddress(ip).version == 4 and ip == '127.0.0.1')
2017-08-25 21:19:03.078 18130 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/netaddr/ip/__init__.py", line 306, in __init__
2017-08-25 21:19:03.078 18130 ERROR nova 'address from %r' % addr)
2017-08-25 21:19:03.078 18130 ERROR nova AddrFormatError: failed to detect a valid IP address from 'my-int.XXX.com'
2017-08-25 21:19:03.078 18130 ERROR nova
2017-08-25 21:19:09.833 18137 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-08-25 21:19:09.836 18137 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net'
2017-08-25 21:19:10.040 18137 CRITICAL nova [-] AddrFormatError: failed to detect a valid IP address from 'my-int.XXX.com'
2017-08-25 21:19:10.040 18137 ERROR nova Traceback (most recent call last):
2017-08-25 21:19:10.040 18137 ERROR nova File "/openstack/venvs/nova-15.1.8/bin/nova-api-metadata", line 11, in <module>
2017-08-25 21:19:10.040 18137 ERROR nova sys.exit(main())
2017-08-25 21:19:10.040 18137 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/nova/cmd/api_metadata.py", line 48, in main
2017-08-25 21:19:10.040 18137 ERROR nova server = service.WSGIService('metadata', use_ssl=should_use_ssl)
2017-08-25 21:19:10.040 18137 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/nova/service.py", line 309, in __init__
2017-08-25 21:19:10.040 18137 ERROR nova self.manager = self._get_manager()
2017-08-25 21:19:10.040 18137 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/nova/service.py", line 364, in _get_manager
2017-08-25 21:19:10.040 18137 ERROR nova return manager_class()
2017-08-25 21:19:10.040 18137 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/nova/api/manager.py", line 30, in __init__
2017-08-25 21:19:10.040 18137 ERROR nova self.network_driver.metadata_accept()
2017-08-25 21:19:10.040 18137 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/nova/network/linux_net.py", line 599, in metadata_accept
2017-08-25 21:19:10.040 18137 ERROR nova (CONF.metadata_port, _iptables_dest(CONF.metadata_host)))
2017-08-25 21:19:10.040 18137 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/nova/network/linux_net.py", line 588, in _iptables_dest
2017-08-25 21:19:10.040 18137 ERROR nova if ((netaddr.IPAddress(ip).version == 4 and ip == '127.0.0.1')
2017-08-25 21:19:10.040 18137 ERROR nova File "/openstack/venvs/nova-15.1.8/lib/python2.7/site-packages/netaddr/ip/__init__.py", line 306, in __init__
2017-08-25 21:19:10.040 18137 ERROR nova 'address from %r' % addr)
2017-08-25 21:19:10.040 18137 ERROR nova AddrFormatError: failed to detect a valid IP address from 'my-int.XXX.com'
2017-08-25 21:19:10.040 18137 ERROR nova

Changing hostname to IP address fixes it.

# Metadata
metadata_host = 172.16.236.2
metadata_port = 8775
metadata_workers = 3
...

Is this a bug? or a new change? or should we now should not use FQDN in host ?

Thanks,
Shashi

[Matt Riedemann (mriedem)]

Which release were you using when this worked, and which release did you upgrade to where it no longer works?

[Matt Riedemann (mriedem)]

Are you using nova-network or neutron?

[admin0 (shashi-eu)]

nova 15.1.8
deployed via openstack-ansible stable/ocata

[Matt Riedemann (mriedem)]

Looks like this code hasn't changed in a long time:

netaddr.IPAddress(ip)

So it's probably a change in the netaddr library. Compare the versions where it works and where it doesn't and check the change log / release notes for netaddr between those versions.

[Matt Riedemann (mriedem)]

Looks like netaddr.IPAddress has always expected this to be a IP rather than a FQDN:

https://github.com/drkjam/netaddr/blob/netaddr-0.7.19/netaddr/ip/__init__.py#L244

I'm not sure how it was working before.

[Matt Riedemann (mriedem)]

Also, metadata_host by default is CONF.my_ip, which is an IP address, not a hostname. These options should both be IPOpts instead of StrOpts probably, but metadata_host is deprecated since 16.0.0 (pike) so we probably won't bother changing the type on that one now.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/498095

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/498095