OpenStack Compute (nova)

ServiceCatalog does not contain "network" service

Bug #1752289 reported by Thomas Bechtold
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Thomas Bechtold
Queens
Fix Committed
High
Matt Riedemann

Bug Description

On SLE12SP3, openstack-nova 17.0.0.0~xrc2~dev160-1.1, I try to boot an cirros instance and get:

2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [req-53fa6935-f60d-4e07-bc75-14b6a5336330 f80483de8573468b869e64262780a903 d9685d1130d74a73af6ee213c421d9de - default default] [instance: 0ae671f6-5241-486f-9054-1100b124f704] Instance failed to spawn: EndpointNotFound: ['internal', 'public'] endpoint for network service not found
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] Traceback (most recent call last):
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2236, in _build_resources
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] yield resources
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2019, in _build_and_run_instance
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] block_device_info=block_device_info)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3011, in spawn
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] mdevs=mdevs)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5256, in _get_guest_xml
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] network_info_str = str(network_info)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 568, in __str__
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] return self._sync_wrapper(fn, *args, **kwargs)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 551, in _sync_wrapper
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] self.wait()
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 583, in wait
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] self[:] = self._gt.wait()
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] return self._exit_event.wait()
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] current.throw(*self._exc)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] result = function(*args, **kwargs)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 906, in context_wrapper
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] return func(*args, **kwargs)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1414, in _allocate_network_async
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] six.reraise(*exc_info)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1397, in _allocate_network_async
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] bind_host_id=bind_host_id)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 918, in allocate_for_instance
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] context, instance, neutron, requested_networks, ordered_networks)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 758, in _validate_requested_network_ids
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] auto_allocate=auto_allocate)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 383, in _get_available_networks
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] nets = neutron.list_networks(**search_opts).get('networks', [])
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] ret = obj(*args, **kwargs)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 809, in list_networks
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] **_params)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] ret = obj(*args, **kwargs)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] for r in self._pagination(collection, path, **params):
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] res = self.get(path, params=params)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] ret = obj(*args, **kwargs)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] headers=headers, params=params)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] ret = obj(*args, **kwargs)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] headers=headers, params=params)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] ret = obj(*args, **kwargs)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 282, in do_request
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] headers=headers)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 342, in do_request
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] self._check_uri_length(url)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 335, in _check_uri_length
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] uri_len = len(self.endpoint_url) + len(url)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 349, in endpoint_url
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] return self.get_endpoint()
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 223, in get_endpoint
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] return self.session.get_endpoint(auth or self.auth, **kwargs)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 942, in get_endpoint
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] return auth.get_endpoint(self, **kwargs)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/context.py", line 78, in get_endpoint
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] region_name=region_name)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 338, in url_for
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] endpoint_id=endpoint_id).url
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 399, in endpoint_data_for
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] raise exceptions.EndpointNotFound(msg)
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] EndpointNotFound: ['internal', 'public'] endpoint for network service not found
2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704]
2018-02-28 10:24:07.555 10768 INFO nova.compute.manager [req-53fa6935-f60d-4e07-bc75-14b6a5336330 f80483de8573468b869e64262780a903 d9685d1130d74a73af6ee213c421d9de - default default] [instance: 0ae671f6-5241-486f-9054-1100b124f704] Terminating instance
2018-02-28 10:24:07.560 10768 INFO nova.virt.libvirt.driver [req-53fa6935-f60d-4e07-bc75-14b6a5336330 f80483de8573468b869e64262780a903 d9685d1130d74a73af6ee213c421d9de - default default] [instance: 0ae671f6-5241-486f-9054-1100b124f704] Instance destroyed successfully.

/etc/nova/nova.conf.d/100-nova.conf contains:
[keystone_authtoken]
auth_type = password
username = nova
password = openstack
user_domain_name = Default
project_name = service
project_domain_name = Default
auth_url = http://192.168.122.80:5000/
[neutron]
auth_type = password
username = neutron
password = openstack
user_domain_name = Default
user_domain_id = default
project_name = service
project_domain_name = Default
project_domain_id = default
auth_url = http://192.168.122.80:5000/
service_metadata_proxy = True
metadata_proxy_shared_secret = a3f876fb3286e8bf304c01064e080fae

Theses credentials are correct. I can list the full service catalog when using these creds.
The problem seems, that the context which is used has only a subset of the complete service_catalog:

> /usr/lib/python2.7/site-packages/nova/compute/manager.py(1440)_build_networks_for_instance()
-> macs = self.driver.macs_for_instance(instance)
(Pdb) pp context.to_dict()
{'auth_token': u'gAAAAABaloMiR0qjo1BapDgt93AP5MLLx_VAXY-8GVUHR4iXZkwXmZ-qdnLcba2PKmQETAYqSWEi03ALlvgOCKB1CU6-XWsOXMuqFT9Urnm4D3Zd9Ss_J3QNRAC8furxu7UGtP5XbHymcE8xfjbPDfQNaHfJhTJo511tPd57TkIOcKWbff7JA_M',
 'domain': None,
 'global_request_id': None,
 'instance_lock_checked': False,
 'is_admin': True,
 'is_admin_project': True,
 'project': u'd9685d1130d74a73af6ee213c421d9de',
 'project_domain': u'default',
 'project_id': u'd9685d1130d74a73af6ee213c421d9de',
 'project_name': u'admin',
 'quota_class': None,
 'read_deleted': u'no',
 'read_only': False,
 'remote_address': u'192.168.122.80',
 'request_id': u'req-53fa6935-f60d-4e07-bc75-14b6a5336330',
 'resource_uuid': None,
 'roles': [u'admin'],
 'service_catalog': [{u'endpoints': [{u'adminURL': u'http://192.168.122.80:8780/',
                                      u'internalURL': u'http://192.168.122.80:8780/',
                                      u'publicURL': u'http://192.168.122.80:8780/',
                                      u'region': u'RegionOne'}],
                      u'name': u'placement',
                      u'type': u'placement'},
                     {u'endpoints': [{u'adminURL': u'http://192.168.122.80:9292',
                                      u'internalURL': u'http://192.168.122.80:9292',
                                      u'publicURL': u'http://192.168.122.80:9292',
                                      u'region': u'RegionOne'}],
                      u'name': u'glance',
                      u'type': u'image'},
                     {u'endpoints': [{u'adminURL': u'http://192.168.122.80:8776/v3/d9685d1130d74a73af6ee213c421d9de',
                                      u'internalURL': u'http://192.168.122.80:8776/v3/d9685d1130d74a73af6ee213c421d9de',
                                      u'publicURL': u'http://192.168.122.80:8776/v3/d9685d1130d74a73af6ee213c421d9de',
                                      u'region': u'RegionOne'}],
                      u'name': u'cinderv3',
                      u'type': u'volumev3'},
                     {u'endpoints': [{u'adminURL': u'http://192.168.122.80:9311',
                                      u'internalURL': u'http://192.168.122.80:9311',
                                      u'publicURL': u'http://192.168.122.80:9311',
                                      u'region': u'RegionOne'}],
                      u'name': u'barbican',
                      u'type': u'key-manager'}],
 'show_deleted': False,
 'tenant': u'd9685d1130d74a73af6ee213c421d9de',
 'timestamp': '2018-02-28T10:23:31.133924',
 'user': u'f80483de8573468b869e64262780a903',
 'user_domain': u'default',
 'user_id': u'f80483de8573468b869e64262780a903',
 'user_identity': u'f80483de8573468b869e64262780a903 d9685d1130d74a73af6ee213c421d9de - default default',
 'user_name': u'admin'}

Thomas Bechtold (toabctl)
Changed in nova:
assignee: nobody → Thomas Bechtold (toabctl)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/548572

Revision history for this message
Matt Riedemann (mriedem) wrote :

This is likely related to https://github.com/openstack/nova/commit/6cde77ebbab85bc8ccd2ab7ad977b1d4af4a13fa if it's a regression in Queens.

tags: added: queens-backport-potential
Changed in nova:
importance: Undecided → High
Revision history for this message
Matt Riedemann (mriedem) wrote :

The issue is the 'url' option is missing, so nova uses the ksa adapter code to lookup the entry in the service catalog, but there is no 'network' entry in the context.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/550259

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/548572
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3a3b0f09db318faf1a1ea711a73bb365cab8b233
Submitter: Zuul
Branch: master

commit 3a3b0f09db318faf1a1ea711a73bb365cab8b233
Author: Thomas Bechtold <email address hidden>
Date: Wed Feb 28 11:45:01 2018 +0100

    Allow 'network' in RequestContext service_catalog

    When booting instances, nova might create neutron resources. For that,
    the network service endpoint needs to be available. Otherwise we run
    into:

    EndpointNotFound: ['internal', 'public'] endpoint for network service \
    not found

    Change-Id: Iaed84826b76ab976ffdd1c93106b7bae700a64a9
    Closes-Bug: #1752289

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/550259
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c50ffc9dc185d8aadb1f3addba4085cb8bfbf4d3
Submitter: Zuul
Branch: stable/queens

commit c50ffc9dc185d8aadb1f3addba4085cb8bfbf4d3
Author: Thomas Bechtold <email address hidden>
Date: Wed Feb 28 11:45:01 2018 +0100

    Allow 'network' in RequestContext service_catalog

    When booting instances, nova might create neutron resources. For that,
    the network service endpoint needs to be available. Otherwise we run
    into:

    EndpointNotFound: ['internal', 'public'] endpoint for network service \
    not found

    Change-Id: Iaed84826b76ab976ffdd1c93106b7bae700a64a9
    Closes-Bug: #1752289
    (cherry picked from commit 3a3b0f09db318faf1a1ea711a73bb365cab8b233)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.1

This issue was fixed in the openstack/nova 17.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.0.0.0b1

This issue was fixed in the openstack/nova 18.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.