Description
===========
As described in test_multi_nodes_isolate() in https://github.com/openstack/nova/blob/master/nova/tests/unit/virt/test_hardware.py#L3006-L3024,
numa_fit_instance_to_host() function returns None for cpuset_reserved for cells with id > 0.
---------
def test_multi_nodes_isolate(self):
host_topo = self._host_topology()
inst_topo = objects.InstanceNUMATopology(
emulator_threads_policy=(
fields.CPUEmulatorThreadsPolicy.ISOLATE),
cells=[objects.InstanceNUMACell(
id=0,
cpuset=set([0]), memory=2048,
cpu_policy=fields.CPUAllocationPolicy.DEDICATED),
objects.InstanceNUMACell(
id=1,
cpuset=set([1]), memory=2048,
cpu_policy=fields.CPUAllocationPolicy.DEDICATED)])
inst_topo = hw.numa_fit_instance_to_host(host_topo, inst_topo)
self.assertEqual({0: 0}, inst_topo.cells[0].cpu_pinning)
self.assertEqual(set([1]), inst_topo.cells[0].cpuset_reserved)
self.assertEqual({1: 2}, inst_topo.cells[1].cpu_pinning)
self.assertIsNone(inst_topo.cells[1].cpuset_reserved)
--------
However, we are testing libvirt driver with non-None cpuset_reserved value in
https://github.com/openstack/nova/blob/master/nova/tests/unit/virt/libvirt/test_driver.py#L3052.
--------
def test_get_guest_config_numa_host_instance_isolated_emulator_threads(
self):
instance_topology = objects.InstanceNUMATopology(
emulator_threads_policy=(
fields.CPUEmulatorThreadsPolicy.ISOLATE),
cells=[
objects.InstanceNUMACell(
id=0, cpuset=set([0, 1]),
memory=1024, pagesize=2048,
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
cpu_pinning={0: 4, 1: 5},
cpuset_reserved=set([6])),
objects.InstanceNUMACell(
id=1, cpuset=set([2, 3]),
memory=1024, pagesize=2048,
cpu_policy=fields.CPUAllocationPolicy.DEDICATED,
cpu_pinning={2: 7, 3: 8},
cpuset_reserved=set([]))]) # <- this should be None!!
...(snip)...
--------
Actually, this causes errors when deploying VMs with multi numa nodes with `emulator_threads_policy=isolate`.
Environment & Steps to reproduce
==================
1. Use devstack to build all-in-one OpenStack with libvirt/KVM driver on a VM whose lscpu looks like this.
--------
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 8
NUMA node(s): 2
NUMA node0 CPU(s): 0-3
NUMA node1 CPU(s): 4-7
--------
2. Try to Build a VM with multi numa nodes with emulator_threads_policy=isolate.
--------
$ openstack flavor create c2r1024d1 --id 6 --ram 1024 --disk 1 --vcpu 2
$ openstack flavor set c2r1024d1 --property hw:cpu_policy=dedicated --property hw:emulator_threads_policy=isolate --property hw:numa_nodes=2
$ openstack server create test1 --image cirros-0.3.5-x86_64-disk --flavor c2r1024d1 --network private
--------
Expected & Actual result
===============
Expected: VM is built without an error
Actual: VM goes into an ERROR state with the following message.
--------
$ openstack server show test1
--------
...(snip)...
| fault | {u'message': u'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e8d7537f-9140-4ae6-bb3f-b98458a862ac.', u'code': 500, u'details': u' File "/opt/stack/nova/nova/conductor/manager.py", line 578, in build_instances\\n raise exception.MaxRetriesExceeded(reason=msg)\\n', u'created': u'2018-02-07T09:48:43Z'} |
...(snip)...
Logs & Configs
==============
--------
$ sudo journalctl -a --unit devstack@n-* | tail -1000 | grep -e ERROR -e INFO
--------
...(snip)...
Feb 07 04:48:38 ubuntu01 nova-scheduler[5195]: INFO nova.virt.hardware [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] Computed NUMA topology CPU pinning: usable pCPUs: [[0], [1], [2], [3]], vCPUs mapping: [(0, 0)]
Feb 07 04:48:38 ubuntu01 nova-scheduler[5195]: INFO nova.virt.hardware [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] Computed NUMA topology reserved pCPUs: usable pCPUs: [[], [1], [2], [3]], reserved pCPUs: set([1])
Feb 07 04:48:38 ubuntu01 nova-scheduler[5195]: INFO nova.virt.hardware [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] Computed NUMA topology CPU pinning: usable pCPUs: [[4], [5], [6], [7]], vCPUs mapping: [(1, 4)]
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Attempting claim on node ubuntu01: memory 1024 MB, disk 1 GB, vcpus 2 CPU
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Total memory: 5824 MB, used: 512.00 MB
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] memory limit not specified, defaulting to unlimited
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Total disk: 90 GB, used: 0.00 GB
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] disk limit not specified, defaulting to unlimited
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Total vcpu: 8 VCPU, used: 0.00 VCPU
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] vcpu limit not specified, defaulting to unlimited
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.virt.hardware [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] Computed NUMA topology CPU pinning: usable pCPUs: [[0], [1], [2], [3]], vCPUs mapping: [(0, 0)]
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.virt.hardware [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] Computed NUMA topology reserved pCPUs: usable pCPUs: [[], [1], [2], [3]], reserved pCPUs: set([1])
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.virt.hardware [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] Computed NUMA topology CPU pinning: usable pCPUs: [[4], [5], [6], [7]], vCPUs mapping: [(1, 4)]
Feb 07 04:48:39 ubuntu01 nova-compute[9951]: INFO nova.compute.claims [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Claim successful on node ubuntu01
Feb 07 04:48:40 ubuntu01 nova-compute[9951]: INFO nova.virt.libvirt.driver [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Creating image
Feb 07 04:48:41 ubuntu01 <email address hidden>[28408]: INFO nova.api.openstack.compute.server_external_events [None req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova] Creating event network-changed:a67cd6cd-1553-4f36-bbcc-eea893f55941 for instance e8d7537f-9140-4ae6-bb3f-b98458a862ac on ubuntu01
Feb 07 04:48:41 ubuntu01 <email address hidden>[28408]: INFO nova.api.openstack.requestlog [None req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova] 192.168.122.101 "POST /compute/v2.1/os-server-external-events" status: 200 len: 179 microversion: 2.1 time: 0.080556
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [None req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Instance failed to spawn: TypeError: 'NoneType' object is not iterable
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Traceback (most recent call last):
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/compute/manager.py", line 2249, in _build_resources
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] yield resources
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/compute/manager.py", line 2033, in _build_and_run_instance
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] block_device_info=block_device_info)
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3005, in spawn
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] mdevs=mdevs)
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5253, in _get_guest_xml
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] context, mdevs)
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4987, in _get_guest_config
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] instance.numa_topology, flavor, allowed_cpus, image_meta)
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4326, in _get_guest_numa_config
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] object_numa_cell.cpuset_reserved)
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] TypeError: 'NoneType' object is not iterable
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: ERROR nova.compute.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac]
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO nova.compute.manager [None req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Terminating instance
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO nova.virt.libvirt.driver [None req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Instance destroyed successfully.
Feb 07 04:48:42 ubuntu01 ovs-vsctl[18497]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=120 -- --if-exists del-port br-int qvoa67cd6cd-15
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO os_vif [None req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:77:cf:80,bridge_name='qbra67cd6cd-15',has_traffic_filtering=True,id=a67cd6cd-1553-4f36-bbcc-eea893f55941,network=Network(f17bbb1a-c562-4722-a95a-5c117af6ee19),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapa67cd6cd-15')
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO nova.virt.libvirt.driver [None req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Deleting instance files /opt/stack/data/nova/instances/e8d7537f-9140-4ae6-bb3f-b98458a862ac_del
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO nova.virt.libvirt.driver [None req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Deletion of /opt/stack/data/nova/instances/e8d7537f-9140-4ae6-bb3f-b98458a862ac_del complete
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO nova.compute.manager [None req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Took 0.19 seconds to destroy the instance on the hypervisor.
Feb 07 04:48:42 ubuntu01 nova-compute[9951]: INFO nova.scheduler.client.report [None req-a9935589-09fd-45aa-b172-0e51ef79ca88 service nova] Deleted allocation for instance e8d7537f-9140-4ae6-bb3f-b98458a862ac
Feb 07 04:48:42 ubuntu01 nova-conductor[9025]: ERROR nova.scheduler.utils [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Error from last host: ubuntu01 (node ubuntu01): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1851, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2119, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance e8d7537f-9140-4ae6-bb3f-b98458a862ac was re-scheduled: 'NoneType' object is not iterable\n"]
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: WARNING nova.scheduler.utils [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Setting instance to ERROR state.: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e8d7537f-9140-4ae6-bb3f-b98458a862ac.
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [None req-ec367a39-e9b3-4403-ab28-00519b5760ce demo admin] [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Failed to deallocate networks: EndpointNotFound: ['internal', 'public'] endpoint for network service not found
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] Traceback (most recent call last):
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/conductor/manager.py", line 364, in _cleanup_allocated_networks
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] context, instance, requested_networks=requested_networks)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 1253, in deallocate_for_instance
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] data = neutron.list_ports(**search_opts)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 114, in wrapper
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] ret = obj(*args, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 786, in list_ports
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] **_params)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 114, in wrapper
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] ret = obj(*args, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 369, in list
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] for r in self._pagination(collection, path, **params):
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 384, in _pagination
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] res = self.get(path, params=params)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 114, in wrapper
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] ret = obj(*args, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 354, in get
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] headers=headers, params=params)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 114, in wrapper
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] ret = obj(*args, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 331, in retry_request
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] headers=headers, params=params)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 114, in wrapper
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] ret = obj(*args, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 282, in do_request
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] headers=headers)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 342, in do_request
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] self._check_uri_length(url)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 335, in _check_uri_length
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] uri_len = len(self.endpoint_url) + len(url)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 349, in endpoint_url
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] return self.get_endpoint()
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 223, in get_endpoint
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] return self.session.get_endpoint(auth or self.auth, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 942, in get_endpoint
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] return auth.get_endpoint(self, **kwargs)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/opt/stack/nova/nova/context.py", line 78, in get_endpoint
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] region_name=region_name)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/access/service_catalog.py", line 338, in url_for
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] endpoint_id=endpoint_id).url
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/access/service_catalog.py", line 400, in endpoint_data_for
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] raise exceptions.EndpointNotFound(msg)
Feb 07 04:48:43 ubuntu01 nova-conductor[9025]: ERROR nova.conductor.manager [instance: e8d7537f-9140-4ae6-bb3f-b98458a862ac] EndpointNotFound: ['internal', 'public'] endpoint for network service not found
...(snip)...
Fix proposed to branch: master /review. openstack. org/539865
Review: https:/