OpenStack Compute (nova)

Unexpected Exception in API Error when creating instance with Neutron

Bug #1563513 reported by Matthew on 2016-03-29

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Invalid	Undecided	Unassigned

Bug Description

I went through the Liberty install guide using networking Option 2. I created a public and a private network in the admin project (logged on as the admin user), then attempted to launch an instance from the Horizon GUI.

In the GUI, I get two error popups: "Error: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.NeutronClientException'> (HTTP 500) (Request-ID: req-2cb42761-a3d0-4c9a-8fe6-10ae746d60e7)" and "Error: Unable to launch instance named "TestInstance"."

On the controller node, the nova-api.log has:

2016-03-29 13:35:37.263 2044 INFO nova.osapi_compute.wsgi.server [req-1963eedc-3a0a-4ee7-bab3-fbcada58d0ec 7e4ea1507e684f068ab68ec1e2ce77c4 138c2905912e4277b9845e5811614301 - - -] 10.0.0.19 "GET /v2/138c2905912e4277b9845e5811614301/os-availability-zone HTTP/1.1" status: 200 len: 293 time: 0.0253379
2016-03-29 13:35:37.280 2044 INFO nova.osapi_compute.wsgi.server [req-b4a0637c-3f9a-4c36-a4e9-b33071c7eb68 7e4ea1507e684f068ab68ec1e2ce77c4 138c2905912e4277b9845e5811614301 - - -] 10.0.0.19 "GET /v2/138c2905912e4277b9845e5811614301/flavors/detail HTTP/1.1" status: 200 len: 2227 time: 0.0146720
2016-03-29 13:35:37.524 2042 INFO nova.osapi_compute.wsgi.server [req-acb0e2f2-17b6-4920-bafb-dd5cebb0a8f8 7e4ea1507e684f068ab68ec1e2ce77c4 138c2905912e4277b9845e5811614301 - - -] 10.0.0.19 "GET /v2/138c2905912e4277b9845e5811614301/os-quota-sets/138c2905912e4277b9845e5811614301 HTTP/1.1" status: 200 len: 568 time: 0.0112181
2016-03-29 13:35:37.584 2043 INFO nova.osapi_compute.wsgi.server [req-926bc8fa-d927-4560-9488-1517ee97c713 7e4ea1507e684f068ab68ec1e2ce77c4 138c2905912e4277b9845e5811614301 - - -] 10.0.0.19 "GET /v2/138c2905912e4277b9845e5811614301/servers/detail?all_tenants=True&tenant_id=138c2905912e4277b9845e5811614301 HTTP/1.1" status: 200 len: 211 time: 0.0455759
2016-03-29 13:35:37.815 2041 INFO nova.osapi_compute.wsgi.server [req-3a615414-417b-4640-bbe1-37462f8ee9c5 7e4ea1507e684f068ab68ec1e2ce77c4 138c2905912e4277b9845e5811614301 - - -] 10.0.0.19 "GET /v2/138c2905912e4277b9845e5811614301/os-keypairs HTTP/1.1" status: 200 len: 212 time: 0.0068719
2016-03-29 13:35:37.957 2040 INFO nova.osapi_compute.wsgi.server [req-f6aa1fa3-08ff-426d-b217-5fb8488e1dca 7e4ea1507e684f068ab68ec1e2ce77c4 138c2905912e4277b9845e5811614301 - - -] 10.0.0.19 "GET /v2/138c2905912e4277b9845e5811614301/extensions HTTP/1.1" status: 200 len: 21880 time: 0.0278120
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions [req-2cb42761-a3d0-4c9a-8fe6-10ae746d60e7 7e4ea1507e684f068ab68ec1e2ce77c4 138c2905912e4277b9845e5811614301 - - -] Unexpected exception in API method
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 611, in create
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions **create_kwargs)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 149, in inner
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1581, in create
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1181, in _create_instance
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions auto_disk_config, reservation_id, max_count)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 923, in _validate_and_build_base_options
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions requested_networks, max_count)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 533, in _check_requested_networks
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions max_count)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1171, in validate_networks
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions context, neutron, requested_networks)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1143, in _ports_needed_per_instance
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions neutron=neutron)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 282, in _get_available_networks
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions nets = neutron.list_networks(**search_opts).get('networks', [])
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102, in with_params
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions ret = self.function(instance, *args, **kwargs)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 574, in list_networks
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions **_params)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 307, in list
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions for r in self._pagination(collection, path, **params):
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 320, in _pagination
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions res = self.get(path, params=params)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 293, in get
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions headers=headers, params=params)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 270, in retry_request
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions headers=headers, params=params)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 211, in do_request
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions self._handle_fault_response(status_code, replybody)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions exception_handler_v20(status_code, des_error_body)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions message=message)
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions NeutronClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions <html><head>
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions <title>404 Not Found</title>
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions </head><body>
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions <h1>Not Found</h1>
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions <p>The requested URL /9696/v2.0/networks.json was not found on this server.</p>
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions <hr>
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions <address>Apache/2.4.7 (Ubuntu) Server at osctrl Port 80</address>
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions </body></html>
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions
2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions
2016-03-29 13:35:38.215 2040 INFO nova.api.openstack.wsgi [req-2cb42761-a3d0-4c9a-8fe6-10ae746d60e7 7e4ea1507e684f068ab68ec1e2ce77c4 138c2905912e4277b9845e5811614301 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.NeutronClientException'>
2016-03-29 13:35:38.216 2040 INFO nova.osapi_compute.wsgi.server [req-2cb42761-a3d0-4c9a-8fe6-10ae746d60e7 7e4ea1507e684f068ab68ec1e2ce77c4 138c2905912e4277b9845e5811614301 - - -] 10.0.0.19 "POST /v2/138c2905912e4277b9845e5811614301/servers HTTP/1.1" status: 500 len: 459 time: 0.2535379
...

I think the relevant line is, "<p>The requested URL /9696/v2.0/networks.json was not found on this server.</p>"

Neutron agent-list shows:

I'm afraid I'm new enough at OpenStack that I need some guidance to troubleshoot this further. Any help is appreciated.

Revision history for this message

Augustina Ragwitz (auggy) wrote on 2016-03-29:

This isn't a very helpful message from the Nova api:

2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions [req-2cb42761-a3d0-4c9a-8fe6-10ae746d60e7 7e4ea1507e684f068ab68ec1e2ce77c4 138c2905912e4277b9845e5811614301 - - -] Unexpected exception in API method

And yes, it looks like this is the line that's causing the failure:

2016-03-29 13:35:38.212 2040 ERROR nova.api.openstack.extensions <p>The requested URL /9696/v2.0/networks.json was not found on this server.</p>

My first inclination is that you're hitting an exception state that isn't currently covered in the Nova API and we should probably fix that. In order to get to the same state, I'll need to know the following:

1) what version of OpenStack Nova and Neutron are you running? I'm assuming Liberty since you mentioned you are using the install guide from that, but please double check.

2) Can you provide detailed steps to reproduce? You mentioned the install guide, could you provide a link to the section you used and if you didn't follow their instructions exactly, could you provide the steps you did follow and mention any options you may have used that they suggest?

This will help me to see how the unhelpful API state is happening, but for actually troubleshooting your 404, you might want to check with the openstack mailing list or on the #openstack channel on irc.freenode.org.

Thanks!

Changed in nova:
status:	New → Incomplete
summary:	- Unexpected API Error creating instance + Unexpected API Error creating instance with Neutron networking
summary:	- Unexpected API Error creating instance with Neutron networking + Unexpected Exception in API Error when creating instance with Neutron

Revision history for this message

Matthew (2-uluntu-r) wrote on 2016-03-30:

1. I am indeed using Liberty.
2. Steps to reproduce: a: log into Horizon as 'admin' user. b: expand Project -> Compute in the left navbar. c: click Instances d: click Launch Instance button. d: Choose availability zone 'nova', give an instance name, select a flavor ('m1.tiny' in my case), instance count 1, boot source of Boot from Image, select the 'cirros' image that is installed in the build guide. e: Click on Networking tab, add 'Public' to selected networks. (Problem also happens with 'Private' network). f: click Launch. The spinny icon appears for ~3 seconds, then two red "Error" popups appear in upper right corner.

I believe that I followed the installation guide faithfully. One interesting facet of my installation is that the Management network is also the Public network. I don't know if that's related to the problem or not.

I'm afraid this information might not be valuable to you. Is there an easy way for me to capture my OpenStack configuration and upload it to this bug report? Which config files are likely to be the most valuable in troubleshooting?

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-04-05:

There is a misconfiguration with neutron, the endpoint isn't valid.

This is really a support request, not a bug report, so I'd mark this invalid. You should post stuff like this to ask.openstack.org or the openstack general mailing list.

See the [neutron] section in the config docs:

http://docs.openstack.org/liberty/config-reference/content/list-of-compute-config-options.html

Your 'url' value is probably wrong.

Here is an example from a neutron CI job run in stable/liberty:

http://logs.openstack.org/10/285710/1/check/gate-tempest-dsvm-neutron-full/6ec0d82/logs/etc/nova/nova.conf.txt.gz

[neutron]
service_metadata_proxy = True
url = http://127.0.0.1:9696
region_name = RegionOne
auth_strategy = keystone
project_domain_name = Default
project_name = service
user_domain_name = Default
password = secretservice
username = neutron
auth_url = http://127.0.0.1:35357/v3
auth_plugin = v3password