OpenStack Compute (nova)

500 error and 2 traces if no free fixed IP left in the neutron network

Bug #1424594 reported by George Shuklin on 2015-02-23

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	Low	Anand Shanmugam	OpenStack Compute (nova) 2015.1.0 "kilo"

Bug Description

If nova recieve 404 from neutron due lack of free fixed IPs, it traces badly and return 500 error to user.

Steps to reproduce:
0. Setup nova & neutron, create network & subnetwork
1. Consume all IP from that network
2. Try to attach interface to that network (nova interface-attach --net-id NET-UUID SERVER-UUID)

Actual behaviour:

ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-994444ec-6a69-428d-9c16-c58d685553dd)

... and traces (see below)

Expected behaviour:

Proper complain about lack of IP (NoMoreFixedIps) and proper HTTP error code.

Traces (nova-api):

nova.api.openstack.wsgi[26783]: DEBUG Action: 'create', calling method: <bound method InterfaceAttachmentController.create of <nova.api.openstack.compute.contrib.attach_interfaces.InterfaceAttachmentController object at 0x7f098e01b390>>, body: {"interfaceAttachment": {"net_id": "f3f2802a-c2a1-4d8b-9f43-cf24d0dc8233"}} [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57] _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:934
nova.api.openstack.compute.contrib.attach_interfaces[26783]: AUDIT [instance: 5f1e84cb-1766-45e1-899b-9de1e535309b] Attach interface [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57]
nova.api.openstack[26783]: ERROR Caught error: Zero fixed ips available.
Traceback (most recent call last):

File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
incoming.message))

File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
return self._do_dispatch(endpoint, method, ctxt, args)

File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
result = getattr(endpoint, method)(ctxt, **new_args)

File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 419, in decorated_function
return function(self, context, *args, **kwargs)

File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
payload)

File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
six.reraise(self.type_, self.value, self.tb)

File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
return f(self, context, *args, **kw)

File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
pass

File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
six.reraise(self.type_, self.value, self.tb)

File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 289, in decorated_function
return function(self, context, *args, **kwargs)

File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 331, in decorated_function
kwargs['instance'], e, sys.exc_info())

File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
six.reraise(self.type_, self.value, self.tb)

File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function
return function(self, context, *args, **kwargs)

File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4787, in attach_interface
context, instance, port_id, network_id, requested_ip)

File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 569, in allocate_port_for_instance
requested_networks=requested_networks)

File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 443, in allocate_for_instance
self._delete_ports(neutron, instance, created_port_ids)

File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
six.reraise(self.type_, self.value, self.tb)

File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 423, in allocate_for_instance
security_group_ids, available_macs, dhcp_opts)

File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 226, in _create_port
raise exception.NoMoreFixedIps()

NoMoreFixedIps: Zero fixed ips available.
[req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57]
nova.api.openstack[26783]: TRACE Traceback (most recent call last):
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
nova.api.openstack[26783]: TRACE return req.get_response(self.application)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
nova.api.openstack[26783]: TRACE application, catch_exc_info=False)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
nova.api.openstack[26783]: TRACE app_iter = application(self.environ, start_response)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26783]: TRACE return resp(environ, start_response)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
nova.api.openstack[26783]: TRACE return self._app(env, start_response)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26783]: TRACE return resp(environ, start_response)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26783]: TRACE return resp(environ, start_response)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
nova.api.openstack[26783]: TRACE response = self.app(environ, start_response)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26783]: TRACE return resp(environ, start_response)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
nova.api.openstack[26783]: TRACE resp = self.call_func(req, *args, **self.kwargs)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
nova.api.openstack[26783]: TRACE return self.func(req, *args, **kwargs)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
nova.api.openstack[26783]: TRACE content_type, body, accept)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
nova.api.openstack[26783]: TRACE action_result = self.dispatch(meth, request, action_args)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
nova.api.openstack[26783]: TRACE return method(req=request, **action_args)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/attach_interfaces.py", line 109, in create
nova.api.openstack[26783]: TRACE instance, network_id, port_id, req_ip)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 220, in wrapped
nova.api.openstack[26783]: TRACE return func(self, context, target, *args, **kwargs)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 210, in inner
nova.api.openstack[26783]: TRACE return function(self, context, instance, *args, **kwargs)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 191, in inner
nova.api.openstack[26783]: TRACE return f(self, context, instance, *args, **kw)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3010, in attach_interface
nova.api.openstack[26783]: TRACE requested_ip=requested_ip)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 335, in attach_interface
nova.api.openstack[26783]: TRACE port_id=port_id, requested_ip=requested_ip)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
nova.api.openstack[26783]: TRACE retry=self.retry)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
nova.api.openstack[26783]: TRACE timeout=timeout, retry=retry)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
nova.api.openstack[26783]: TRACE retry=retry)
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 399, in _send
nova.api.openstack[26783]: TRACE raise result
nova.api.openstack[26783]: TRACE NoMoreFixedIps_Remote: Zero fixed ips available.
nova.api.openstack[26783]: TRACE Traceback (most recent call last):
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
nova.api.openstack[26783]: TRACE incoming.message))
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
nova.api.openstack[26783]: TRACE return self._do_dispatch(endpoint, method, ctxt, args)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
nova.api.openstack[26783]: TRACE result = getattr(endpoint, method)(ctxt, **new_args)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 419, in decorated_function
nova.api.openstack[26783]: TRACE return function(self, context, *args, **kwargs)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
nova.api.openstack[26783]: TRACE payload)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
nova.api.openstack[26783]: TRACE six.reraise(self.type_, self.value, self.tb)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
nova.api.openstack[26783]: TRACE return f(self, context, *args, **kw)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
nova.api.openstack[26783]: TRACE pass
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
nova.api.openstack[26783]: TRACE six.reraise(self.type_, self.value, self.tb)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 289, in decorated_function
nova.api.openstack[26783]: TRACE return function(self, context, *args, **kwargs)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 331, in decorated_function
nova.api.openstack[26783]: TRACE kwargs['instance'], e, sys.exc_info())
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
nova.api.openstack[26783]: TRACE six.reraise(self.type_, self.value, self.tb)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function
nova.api.openstack[26783]: TRACE return function(self, context, *args, **kwargs)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4787, in attach_interface
nova.api.openstack[26783]: TRACE context, instance, port_id, network_id, requested_ip)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 569, in allocate_port_for_instance
nova.api.openstack[26783]: TRACE requested_networks=requested_networks)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 443, in allocate_for_instance
nova.api.openstack[26783]: TRACE self._delete_ports(neutron, instance, created_port_ids)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
nova.api.openstack[26783]: TRACE six.reraise(self.type_, self.value, self.tb)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 423, in allocate_for_instance
nova.api.openstack[26783]: TRACE security_group_ids, available_macs, dhcp_opts)
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 226, in _create_port
nova.api.openstack[26783]: TRACE raise exception.NoMoreFixedIps()
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE NoMoreFixedIps: Zero fixed ips available.
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: TRACE
nova.api.openstack[26783]: INFO http://isokolov1.compute.dev.internal:8774/v2/78ea359977584bcc9feceef2553dbe57/servers/5f1e84cb-1766-45e1-899b-9de1e535309b/os-interface returned with HTTP 500 [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57]
nova.api.openstack.wsgi[26783]: DEBUG Returning 500 to user: The server has either erred or is incapable of performing the requested operation. [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57] __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1201
nova.osapi_compute.wsgi.server[26783]: INFO 192.168.32.70 "POST /v2/78ea359977584bcc9feceef2553dbe57/servers/5f1e84cb-1766-45e1-899b-9de1e535309b/os-interface HTTP/1.1" status: 500 len: 335 time: 0.5942502 [req-57f4e821-a968-48cd-8358-f73fa16b4ff7 4aac5cb61b1741b2a32067619555ecc1 78ea359977584bcc9feceef2553dbe57]
nova.api.openstack.wsgi[26782]: DEBUG Calling method '<bound method Controller.detail of <nova.api.openstack.compute.servers.Controller object at 0x7f098ebe6610>>' [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:937
nova.compute.api[26782]: DEBUG Searching by: {'deleted': False} [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1920
neutronclient.client[26782]: DEBUG
REQ: curl -i http://neutron-server.p.isokolov1.compute.serverscom.dev:29696/v2.0/ports.json?device_id=3ede6afc-7695-414c-bff0-4b807fe4644c&device_id=bf4abf26-b937-4f2d-8c29-6e69ad4be28c&device_id=375cd096-8022-4721-bcc9-5d65f1d21865&device_id=e8378206-ce54-45c3-976c-d3a38a8c619d&device_id=5f1e84cb-1766-45e1-899b-9de1e535309b -X GET -H "X-Auth-Token: bcc1f6eff61a434d9165794c082e7102" -H "User-Agent: python-neutronclient"
[req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_req /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:138
neutronclient.client[26782]: DEBUG RESP:200 CaseInsensitiveDict({'date': 'Mon, 23 Feb 2015 11:07:39 GMT', 'content-length': '2182', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-809c50d1-d35c-4b0e-88c6-f9730d540216'}) {"ports": [{"status": "ACTIVE", "binding:host_id": "compute1", "allowed_address_pairs": [], "extra_dhcp_opts": [], "device_owner": "compute:nova", "binding:profile": {}, "fixed_ips": [{"subnet_id": "a4180ba0-6681-4e07-8cc8-0617cccfd12e", "ip_address": "191.168.1.10"}], "id": "11f8456d-41f9-46e7-b6c8-5f6becc59a07", "security_groups": ["0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"], "device_id": "375cd096-8022-4721-bcc9-5d65f1d21865", "name": "", "admin_state_up": true, "network_id": "7ad70c6b-82f6-4723-be8e-d96047755f8e", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": true}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "mac_address": "fa:16:3e:9a:09:e0"}, {"status": "ACTIVE", "binding:host_id": "compute1", "allowed_address_pairs": [], "extra_dhcp_opts": [], "device_owner": "compute:nova", "binding:profile": {}, "fixed_ips": [{"subnet_id": "85412a00-c7da-472a-8cfe-3f66f911263a", "ip_address": "192.168.1.1"}], "id": "3e66374d-54d2-4efc-be86-0c2359c90c7c", "security_groups": ["0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"], "device_id": "3ede6afc-7695-414c-bff0-4b807fe4644c", "name": "", "admin_state_up": true, "network_id": "674ffba2-6393-4a77-85d2-cceab7960f28", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": true}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "mac_address": "fa:16:3e:8c:ef:3a"}, {"status": "ACTIVE", "binding:host_id": "compute1", "allowed_address_pairs": [], "extra_dhcp_opts": [], "device_owner": "compute:nova", "binding:profile": {}, "fixed_ips": [{"subnet_id": "50ebc948-752c-48a4-9825-77a43deb1363", "ip_address": "193.168.1.10"}], "id": "a9deee70-6a99-4d4f-8e0c-9ba05a0d61f3", "security_groups": ["0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"], "device_id": "bf4abf26-b937-4f2d-8c29-6e69ad4be28c", "name": "", "admin_state_up": true, "network_id": "f3f2802a-c2a1-4d8b-9f43-cf24d0dc8233", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": true}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "mac_address": "fa:16:3e:a5:10:3e"}]}
[req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_resp /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:147
neutronclient.client[26782]: DEBUG
REQ: curl -i http://neutron-server.p.isokolov1.compute.serverscom.dev:29696/v2.0/security-groups.json?id=0700a947-91aa-49a2-8cd6-c1e2abd8d8c4 -X GET -H "X-Auth-Token: bcc1f6eff61a434d9165794c082e7102" -H "User-Agent: python-neutronclient"
[req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_req /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:138
neutronclient.client[26782]: DEBUG RESP:200 CaseInsensitiveDict({'date': 'Mon, 23 Feb 2015 11:07:39 GMT', 'content-length': '1535', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-d048f970-115c-4ed1-a7d4-5c95d7a4642d'}) {"security_groups": [{"tenant_id": "b6749c7410594a01ae857f78d24884e3", "description": "default", "id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4", "security_group_rules": [{"remote_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4", "direction": "ingress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv4", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "port_range_max": null, "port_range_min": null, "id": "259b62a6-7283-41db-8653-6578418aec00", "security_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"}, {"remote_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4", "direction": "ingress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv6", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "port_range_max": null, "port_range_min": null, "id": "7b36cbe8-f616-4997-90b1-d0a684fff7f1", "security_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"}, {"remote_group_id": null, "direction": "egress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv4", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "port_range_max": null, "port_range_min": null, "id": "9e9aa8e9-5bf8-427d-8c65-27022d39355f", "security_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"}, {"remote_group_id": null, "direction": "egress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv6", "tenant_id": "b6749c7410594a01ae857f78d24884e3", "port_range_max": null, "port_range_min": null, "id": "dbd6a976-0cf1-48d3-a437-6bf4a5dbb0ab", "security_group_id": "0700a947-91aa-49a2-8cd6-c1e2abd8d8c4"}], "name": "default"}]}
[req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_resp /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:147
nova.osapi_compute.wsgi.server[26782]: INFO 192.168.32.70 "GET /v2/b80de3539fd2406ea4c0d7c827c4fdd7/servers/detail?all_tenants=True HTTP/1.1" status: 200 len: 9633 time: 0.3986661 [req-f51d4aa9-0b6b-4b84-a247-04e650055589 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7]
nova.api.openstack.wsgi[26781]: DEBUG Calling method '<bound method FloatingIPController.index of <nova.api.openstack.compute.contrib.floating_ips.FloatingIPController object at 0x7f098e02d890>>' [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:937
neutronclient.client[26781]: DEBUG
REQ: curl -i http://neutron-server.p.isokolov1.compute.serverscom.dev:29696/v2.0/floatingips.json?tenant_id=b80de3539fd2406ea4c0d7c827c4fdd7 -X GET -H "X-Auth-Token: e775fc86f0f4418999241bfc779bc397" -H "User-Agent: python-neutronclient"
[req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_req /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:138
neutronclient.client[26781]: DEBUG RESP:404 CaseInsensitiveDict({'date': 'Mon, 23 Feb 2015 11:07:40 GMT', 'content-length': '52', 'content-type': 'text/plain; charset=UTF-8', 'x-openstack-request-id': 'req-79af3c6f-c2d6-4899-af28-f31313a82dbe'}) 404 Not Found

The resource could not be found.

[req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] http_log_resp /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:147
neutronclient.v2_0.client[26781]: DEBUG Error message: 404 Not Found

The resource could not be found.

[req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] _handle_fault_response /usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:1208
nova.api.openstack[26781]: ERROR Caught error: 404 Not Found

The resource could not be found.

[req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7]
nova.api.openstack[26781]: TRACE Traceback (most recent call last):
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
nova.api.openstack[26781]: TRACE return req.get_response(self.application)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
nova.api.openstack[26781]: TRACE application, catch_exc_info=False)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
nova.api.openstack[26781]: TRACE app_iter = application(self.environ, start_response)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26781]: TRACE return resp(environ, start_response)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 661, in __call__
nova.api.openstack[26781]: TRACE return self._app(env, start_response)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26781]: TRACE return resp(environ, start_response)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26781]: TRACE return resp(environ, start_response)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
nova.api.openstack[26781]: TRACE response = self.app(environ, start_response)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
nova.api.openstack[26781]: TRACE return resp(environ, start_response)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
nova.api.openstack[26781]: TRACE resp = self.call_func(req, *args, **self.kwargs)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
nova.api.openstack[26781]: TRACE return self.func(req, *args, **kwargs)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
nova.api.openstack[26781]: TRACE content_type, body, accept)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
nova.api.openstack[26781]: TRACE action_result = self.dispatch(meth, request, action_args)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
nova.api.openstack[26781]: TRACE return method(req=request, **action_args)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 135, in index
nova.api.openstack[26781]: TRACE floating_ips = self.network_api.get_floating_ips_by_project(context)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1096, in get_floating_ips_by_project
nova.api.openstack[26781]: TRACE fips = client.list_floatingips(tenant_id=project_id)['floatingips']
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
nova.api.openstack[26781]: TRACE ret = self.function(instance, *args, **kwargs)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 444, in list_floatingips
nova.api.openstack[26781]: TRACE **_params)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1335, in list
nova.api.openstack[26781]: TRACE for r in self._pagination(collection, path, **params):
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1348, in _pagination
nova.api.openstack[26781]: TRACE res = self.get(path, params=params)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1321, in get
nova.api.openstack[26781]: TRACE headers=headers, params=params)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1298, in retry_request
nova.api.openstack[26781]: TRACE headers=headers, params=params)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1252, in do_request
nova.api.openstack[26781]: TRACE self._handle_fault_response(status_code, replybody)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1217, in _handle_fault_response
nova.api.openstack[26781]: TRACE exception_handler_v20(status_code, des_error_body)
nova.api.openstack[26781]: TRACE File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 80, in exception_handler_v20
nova.api.openstack[26781]: TRACE message=message)
nova.api.openstack[26781]: TRACE NeutronClientException: 404 Not Found
nova.api.openstack[26781]: TRACE
nova.api.openstack[26781]: TRACE The resource could not be found.
nova.api.openstack[26781]: TRACE
nova.api.openstack[26781]: TRACE
nova.api.openstack[26781]: TRACE
nova.api.openstack[26781]: INFO http://isokolov1.compute.dev.internal:8774/v2/b80de3539fd2406ea4c0d7c827c4fdd7/os-floating-ips returned with HTTP 500 [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7]
nova.api.openstack.wsgi[26781]: DEBUG Returning 500 to user: The server has either erred or is incapable of performing the requested operation. [req-0383f71c-5767-482b-9ee4-53c0b4ff6806 44cc5bff123b4fe0b06f6ac90418756d b80de3539fd2406ea4c0d7c827c4fdd7] __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1201

Tags:

Anand Shanmugam (anand1712) on 2015-02-26

Changed in nova:
assignee:	nobody → Anand Shanmugam (anand1712)

Anand Shanmugam (anand1712) on 2015-02-26

Changed in nova:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-26: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/159602

Sahid Orentino (sahid-ferdjaoui) on 2015-02-27

Changed in nova:
importance:	Undecided → Low

Revision history for this message

Matt Riedemann (mriedem) wrote on 2015-02-27:

Which release is this on? Juno? I see the problem in the kilo code but wondering how far back this needs to go since the line numbers don't match up with trunk.

tags:

added: api network neutron

Revision history for this message

George Shuklin (george-shuklin) wrote on 2015-02-27:

Juno at least. I'm running Ubnutu's juno (and lines can be shifted a bit due few harmless patches in other functions).

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-28: Fix merged to nova (master)

Reviewed: https://review.openstack.org/159602
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=82bf96fb0b3997680069dc2b1f772a46964cb9a9
Submitter: Jenkins
Branch: master

commit 82bf96fb0b3997680069dc2b1f772a46964cb9a9
Author: Anand Shanmugam <email address hidden>
Date: Fri Feb 27 02:06:57 2015 +0530

Fixes 500 error message and traces when no free ip is left

    When trying to attach a network to an instance if there are
    no free fixed ips available nova-api presently throws a 500
    internal server error.This is because the NoMoreFixedIps is
    not handled properly by the attach-interfaces code flow.This
    fix handles the exception and returns a bad request to the user.

Change-Id: I870177f51fa7546808c505a1f9476c195c3afaf2
Closes-Bug: #1424594

Changed in nova:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-03: Fix proposed to nova (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/160627

Revision history for this message

Anand Shanmugam (anand1712) wrote on 2015-03-03:

I have cherry picked this to the juno branch.The unit test case file is not there in juno.I haven't added this as it is a stable branch

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-09: Change abandoned on nova (stable/juno)

Change abandoned by Anand Shanmugam (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/160627
Reason: Abandoning Change as Stable branch api changes are not allowed

Revision history for this message

George Shuklin (george-shuklin) wrote on 2015-03-09:

Is that type of fix constitute a 'API change'?

Thierry Carrez (ttx) on 2015-03-20

Changed in nova:
milestone:	none → kilo-3
status:	Fix Committed → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-21:

Change abandoned by Anand Shanmugam (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/160627

Thierry Carrez (ttx) on 2015-04-30

Changed in nova:
milestone:	kilo-3 → 2015.1.0

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.