nova API returns 404 for floating-ip-list if an instance has disappeared

[Johnny Boy]

I'm not exactly sure how to reproduce the problem.
In my case, one (out of two) compute nodes crashed, leaving a few instances listed which needed replacing.

First the old instances was deleted, but it's still hanging around in the list since that compute node was crashed.
Then released the floating ip via the horizon interface, and assigned it via nova client.

While deleting and re-assigning floating ip's to new instances, something got set wrong. Which now result in floating ip's being broken for that tenant.

```
nova --os-tenant-name API floating-ip-list
ERROR: The resource could not be found. (HTTP 404) (Request-ID: req-9c6165ab-6312-4b98-8cf7-9ebc5d067e7c)
```

The log from nova-api shows:

```
2013-01-22 10:20:21 ERROR nova.api.openstack [req-9c6165ab-6312-4b98-8cf7-9ebc5d067e7c 4629023da94042e5ab9b1fab79ddc2ce ab3f2999ea354cb2a1b8cd7d3066abf0] Caught error: Instance 10502b84-3d5d-459a-ab97-de762ac48a49 could not be found.
2013-01-22 10:20:21 TRACE nova.api.openstack Traceback (most recent call last):
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 78, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack return req.get_response(self.application)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
2013-01-22 10:20:21 TRACE nova.api.openstack application, catch_exc_info=False)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
2013-01-22 10:20:21 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack return resp(environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 278, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack return self.app(env, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack return resp(environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack return resp(environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack return resp(environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack response = self.app(environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack return resp(environ, start_response)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
2013-01-22 10:20:21 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 885, in __call__
2013-01-22 10:20:21 TRACE nova.api.openstack content_type, body, accept)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 933, in _process_stack
2013-01-22 10:20:21 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1021, in dispatch
2013-01-22 10:20:21 TRACE nova.api.openstack return method(req=request, **action_args)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 158, in index
2013-01-22 10:20:21 TRACE nova.api.openstack self._set_metadata(context, floating_ip)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 130, in _set_metadata
2013-01-22 10:20:21 TRACE nova.api.openstack instance_uuid)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 118, in _get_instance
2013-01-22 10:20:21 TRACE nova.api.openstack return self.compute_api.get(context, instance_id)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1018, in get
2013-01-22 10:20:21 TRACE nova.api.openstack instance = self.db.instance_get_by_uuid(context, instance_id)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 576, in instance_get_by_uuid
2013-01-22 10:20:21 TRACE nova.api.openstack return IMPL.instance_get_by_uuid(context, uuid)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 129, in wrapper
2013-01-22 10:20:21 TRACE nova.api.openstack return f(*args, **kwargs)
2013-01-22 10:20:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1501, in instance_get_by_uuid
2013-01-22 10:20:21 TRACE nova.api.openstack raise exception.InstanceNotFound(instance_id=uuid)
2013-01-22 10:20:21 TRACE nova.api.openstack InstanceNotFound: Instance 10502b84-3d5d-459a-ab97-de762ac48a49 could not be found.
2013-01-22 10:20:21 TRACE nova.api.openstack
2013-01-22 10:20:21 INFO nova.api.openstack [req-9c6165ab-6312-4b98-8cf7-9ebc5d067e7c 4629023da94042e5ab9b1fab79ddc2ce ab3f2999ea354cb2a1b8cd7d3066abf0] http://10.0.1.203:8774/v2/ab3f2999ea354cb2a1b8cd7d3066abf0/os-floating-ips returned with HTTP 404
2013-01-22 10:20:21 INFO nova.osapi_compute.wsgi.server [req-9c6165ab-6312-4b98-8cf7-9ebc5d067e7c 4629023da94042e5ab9b1fab79ddc2ce ab3f2999ea354cb2a1b8cd7d3066abf0] 10.0.1.50 - - [22/Jan/2013 10:20:21] "GET /v2/ab3f2999ea354cb2a1b8cd7d3066abf0/os-floating-ips HTTP/1.1" 404 272 0.534521
```

Maybe floating ip's assigned to non-existing instances can be reclaimed somehow without poking in the database?

[Johnny Boy (balboah)]

So these are also broken, returning 404:

nova --os-tenant-name API remove-floating-ip 10502b84-3d5d-459a-ab97-de762ac48a49 10.101.100.22

nova --os-tenant-name API floating-ip-delete 10.101.100.22

[Johnny Boy (balboah)]

This made it work again:

nova-manage floating delete --ip_range=10.101.100.22
nova-manage floating create --ip_range=10.101.100.22

[Igor Savchenko (igor-scalr)]

I was able to reproduce this issue. This happens when:
1. Associate floating IP with running instance.
2. Terminate instances from step 2.
3. list floating ips.

Seems like when you terminating the instances, floating ip not being released and still links to the terminated server.

Traceback attached.

[Vish Ishaya (vishvananda)]

What version is this? I believe this is working in Trunk and stable/folsom

[Michael Still (mikal)]

Hi. Any chance you can let us know what version you experienced this problem with? Thanks!

[Johnny Boy (balboah)]

This was with Folsom on ubuntu 12.04 using the ubuntu packages from http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/folsom.
Not sure which minor version it was as I just upgraded to the latest

[Igor Savchenko (igor-scalr)]

In my case it was: Folsom 2012.2.2

[ashwini (ashwini-mhatre)]

I tried to reproduce this bug but in my case the floating ip's I am assigning to server are getting released as soon as I delete the VM.
So can anyone suggest some other way to reproduce this bug?

[Sean Dague (sdague)]

Honestly, I'm pretty sure we managed to kill this bug. Please reopen if you see it still crop up.

[Chuck Burkett (cburkett)]

FYI for anyone who stumbles upon this page after searching for the error message like I did...

We ran into this issue in one of our environments where we're still running an old version of nova (2012.1.3). Normally this version takes care of releasing a floating IP when its associated instance is terminated, but in our case, the compute node running the instance crashed.

The solution was to manually release the fixed and floating IPs in the database.