Floating IP addresses leak if you delete an instance with one assigned

[Ian Wells]

(This is with the stock Ubuntu Essex packages)

I've been running VMs by hand on my test machine. I've been assigning IP addresses from the floating pool:

nova floating-ip-create

nova add-floating-ip <machine> <ip>

However, because I'm testing by hand at the moment, sometimes I've been killing the machines without removing IP addresses first:

nova delete <machine>

I've been doing this for some time and apparently I've exhausted my limited pool of available floating IPs. I am now in the situation where I can't assign a floating IP because they're all inaccessible:

$ nova floating-ip-create
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
$ nova add-floating-ip <machine> 172.29.50.72
ERROR: Associate floating ip failed (HTTP 500)

(Obviously I can't remove a floating IP because the machine it's assigned to has been deleted by this point)

$ nova floating-ip-delete 172.29.50.72
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

Relavant traces from the nova-api log:

(floating-ip-create):

2012-08-11 10:14:21 INFO nova.api.openstack.wsgi [req-0c047374-917f-4411-819e-bdd48b6bc6a0 b2cb7d0476974261af316616c4861ddb 6c949cf379d14971b20ec3f9a42fabb0] POST http://172.29.50.151:8774/v2/6c949cf379d14971b20ec3f9a42fabb0/os-floating-ips
2012-08-11 10:14:21 ERROR nova.api.openstack [req-0c047374-917f-4411-819e-bdd48b6bc6a0 b2cb7d0476974261af316616c4861ddb 6c949cf379d14971b20ec3f9a42fabb0] Caught error: Remote error: QuotaError Quota exceeded: code=AddressLimitExceeded
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 253, in _process_data\n rval = node_func(context=ctxt, **node_args)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 257, in wrapped\n return func(self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 403, in allocate_floating_ip\n raise exception.QuotaError(code=\'AddressLimitExceeded\')\n', u'QuotaError: Quota exceeded: code=AddressLimitExceeded\n'].
2012-08-11 10:14:21 TRACE nova.api.openstack Traceback (most recent call last):
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 41, in __call__
2012-08-11 10:14:21 TRACE nova.api.openstack return req.get_response(self.application)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
2012-08-11 10:14:21 TRACE nova.api.openstack application, catch_exc_info=False)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
2012-08-11 10:14:21 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 176, in __call__
2012-08-11 10:14:21 TRACE nova.api.openstack return self.app(env, start_response)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-08-11 10:14:21 TRACE nova.api.openstack return resp(environ, start_response)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-08-11 10:14:21 TRACE nova.api.openstack return resp(environ, start_response)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2012-08-11 10:14:21 TRACE nova.api.openstack response = self.app(environ, start_response)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-08-11 10:14:21 TRACE nova.api.openstack return resp(environ, start_response)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
2012-08-11 10:14:21 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
2012-08-11 10:14:21 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 803, in __call__
2012-08-11 10:14:21 TRACE nova.api.openstack content_type, body, accept)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 851, in _process_stack
2012-08-11 10:14:21 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 926, in dispatch
2012-08-11 10:14:21 TRACE nova.api.openstack return method(req=request, **action_args)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 153, in create
2012-08-11 10:14:21 TRACE nova.api.openstack address = self.network_api.allocate_floating_ip(context, pool)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 130, in allocate_floating_ip
2012-08-11 10:14:21 TRACE nova.api.openstack 'pool': pool}})
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 68, in call
2012-08-11 10:14:21 TRACE nova.api.openstack return _get_impl().call(context, topic, msg, timeout)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 674, in call
2012-08-11 10:14:21 TRACE nova.api.openstack return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 343, in call
2012-08-11 10:14:21 TRACE nova.api.openstack rv = list(rv)
2012-08-11 10:14:21 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 311, in __iter__
2012-08-11 10:14:21 TRACE nova.api.openstack raise result
2012-08-11 10:14:21 TRACE nova.api.openstack RemoteError: Remote error: QuotaError Quota exceeded: code=AddressLimitExceeded
2012-08-11 10:14:21 TRACE nova.api.openstack [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 253, in _process_data\n rval = node_func(context=ctxt, **node_args)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 257, in wrapped\n return func(self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 403, in allocate_floating_ip\n raise exception.QuotaError(code=\'AddressLimitExceeded\')\n', u'QuotaError: Quota exceeded: code=AddressLimitExceeded\n'].
2012-08-11 10:14:21 TRACE nova.api.openstack
2012-08-11 10:14:21 INFO nova.api.openstack [req-0c047374-917f-4411-819e-bdd48b6bc6a0 b2cb7d0476974261af316616c4861ddb 6c949cf379d14971b20ec3f9a42fabb0] http://172.29.50.151:8774/v2/6c949cf379d14971b20ec3f9a42fabb0/os-floating-ips returned with HTTP 500

(floating-ip-delete):

2012-08-11 10:15:50 INFO nova.api.openstack.wsgi [req-5c00329c-b13c-4dc3-b301-87aa105e9a9c b2cb7d0476974261af316616c4861ddb 6c949cf379d14971b20ec3f9a42fabb0] GET http://172.29.50.151:8774/v2/6c949cf379d14971b20ec3f9a42fabb0/os-floating-ips
2012-08-11 10:15:50 ERROR nova.api.openstack [req-5c00329c-b13c-4dc3-b301-87aa105e9a9c b2cb7d0476974261af316616c4861ddb 6c949cf379d14971b20ec3f9a42fabb0] Caught error: Instance 5 could not be found.
2012-08-11 10:15:50 TRACE nova.api.openstack Traceback (most recent call last):
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 41, in __call__
2012-08-11 10:15:50 TRACE nova.api.openstack return req.get_response(self.application)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
2012-08-11 10:15:50 TRACE nova.api.openstack application, catch_exc_info=False)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
2012-08-11 10:15:50 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 176, in __call__
2012-08-11 10:15:50 TRACE nova.api.openstack return self.app(env, start_response)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-08-11 10:15:50 TRACE nova.api.openstack return resp(environ, start_response)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-08-11 10:15:50 TRACE nova.api.openstack return resp(environ, start_response)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2012-08-11 10:15:50 TRACE nova.api.openstack response = self.app(environ, start_response)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-08-11 10:15:50 TRACE nova.api.openstack return resp(environ, start_response)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
2012-08-11 10:15:50 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
2012-08-11 10:15:50 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 803, in __call__
2012-08-11 10:15:50 TRACE nova.api.openstack content_type, body, accept)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 851, in _process_stack
2012-08-11 10:15:50 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 926, in dispatch
2012-08-11 10:15:50 TRACE nova.api.openstack return method(req=request, **action_args)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 140, in index
2012-08-11 10:15:50 TRACE nova.api.openstack self._set_metadata(context, floating_ip)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 112, in _set_metadata
2012-08-11 10:15:50 TRACE nova.api.openstack instance_id)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 100, in _get_instance
2012-08-11 10:15:50 TRACE nova.api.openstack return self.compute_api.get(context, instance_id)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1029, in get
2012-08-11 10:15:50 TRACE nova.api.openstack instance = self.db.instance_get(context, instance_id)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 540, in instance_get
2012-08-11 10:15:50 TRACE nova.api.openstack return IMPL.instance_get(context, instance_id)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 120, in wrapper
2012-08-11 10:15:50 TRACE nova.api.openstack return f(*args, **kwargs)
2012-08-11 10:15:50 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1339, in instance_get
2012-08-11 10:15:50 TRACE nova.api.openstack raise exception.InstanceNotFound(instance_id=instance_id)
2012-08-11 10:15:50 TRACE nova.api.openstack InstanceNotFound: Instance 5 could not be found.
2012-08-11 10:15:50 TRACE nova.api.openstack
2012-08-11 10:15:50 INFO nova.api.openstack [req-5c00329c-b13c-4dc3-b301-87aa105e9a9c b2cb7d0476974261af316616c4861ddb 6c949cf379d14971b20ec3f9a42fabb0] http://172.29.50.151:8774/v2/6c949cf379d14971b20ec3f9a42fabb0/os-floating-ips returned with HTTP 500

(add-floating-ip):

2012-08-11 10:16:26 INFO nova.api.openstack.wsgi [req-b23b9d8a-85a1-4864-9cb2-8fbe567d7f42 b2cb7d0476974261af316616c4861ddb 6c949cf379d14971b20ec3f9a42fabb0] GET http://172.29.50.151:8774/v2/6c949cf379d14971b20ec3f9a42fabb0/servers/detail
2012-08-11 10:16:26 INFO nova.api.openstack.wsgi [req-b23b9d8a-85a1-4864-9cb2-8fbe567d7f42 b2cb7d0476974261af316616c4861ddb 6c949cf379d14971b20ec3f9a42fabb0] http://172.29.50.151:8774/v2/6c949cf379d14971b20ec3f9a42fabb0/servers/detail returned with HTTP 200
2012-08-11 10:16:27 INFO nova.api.openstack.wsgi [req-e56ce5ef-3c05-45e7-a288-37a73ba48e49 b2cb7d0476974261af316616c4861ddb 6c949cf379d14971b20ec3f9a42fabb0] POST http://172.29.50.151:8774/v2/6c949cf379d14971b20ec3f9a42fabb0/servers/bbd48232-e56e-43be-8a65-cf38e6fc28df/action 2012-08-11 10:16:27 INFO nova.api.openstack.wsgi [req-e56ce5ef-3c05-45e7-a288-37a73ba48e49 b2cb7d0476974261af316616c4861ddb 6c949cf379d14971b20ec3f9a42fabb0] HTTP exception thrown: Associate floating ip failed2012-08-11 10:16:27 INFO nova.api.openstack.wsgi [req-e56ce5ef-3c05-45e7-a288-37a73ba48e49 b2cb7d0476974261af316616c4861ddb 6c949cf379d14971b20ec3f9a42fabb0] http://172.29.50.151:8774/v2/6c949cf379d14971b20ec3f9a42fabb0/servers/bbd48232-e56e-43be-8a65-cf38e6fc28df/action returned with HTTP 500

[Ian Wells (ijw-ubuntu)]

Quick analysis:

- floating-ip-create is failing with a quota violation, which is totally reasonable, but that error turns into a nondescript error before coming out of the CLI

- floating-ip-delete won't delete an IP from a nonexistent instance

I think IPs ought to be freed automatically on instance death, but I've found no documentation to suggest that there's any behaviour specified for this case.

[Mark McLoughlin (markmc)]

I've just tried reproducing this with latest upstream folsom and can't

Moving to Ubuntu. If someone can confirm that it's an issue with latest stable/essex upstream, please do nominate it for the essex series

Thanks for the report!

[Adam Gandelman (gandelman-a)]

Having trouble reproducing using the following steps with stable/essex on Ubuntu 12.04 (via 2012.1.3+stable-20120827-4d2a4afe-0ubuntu1 package):

nova floating-ip-create
nova add-floating-ip b8e0295d-e515-4838-b91a-eb3c4b1beebe 192.168.25.237
nova delete b8e0295d-e515-4838-b91a-eb3c4b1beebe
nova floating-ip-delete 192.168.25.237

Watching the database (floating_ips), the floating ip to fixed_ip_id mapping is appropriately removed on instance deletion, as is the floating ip <-> project_id mapping on floating-ip-delete.

Curious to know which network manager you're using.

[Ian Wells (ijw-ubuntu)]

I was testing with devstack, and the latest Folsom of the moment. If it's not turning up in either case at the moment then apparently it's gone away - I'll reopen with more detail if I see it again.