Network related cleanup might fail on instance deletion

[François Charlier]

When an instance is deleted/destroyed, an async call is sent to nova-network while the nova-compute related deletion continues.

In various places, nova-network fetches data from the db without checking on deleted instances and floating ips, fixed ips, security groups, … might then not be deallocated.

Relevant information from nova-network.log :

2012-02-21 18:57:04,303 DEBUG nova.network.manager [req-7dc512a4-a895-41ea-86e1-ff8ce8f27901 dd5093c768c24ed391955b1dde5cf411 08972d4e0424497483de1c0a5123ea9b] floating IP deallocation for instance |86| from (pid=14288) debug /usr/lib/python2.7/dist-packages/nova/log.py:175
2012-02-21 18:57:04,359 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 244, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 200, in wrapped
(nova.rpc): TRACE: return func(self, context, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 304, in deallocate_for_instance
(nova.rpc): TRACE: affect_auto_assigned=True)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 200, in wrapped
(nova.rpc): TRACE: return func(self, context, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 464, in disassociate_floating_ip
(nova.rpc): TRACE: instance = self.db.instance_get(context, fixed_ip['instance_id'])
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 580, in instance_get
(nova.rpc): TRACE: return IMPL.instance_get(context, instance_id)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 119, in wrapper
(nova.rpc): TRACE: return f(*args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1457, in instance_get
(nova.rpc): TRACE: raise exception.InstanceNotFound(instance_id=instance_id)
(nova.rpc): TRACE: InstanceNotFound: Instance 86 could not be found.
(nova.rpc): TRACE:

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/4449

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/4449
Committed: http://github.com/openstack/nova/commit/5ad1dea4daf264fe11429288597d26dc4974fe05
Submitter: Jenkins
Branch: master

commit 5ad1dea4daf264fe11429288597d26dc4974fe05
Author: François Charlier <email address hidden>
Date: Thu Feb 23 16:32:54 2012 +0100

    nova-network can't deallocate ips from deleted instances

    Allow nova-network to deallocate assigned IPs from deleted instances

    Fix bug #939580

    Change-Id: Ib24fea0701cf640721cd0b021c78c8c5dfbb507b
    Signed-off-by: François Charlier <email address hidden>

[Mathew Odden (locke105)]

I am still running into this issue in 2012.1

I verified that the proposed code changes are in my nova/network/manage.py file as well.

I don't even see a log message from nova-network about trying to deallocate the IPs when an instance is deleted.

[Mathew Odden (locke105)]

Alright so I'm not sure if this is the same issue at all, but here is my steps to recreate what my problem is.

1. Create instance
2. Assign floating ip from any pool
3. Delete instance from nova client or Dashboard
4. Try to access the Access & Security tab or do a nova floatin-ip-list
5. Get a " ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) "

To fix this I have to go into the database and disassociate the floating ip and fixed IP from the instance ID

[Msekni (msekni)]

i am having this issue too even after the fix !!

[Christian Parpart (trapni)]

how can I officially fix this issue? I just ran into the same issue as in #4, however, I did not know what exactly was meant by "disassociating", in fact, I deleted my floating IP instead.

 best would be to auto-detach the floating-IP on instance termination.

[Edmund Haselwanter (ehaselwanter)]

does this correspond to the bug in https://bugs.launchpad.net/nova/essex/+bug/997763 ?