--auto_assign_floating_ip=true crashes on logout

[Serguei Koubli]

In auto assign floating ip mode (--auto_assign_floating_ip=true)
run instance:
# euca-run-instances ami-00000003 --kernel aki-00000001 --ramdisk ari-00000002 -k rhelkey -t m1.tiny
RESERVATION r-ck27rzam rhelimg default
INSTANCE i-00000001 ami-00000003 scheduling rhelkey (rhelimg, None) 0 m1.tiny 2011-08-26T11:25:47Z unknown zone

# cat /var/log/nova/nova-network.log | grep TRACE
2011-08-26 15:21:48,006 nova: logging_exception_prefix : (%(name)s): TRACE:
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/lib/python2.6/site-packages/nova/rpc.py", line 232, in _process_data
(nova): TRACE: rval = node_func(context=ctxt, **node_args)
(nova): TRACE: File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 231, in allocate_for_instance
(nova): TRACE: affect_auto_assigned=True)
(nova): TRACE: File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 96, in associate_floating_ip
(nova): TRACE: floating_ip = self.db.floating_ip_get_by_address(context, floating_ip)
(nova): TRACE: File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 287, in floating_ip_get_by_address
(nova): TRACE: return IMPL.floating_ip_get_by_address(context, address)
(nova): TRACE: File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 115, in wrapper
(nova): TRACE: return f(*args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 636, in floating_ip_get_by_address
(nova): TRACE: raise exception.FloatingIpNotFoundForAddress(address=address)
(nova): TRACE: FloatingIpNotFoundForAddress: Floating ip not found for address <nova.db.sqlalchemy.models.FloatingIp object at 0x35e7590>.
(nova): TRACE:

[Serguei Koubli (skoubli)]

I deleted the previous branch related with lp834633 opened new one but I can't commit changes

[ lp834633]# bzr commit --fixes lp834633 -m '--auto_assign_floating_ip=true crashes on logout in OpenStack Compute (nova)'
bzr: ERROR: Invalid bug lp834633. Must be in the form of 'tracker:id'. See "bzr help bugs" for more information on this feature.
Commit refused.

[Serguei Koubli (skoubli)]

Sorry
I find the problem

[Kiall Mac Innes (kiall)]

I can verify this path fixes the exception while starting an instance.

But - There is another bug preventing the address from being disassociated + released correctly on termination, which in turn leaves the address attached to a non-existent instance.

2011-10-10 15:58:21,785 DEBUG nova.network.manager [5220cafc-9da7-4000-a164-2089b2ec1079 kiall managedit] floating IP deallocation for instance |49| from (pid=1105) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:251
2011-10-10 15:58:21,833 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 265, in deallocate_for_instance
(nova.rpc): TRACE: True)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 68, in release_floating_ip
(nova.rpc): TRACE: raise exception.ApiError(_('Floating ip is in use. '
(nova.rpc): TRACE: ApiError: Floating ip is in use. Disassociate it before releasing.

[Kiall Mac Innes (kiall)]

Looks like this area of the code has got a cleanup already .. https://review.openstack.org/628

And - it includes a fix for the original issue..

[Nathan Koterba (nkoterba)]

I just installed Openstack Diablo on a new oneiric installation using the packages from apt-get and I'm still seeing this issue.

Are the changes listed in the link above in the latest packages? It doesn't look like it.

[Kiall Mac Innes (kiall)]

The review listed above won't be merged into diablo (it has loads of essex changes mixed in), and the launchpad branch is only half the fix..

I've just submitted the the minimal changes needed to fix this for review https://review.openstack.org/1328

Hopefully this will get approved and merged in...

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/1328
Committed: http://github.com/openstack/nova/commit/80d73a2df2c33212217c59783f62d74673776ebb
Submitter: Jenkins
Branch: stable/diablo

 status fixcommitted
 done

commit 80d73a2df2c33212217c59783f62d74673776ebb
Author: Kiall Mac Innes <email address hidden>
Date: Fri Nov 4 19:43:50 2011 +0000

    Fixes bug 834633 - Auto assigning floating IPs

    Details:

     - This fix is specific to the auto_assign_floating_ip=true feature, which was new in diablo[1] and looks like it never actually worked.
     - Trey's refactoring on master[2] fixed bug #834633 where a FloatingIp object was passed to network_api.associate_floating_ip() instead of an address string.
     - In bug #834633, Serguei Koubli had previously (i.e. before Trey's refactoring) proposed a much more isolated fix for this issue.
     - This commit includes Serguei Koubli's fix, and, has also fixed a seperate issue where auto-assigned flating IPs could not be deallocated. (This fix was also included in Trey's refactoring)

    [1] https://blueprints.launchpad.net/nova/+spec/floating-ip-auto-assignment
    [2] https://review.openstack.org/628

    Change-Id: Ie439c6743f066f8504ef88de861dee8a7762940d

[Thierry Carrez (ttx)]

This is not fixed in essex-1, only in stable/diablo ?

[Thierry Carrez (ttx)]

It's actually fixed by network refactor in essex already. Sorry for confusion.

[Martin Pitt (pitti)]

Hello Serguei, or anyone else affected,

Accepted nova into oneiric-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Kevin Jackson (kevin-linuxservices)]

Hi Martin,
I've managed to update my packages from the proposed archive and confirm that this is working as expected now.

Cheers,

Kev