OpenStack Compute (Nova)

--auto_assign_floating_ip=true crashes on logout

Reported by Serguei Koubli on 2011-08-26
38
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Vish Ishaya
Diablo
Undecided
Unassigned

Bug Description

In auto assign floating ip mode (--auto_assign_floating_ip=true)
run instance:
# euca-run-instances ami-00000003 --kernel aki-00000001 --ramdisk ari-00000002 -k rhelkey -t m1.tiny
RESERVATION r-ck27rzam rhelimg default
INSTANCE i-00000001 ami-00000003 scheduling rhelkey (rhelimg, None) 0 m1.tiny 2011-08-26T11:25:47Z unknown zone

# cat /var/log/nova/nova-network.log | grep TRACE
2011-08-26 15:21:48,006 nova: logging_exception_prefix : (%(name)s): TRACE:
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/lib/python2.6/site-packages/nova/rpc.py", line 232, in _process_data
(nova): TRACE: rval = node_func(context=ctxt, **node_args)
(nova): TRACE: File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 231, in allocate_for_instance
(nova): TRACE: affect_auto_assigned=True)
(nova): TRACE: File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 96, in associate_floating_ip
(nova): TRACE: floating_ip = self.db.floating_ip_get_by_address(context, floating_ip)
(nova): TRACE: File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 287, in floating_ip_get_by_address
(nova): TRACE: return IMPL.floating_ip_get_by_address(context, address)
(nova): TRACE: File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 115, in wrapper
(nova): TRACE: return f(*args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 636, in floating_ip_get_by_address
(nova): TRACE: raise exception.FloatingIpNotFoundForAddress(address=address)
(nova): TRACE: FloatingIpNotFoundForAddress: Floating ip not found for address <nova.db.sqlalchemy.models.FloatingIp object at 0x35e7590>.
(nova): TRACE:

Thierry Carrez (ttx) on 2011-08-26
Changed in nova:
assignee: nobody → Serguei Koubli (skoubli)
importance: Undecided → Medium
status: New → In Progress
Serguei Koubli (skoubli) wrote :

I deleted the previous branch related with lp834633 opened new one but I can't commit changes

[ lp834633]# bzr commit --fixes lp834633 -m '--auto_assign_floating_ip=true crashes on logout in OpenStack Compute (nova)'
bzr: ERROR: Invalid bug lp834633. Must be in the form of 'tracker:id'. See "bzr help bugs" for more information on this feature.
Commit refused.

Serguei Koubli (skoubli) wrote :

Sorry
I find the problem

Kiall Mac Innes (kiall) wrote :

I can verify this path fixes the exception while starting an instance.

But - There is another bug preventing the address from being disassociated + released correctly on termination, which in turn leaves the address attached to a non-existent instance.

2011-10-10 15:58:21,785 DEBUG nova.network.manager [5220cafc-9da7-4000-a164-2089b2ec1079 kiall managedit] floating IP deallocation for instance |49| from (pid=1105) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:251
2011-10-10 15:58:21,833 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 265, in deallocate_for_instance
(nova.rpc): TRACE: True)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 68, in release_floating_ip
(nova.rpc): TRACE: raise exception.ApiError(_('Floating ip is in use. '
(nova.rpc): TRACE: ApiError: Floating ip is in use. Disassociate it before releasing.

Kiall Mac Innes (kiall) wrote :

Looks like this area of the code has got a cleanup already .. https://review.openstack.org/628

And - it includes a fix for the original issue..

Nathan Koterba (nkoterba) wrote :

I just installed Openstack Diablo on a new oneiric installation using the packages from apt-get and I'm still seeing this issue.

Are the changes listed in the link above in the latest packages? It doesn't look like it.

Kiall Mac Innes (kiall) wrote :

The review listed above won't be merged into diablo (it has loads of essex changes mixed in), and the launchpad branch is only half the fix..

I've just submitted the the minimal changes needed to fix this for review https://review.openstack.org/1328

Hopefully this will get approved and merged in...

Thierry Carrez (ttx) on 2011-11-09
Changed in nova:
importance: Medium → High
Changed in nova:
milestone: none → essex-1

Reviewed: https://review.openstack.org/1328
Committed: http://github.com/openstack/nova/commit/80d73a2df2c33212217c59783f62d74673776ebb
Submitter: Jenkins
Branch: stable/diablo

 status fixcommitted
 done

commit 80d73a2df2c33212217c59783f62d74673776ebb
Author: Kiall Mac Innes <email address hidden>
Date: Fri Nov 4 19:43:50 2011 +0000

    Fixes bug 834633 - Auto assigning floating IPs

    Details:

     - This fix is specific to the auto_assign_floating_ip=true feature, which was new in diablo[1] and looks like it never actually worked.
     - Trey's refactoring on master[2] fixed bug #834633 where a FloatingIp object was passed to network_api.associate_floating_ip() instead of an address string.
     - In bug #834633, Serguei Koubli had previously (i.e. before Trey's refactoring) proposed a much more isolated fix for this issue.
     - This commit includes Serguei Koubli's fix, and, has also fixed a seperate issue where auto-assigned flating IPs could not be deallocated. (This fix was also included in Trey's refactoring)

    [1] https://blueprints.launchpad.net/nova/+spec/floating-ip-auto-assignment
    [2] https://review.openstack.org/628

    Change-Id: Ie439c6743f066f8504ef88de861dee8a7762940d

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) wrote :

This is not fixed in essex-1, only in stable/diablo ?

Changed in nova:
status: Fix Committed → In Progress
assignee: Serguei Koubli (skoubli) → Vish Ishaya (vishvananda)
Thierry Carrez (ttx) wrote :

It's actually fixed by network refactor in essex already. Sorry for confusion.

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2011-11-17
Changed in nova:
status: Fix Committed → Fix Released

Hello Serguei, or anyone else affected,

Accepted nova into oneiric-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

tags: added: verification-needed

Hi Martin,
I've managed to update my packages from the proposed archive and confirm that this is working as expected now.

Cheers,

Kev

Martin Pitt (pitti) on 2011-12-20
tags: added: verification-done
removed: verification-needed
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-1 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers