Fuel for OpenStack

nova error migrating VMs with floating ips: 'FixedIP' object has no attribute '_sa_instance_state'

Bug #1334164 reported by Roman Podoliaka
36
This bug affects 7 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Roman Podoliaka
Fuel for OpenStack 5.1
5.0.x
Fix Released
Critical
Roman Podoliaka
Fuel for OpenStack 5.0.1
Mirantis OpenStack
Fix Released
High
Roman Podoliaka
Mirantis OpenStack 5.1
5.0.x
Fix Released
Critical
Roman Podoliaka
Mirantis OpenStack 5.0.1
5.1.x
Fix Released
High
Roman Podoliaka
Mirantis OpenStack 5.1
OpenStack Compute (nova)
Fix Released
High
Dan Smith
OpenStack Compute (nova) 2014.2 "juno"
Icehouse
Fix Released
High
Nathanael Burton
OpenStack Compute (nova) 2014.1.3

Bug Description

Seeing this in conductor logs when migrating a VM with a floating IP assigned:

2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py", line 1019, in network_migrate_instance_start
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher migration)
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py", line 527, in network_migrate_instance_start
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher self.network_api.migrate_instance_start(context, instance, migration)
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 94, in wrapped
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher return func(self, context, *args, **kwargs)
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 543, in migrate_instance_start
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher self.network_rpcapi.migrate_instance_start(context, **args)
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/rpcapi.py", line 350, in migrate_instance_start
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher floating_addresses=floating_addresses)
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher wait_for_reply=True, timeout=timeout)
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/transport.py", line 90, in _send
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher timeout=timeout)
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 409, in send
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 402, in _send
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher raise result
2014-06-23 09:22:38.899 20314 TRACE oslo.messaging.rpc.dispatcher AttributeError: 'FixedIP' object has no attribute '_sa_instance_state'

Roman Podoliaka (rpodolyaka)
Changed in nova:
assignee: nobody → Roman Podoliaka (rpodolyaka)
tags: added: nova
Changed in mos:
importance: Undecided → High
assignee: nobody → Roman Podoliaka (rpodolyaka)
Changed in nova:
status: New → Incomplete
status: Incomplete → Confirmed
Changed in mos:
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/102514

Changed in nova:
status: Confirmed → In Progress
Roman Podoliaka (rpodolyaka)
summary: - nova error migrating with floating ips: 'FixedIP' object has no
+ nova error migrating VMs with floating ips: 'FixedIP' object has no
attribute '_sa_instance_state'
Revision history for this message
Noel Burton-Krahn (noelbk) wrote :
Download full text (6.8 KiB)

That fix did get past the sa_instance_state problem, but I bumped into something new. It looks like floating_ip_update() doesn't like the SA fixed ip that's fetched here:

+ updates['fixed_ip'] = db.fixed_ip_get(context, id_)

I'm not sure what "A conflicting state is already present in the identity map" means. Is there another way to fetch the fixed ip?

Traceback:

Jun 25 17:30:21 10.35.0.14 nova-network 2014-06-25 17:30:21.764 4992 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: A conflicting state is already present in the identity map for key (<class 'nova.db.sqlalchemy.models.FixedIp'>, (3L,))
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/network/floating_ips.py", line 575, in migrate_instance_start
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/objects/base.py", line 151, in wrapper
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/objects/floating_ip.py", line 135, in save
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/db/api.py", line 386, in floating_ip_update
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 164, in wrapper
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1024, in floating_ip_update
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/openstack/common/db/sqlalchemy/models.py", line 88, in update
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 220, in __set__
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 780, in set
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 801, in fire_replace_event
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 87, in set_
2014-06-25 17:30:21.764 4992 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1490, in _save_o...

Read more...

Revision history for this message
Noel Burton-Krahn (noelbk) wrote :

This fix worked for me, but it seems ugly to have a special case for a field by name. Is there a less hard-coded way to do this?

    def save(self, context):
        ...
        if 'fixed_ip' in updates:
            updates['fixed_ip'].save(context)
            del(updates['fixed_ip'])

Roman Podoliaka (rpodolyaka)
Changed in fuel:
status: New → In Progress
assignee: nobody → Roman Podoliaka (rpodolyaka)
Roman Podoliaka (rpodolyaka)
Changed in fuel:
importance: Undecided → Critical
milestone: none → 5.0.1
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Fixed in https://gerrit.mirantis.com/#/c/17107/

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Artem Panchenko (apanchenko-8) wrote :

Fix released for 5.0.1, verified on iso # 88

Nathanael Burton (mathrock)
tags: added: icehouse-backport-potential
Revision history for this message
Meg McRoberts (dreidellhasa) wrote :

Should this be in the Release Notes list of Fixed Issues? If so, did the customer see any functional problems or is it just the log errors?

Matt Riedemann (mriedem)
tags: added: migration
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/109310

Changed in nova:
assignee: Roman Podoliaka (rpodolyaka) → Dan Smith (danms)
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/109310
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=48de2895b9a550a0944b31212349275605a4061d
Submitter: Jenkins
Branch: master

commit 48de2895b9a550a0944b31212349275605a4061d
Author: Dan Smith <email address hidden>
Date: Thu Jul 24 07:35:27 2014 -0700

    Fix FloatingIP.save() passing FixedIP object to sqlalchemy

    This prevents the FloatingIP.save() method from passing the
    calculated FixedIP object to the sqlalchemy floating_ip_update()
    function, which would expect it to be an SA object. It also
    aborts any attempt to save the object with a modified fixed_ip_id
    linkage, as associate/disassociate should be used for that.

    This also fixes a bug where FloatingIP expects the result of
    floating_ip_update() to be a FloatingIp SA object.

    Change-Id: I065caedf4d81c8583a3b390934a1d403cf2e87bd
    Closes-bug: #1334164

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Roman Podoliaka (<email address hidden>) on branch: master
Review: https://review.openstack.org/102514

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/112184

Revision history for this message
Oleksiy (lev4ykaol) wrote :
Download full text (33.1 KiB)

after applying patch still getting the following error:

2014-08-13 20:25:58.387 12971 ERROR nova.openstack.common.loopingcall [-] in fixed duration looping call
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall Traceback (most recent call last):
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.7/dist-packages/nova/openstack/common/loopingcall.py", line 78, in _inner
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall self.f(*self.args, **self.kw)
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4518, in wait_for_live_migration
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall migrate_data)
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall payload)
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall six.reraise(self.type_, self.value, self.tb)
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall return f(self, context, *args, **kw)
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall e, sys.exc_info())
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall six.reraise(self.type_, self.value, self.tb)
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall return function(self, context, *args, **kwargs)
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4618, in _post_live_migration
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall migration)
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 254, in network_migrate_instance_start
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall migration)
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingcall File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 384, in network_migrate_instance_start
2014-08-13 20:25:58.387 12971 TRACE nova.openstack.common.loopingc...

Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/6.0.x
no longer affects: fuel/5.1.x
Nathanael Burton (mathrock)
tags: added: ops
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/112184
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=526853ea2058a9fb650ae2f34bc0404f4438b34e
Submitter: Jenkins
Branch: stable/icehouse

commit 526853ea2058a9fb650ae2f34bc0404f4438b34e
Author: Dan Smith <email address hidden>
Date: Thu Jul 24 07:35:27 2014 -0700

    Fix FloatingIP.save() passing FixedIP object to sqlalchemy

    This prevents the FloatingIP.save() method from passing the
    calculated FixedIP object to the sqlalchemy floating_ip_update()
    function, which would expect it to be an SA object. It also
    aborts any attempt to save the object with a modified fixed_ip_id
    linkage, as associate/disassociate should be used for that.

    This also fixes a bug where FloatingIP expects the result of
    floating_ip_update() to be a FloatingIp SA object.

    Added missing import to allow tests to work in stable/icehouse.

    Change-Id: I065caedf4d81c8583a3b390934a1d403cf2e87bd
    Closes-bug: #1334164
    (cherry picked from commit 48de2895b9a550a0944b31212349275605a4061d)

tags: added: in-stable-icehouse
Thierry Carrez (ttx)
Changed in nova:
milestone: none → juno-3
status: Fix Committed → Fix Released
Roman Podoliaka (rpodolyaka)
no longer affects: mos/6.0.x
Alexander Gubanov (ogubanov)
Changed in mos:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-3 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.