Key error when try to assign a floating ip to a VM

Bug #1629484 reported by Hao Chen
38
This bug affects 9 people
Affects Status Importance Assigned to Milestone
neutron
High
Hao Chen

Bug Description

We have an environment with designate and DNS plugin enabled. When we upgrade to newton we got a problem that we can't assign the floating ip to the VM because of an key error.

Here is the error message from neutron server

./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource Traceback (most recent call last):
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 79, in resource
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource result = method(request=request, **args)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 604, in update
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource return self._update(request, id, body, **kwargs)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 88, in wrapped
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource self.force_reraise()
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 84, in wrapped
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource return f(*args, **kwargs)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource self.force_reraise()
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource return f(*args, **kwargs)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource traceback.format_exc())
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource self.force_reraise()
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 119, in wrapped
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 652, in _update
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource obj = obj_updater(request.context, id, **kwargs)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 159, in wrapped
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource return method(*args, **kwargs)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 88, in wrapped
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource self.force_reraise()
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 84, in wrapped
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource return f(*args, **kwargs)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource self.force_reraise()
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource return f(*args, **kwargs)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource traceback.format_exc())
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource self.force_reraise()
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 119, in wrapped
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 1015, in update_floatingip
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource context, id, floatingip)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1355, in _update_floatingip
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource context, floatingip_db)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/dns_db.py", line 210, in _process_dns_floatingip_update_precommit
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource context, floatingip_data))
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/db/dns_db.py", line 313, in _get_requested_state_for_external_dns_service_update
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource if floatingip_data['port_id']:
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 54, in __getitem__
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource return getattr(self, key)
./neutron/neutron-server.log:2016-09-30 22:00:36.797 527 ERROR neutron.api.v2.resource AttributeError: 'FloatingIP' object has no attribute 'port_id'

After digging more it seems like there are several places in the code try to get the value for "port_id" which doesn't exist

https://github.com/openstack/neutron/blob/stable/newton/neutron/db/dns_db.py#L313
https://github.com/openstack/neutron/blob/stable/newton/neutron/db/dns_db.py#L275
https://github.com/openstack/neutron/blob/stable/newton/neutron/db/dns_db.py#L281

from https://github.com/openstack/neutron/blob/stable/newton/neutron/db/l3_db.py#L1041 we can see that port_id is actually the "fix_port_id" attribute in floatingip_db object.

Fix the previous key I mentioned will fix the floating ip assignment.

Hao Chen (chenh1987)
Changed in neutron:
assignee: nobody → Hao Chen (chenh1987)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/380669

Changed in neutron:
status: New → In Progress
tags: added: db l3-ipam-dhcp newton-rc-potential
Changed in neutron:
importance: Undecided → High
tags: added: dns
Changed in neutron:
milestone: none → ocata-1
Revision history for this message
Miguel Lavalle (minsel) wrote :

Unit tests in this patchset can be used to verify the fix for this bug: https://review.openstack.org/#/c/301809

tags: added: newton-backport-potential
removed: newton-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/380669
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=19af726e39b30670314b3bf7025b3a682e523953
Submitter: Jenkins
Branch: master

commit 19af726e39b30670314b3bf7025b3a682e523953
Author: chen, hao <email address hidden>
Date: Fri Sep 30 16:06:38 2016 -0700

    Fix the 'port_id' attribute not exist error

    Change-Id: Ie9e42771788d8a20572cb650b15edb95923dd426
    Closes-Bug: #1629484

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/381829

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/381829
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1865eb5de087d5f2e7d47ef27f7a3ac2d8dd6996
Submitter: Jenkins
Branch: stable/newton

commit 1865eb5de087d5f2e7d47ef27f7a3ac2d8dd6996
Author: chen, hao <email address hidden>
Date: Fri Sep 30 16:06:38 2016 -0700

    Fix the 'port_id' attribute not exist error

    Change-Id: Ie9e42771788d8a20572cb650b15edb95923dd426
    Closes-Bug: #1629484
    (cherry picked from commit 19af726e39b30670314b3bf7025b3a682e523953)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.1.0

This issue was fixed in the openstack/neutron 9.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 10.0.0.0b1

This issue was fixed in the openstack/neutron 10.0.0.0b1 development milestone.

tags: added: neutron-proactive-backport-potential
tags: removed: in-stable-newton neutron-proactive-backport-potential newton-backport-potential
tags: added: in-stable-newton
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers