L3 agent not always properly update floatingip status on server

Bug #1505557 reported by Oleg Bondarev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Oleg Bondarev

Bug Description

commit c44506bfd60b2dd6036e113464f1ea682cfaeb6c introduced an optimization to not send floating ip status update when status didn't change: so if server returned floating ip as ACTIVE we don't need to update it's status after successfull processing.

This migh be wrong in DVR case: when floatingip's associated fixed port is moved from one host to another, the notification is sent to both l3 agents on compute nodes (old and new). Here is what happens next:
 - old agent receives notification and requests router info from server
 - same for new agent
 - server returns router info without floating ip to old agent
 - server returns router info with floating ip to new agent. The status of floating ip is ACTIVE.
 - old agent removes floating ip and sends status update so server puts floatingip to DOWN state
 - new agent adds floatingip and doesn't send status update since it didn't changed from agent's point of view
 - floating ip stays in DOWN state though it's actually active

The fix would be to always update status of floating ip if agent actually applies it.

tags: added: l3-ipam-dhcp
removed: l3-dvr-backlog
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/234043

Changed in neutron:
status: New → In Progress
Changed in neutron:
importance: Undecided → Medium
tags: added: l3-dvr-backlog
tags: added: liberty-rc-potential
tags: added: kilo-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/234043
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7592b17b8793c0e296c77bd17929f3db37f1483e
Submitter: Jenkins
Branch: master

commit 7592b17b8793c0e296c77bd17929f3db37f1483e
Author: Oleg Bondarev <email address hidden>
Date: Tue Oct 13 11:55:02 2015 +0300

    Always send status update for processed floating ips

    Currently l3 agent skips status update for floating ips in case
    status didn't change: this might be wrong if status has changed
    on server side while agent was processing. See bug for details.

    L3 agent skips floating ip processing in case ip address exists
    on external device. So we can still skip status update for such
    floating ips.

    Closes-Bug: #1505557
    Change-Id: I908fe5a0555f68ab85e7d199c36a903b915e103f

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/235852

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/235855

Revision history for this message
Miguel Angel Ajo (mangelajo) wrote :

After fix, what if those updates happen in the inverted order, instead of :

- old agent removes floating ip and sends status update so server puts floatingip to DOWN state
 - new agent adds floatingip and send status update to ACTIVE

server receives:

- new agent adds floatingip and send status update to ACTIVE
- old agent removes floating ip and sends status update so server puts floatingip to DOWN state

Akihiro Motoki (amotoki)
tags: added: liberty-backport-potential
removed: liberty-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/liberty)

Reviewed: https://review.openstack.org/235852
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4a23ebfd2bfdcde6dc27bef30b7ef743ff605d91
Submitter: Jenkins
Branch: stable/liberty

commit 4a23ebfd2bfdcde6dc27bef30b7ef743ff605d91
Author: Oleg Bondarev <email address hidden>
Date: Tue Oct 13 11:55:02 2015 +0300

    Always send status update for processed floating ips

    Currently l3 agent skips status update for floating ips in case
    status didn't change: this might be wrong if status has changed
    on server side while agent was processing. See bug for details.

    L3 agent skips floating ip processing in case ip address exists
    on external device. So we can still skip status update for such
    floating ips.

    Closes-Bug: #1505557
    Change-Id: I908fe5a0555f68ab85e7d199c36a903b915e103f
    (cherry picked from commit 7592b17b8793c0e296c77bd17929f3db37f1483e)

tags: added: in-stable-liberty
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/235855
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0a0b5c452b15952877493abd85387e3c4eef8872
Submitter: Jenkins
Branch: stable/kilo

commit 0a0b5c452b15952877493abd85387e3c4eef8872
Author: Oleg Bondarev <email address hidden>
Date: Tue Oct 13 11:55:02 2015 +0300

    Always send status update for processed floating ips

    Currently l3 agent skips status update for floating ips in case
    status didn't change: this might be wrong if status has changed
    on server side while agent was processing. See bug for details.

    L3 agent skips floating ip processing in case ip address exists
    on external device. So we can still skip status update for such
    floating ips.

    Closes-Bug: #1505557

    Conflicts:

     neutron/tests/unit/agent/l3/test_agent.py

    Change-Id: I908fe5a0555f68ab85e7d199c36a903b915e103f
    (cherry picked from commit 7592b17b8793c0e296c77bd17929f3db37f1483e)

tags: added: in-stable-kilo
tags: removed: kilo-backport-potential
tags: removed: liberty-backport-potential
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/neutron 8.0.0.0b1

This issue was fixed in the openstack/neutron 8.0.0.0b1 development milestone.

Changed in neutron:
status: Fix Committed → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/neutron 7.0.1

This issue was fixed in the openstack/neutron 7.0.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.