[L3] snat gateway port may stay 4095 after router fully initialized in l3 agent

Bug #1827754 reported by LIU Yulong
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Unassigned

Bug Description

ENV: stable/queens, but master branch has the same code.

There are some extremly conditions which will result the unbound router gateway port. Then all the centrilized floating IPs will not be reachable since the gateway port was set to 4095 tag.

How to reproduce:
just simply creating dvr HA router, set gateway, and add interface.
You can run the following script 100 times to create 100 routers:
for i in {1..100};do script.sh $i;done
Then you may see there are some gatway ports are set to 4095 in the snat node.

Test script:
function create_net_struct()
{
  neutron router-create scale-test-router-${1}
  neutron net-create scale-test-net-${1}
  neutron subnet-create --name scale-test-subnet-${1} scale-test-net-${1} 192.168.${1}.0/24
  neutron router-gateway-set scale-test-router-${1} public
  neutron router-interface-add scale-test-router-${1} subnet=scale-test-subnet-${1}
}

create_net_struct $1

LIU Yulong (dragon889)
Changed in neutron:
assignee: nobody → LIU Yulong (dragon889)
Changed in neutron:
status: New → Confirmed
tags: added: l3-dvr-backlog
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/657207

Changed in neutron:
status: Confirmed → In Progress
tags: added: queens-backport-potential rocky-backport-potential stein-backport-potential
Changed in neutron:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/657207
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3d99147e732ab4cdad0b8243e5fabb6a8954aa72
Submitter: Zuul
Branch: master

commit 3d99147e732ab4cdad0b8243e5fabb6a8954aa72
Author: LIU Yulong <email address hidden>
Date: Wed Apr 24 14:19:36 2019 +0800

    Ensure dvr ha router gateway port binding host

    There are some extreme conditions which will result the unbound
    router gateway port. Then all the centralized floating IPs will
    not be reachable since the gateway port was set to 4095 tag.

    This patch adds the HA status to the router related port
    processing code path. If it is HA router, the gateway port
    will go to the right HA router processing code branch.

    Closes-Bug: #1827754
    Change-Id: Ida1c9f3a38171ea82adc2f11cb17945d6e2434be

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/659647

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/659648

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/659649

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/659650

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.opendev.org/659651

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/659647
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fef6b12bfef1f4f2aec98e1e3e85c94cb64e5cc2
Submitter: Zuul
Branch: stable/stein

commit fef6b12bfef1f4f2aec98e1e3e85c94cb64e5cc2
Author: LIU Yulong <email address hidden>
Date: Wed Apr 24 14:19:36 2019 +0800

    Ensure dvr ha router gateway port binding host

    There are some extreme conditions which will result the unbound
    router gateway port. Then all the centralized floating IPs will
    not be reachable since the gateway port was set to 4095 tag.

    This patch adds the HA status to the router related port
    processing code path. If it is HA router, the gateway port
    will go to the right HA router processing code branch.

    Closes-Bug: #1827754
    Change-Id: Ida1c9f3a38171ea82adc2f11cb17945d6e2434be
    (cherry picked from commit 3d99147e732ab4cdad0b8243e5fabb6a8954aa72)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/659649
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=265d1cffdd9e876452baeae8688969e9dfac1fcc
Submitter: Zuul
Branch: stable/queens

commit 265d1cffdd9e876452baeae8688969e9dfac1fcc
Author: LIU Yulong <email address hidden>
Date: Wed Apr 24 14:19:36 2019 +0800

    Ensure dvr ha router gateway port binding host

    There are some extreme conditions which will result the unbound
    router gateway port. Then all the centralized floating IPs will
    not be reachable since the gateway port was set to 4095 tag.

    This patch adds the HA status to the router related port
    processing code path. If it is HA router, the gateway port
    will go to the right HA router processing code branch.

    Closes-Bug: #1827754
    Change-Id: Ida1c9f3a38171ea82adc2f11cb17945d6e2434be
    (cherry picked from commit 3d99147e732ab4cdad0b8243e5fabb6a8954aa72)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/pike)

Reviewed: https://review.opendev.org/659650
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=66f5e0acb60ddc80a55af212f639c54a86125fd5
Submitter: Zuul
Branch: stable/pike

commit 66f5e0acb60ddc80a55af212f639c54a86125fd5
Author: LIU Yulong <email address hidden>
Date: Wed Apr 24 14:19:36 2019 +0800

    Ensure dvr ha router gateway port binding host

    There are some extreme conditions which will result the unbound
    router gateway port. Then all the centralized floating IPs will
    not be reachable since the gateway port was set to 4095 tag.

    This patch adds the HA status to the router related port
    processing code path. If it is HA router, the gateway port
    will go to the right HA router processing code branch.

    Closes-Bug: #1827754
    Change-Id: Ida1c9f3a38171ea82adc2f11cb17945d6e2434be
    (cherry picked from commit 3d99147e732ab4cdad0b8243e5fabb6a8954aa72)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/659648
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2f91450fcf64dacdf98cbb7fbf73fdd09bc17f66
Submitter: Zuul
Branch: stable/rocky

commit 2f91450fcf64dacdf98cbb7fbf73fdd09bc17f66
Author: LIU Yulong <email address hidden>
Date: Wed Apr 24 14:19:36 2019 +0800

    Ensure dvr ha router gateway port binding host

    There are some extreme conditions which will result the unbound
    router gateway port. Then all the centralized floating IPs will
    not be reachable since the gateway port was set to 4095 tag.

    This patch adds the HA status to the router related port
    processing code path. If it is HA router, the gateway port
    will go to the right HA router processing code branch.

    Closes-Bug: #1827754
    Change-Id: Ida1c9f3a38171ea82adc2f11cb17945d6e2434be
    (cherry picked from commit 3d99147e732ab4cdad0b8243e5fabb6a8954aa72)

tags: added: in-stable-rocky
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 13.0.4

This issue was fixed in the openstack/neutron 13.0.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 14.0.2

This issue was fixed in the openstack/neutron 14.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 12.1.0

This issue was fixed in the openstack/neutron 12.1.0 release.

tags: removed: neutron-proactive-backport-potential rocky-backport-potential stein-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 15.0.0.0b1

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

Revision history for this message
Slawek Kaplonski (slaweq) wrote : auto-abandon-script

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

Changed in neutron:
assignee: LIU Yulong (dragon889) → nobody
tags: added: timeout-abandon
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/ocata)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/ocata
Review: https://review.opendev.org/659651
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron pike-eol

This issue was fixed in the openstack/neutron pike-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.