DHCP agent takes very long time to report when port is provisioned

Bug #1850864 reported by Slawek Kaplonski on 2019-10-31
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Medium
Slawek Kaplonski

Bug Description

DHCP agent sends through rpc to neutron-server messages about ports which are already configured by agent: https://github.com/openstack/neutron/blob/3ad6af76266f466953de08451b922e033b5c8c43/neutron/agent/dhcp/agent.py#L255
It sends them in chunks with 64 ports at max.
Usually this works fine but in some cases when e.g. network with many ports (like 1000 for example) was resynced for some reason in "dhcp_ready_ports" there will be all those 1000 ports from resynced network.
And than new port may be at the end of this set and will be processed (send to neutron-server) after very long time. In our production cloud it could be even more than 15 minutes.
This cause problem with spawning vm which is in errors state due to "Failed to allocate network" error.

Fix proposed to branch: master
Review: https://review.opendev.org/692459

Changed in neutron:
status: Confirmed → In Progress

Reviewed: https://review.opendev.org/692459
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1a686fb401eca1843b81292fc88c13a2e6fd274d
Submitter: Zuul
Branch: master

commit 1a686fb401eca1843b81292fc88c13a2e6fd274d
Author: Slawek Kaplonski <email address hidden>
Date: Thu Oct 31 23:12:03 2019 +0100

    Switch to use cast method in dhcp_ready_on_ports method

    When DHCP agent reports to the neutron-server which ports are
    ready, it was using call() method from rpc client.
    That caused blocking dhcp agent's dhcp_ready_ports_loop thread which
    blocks to send info about other ready ports if there are any.

    Method call() should be used when RPC caller returns a value to the
    caller but that's not the case here. On neutron server side this
    RPC method is only calling provisioning_complete() method to
    finish provisioning of ports. And is not returning anything.

    So to make sending dhcp ready ports to neutron-server much faster
    this patch switch to use cast() method from rpc client.
    This method don't block to wait for return value from RPC caller.

    Change-Id: Ie119693854aa283b863a1eac2bdae3330c2b6a9d
    Closes-Bug: #1850864

Changed in neutron:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/694558
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=69250307ee3654f46efed4f9d81e2a555d635361
Submitter: Zuul
Branch: stable/train

commit 69250307ee3654f46efed4f9d81e2a555d635361
Author: Slawek Kaplonski <email address hidden>
Date: Thu Oct 31 23:12:03 2019 +0100

    Switch to use cast method in dhcp_ready_on_ports method

    When DHCP agent reports to the neutron-server which ports are
    ready, it was using call() method from rpc client.
    That caused blocking dhcp agent's dhcp_ready_ports_loop thread which
    blocks to send info about other ready ports if there are any.

    Method call() should be used when RPC caller returns a value to the
    caller but that's not the case here. On neutron server side this
    RPC method is only calling provisioning_complete() method to
    finish provisioning of ports. And is not returning anything.

    So to make sending dhcp ready ports to neutron-server much faster
    this patch switch to use cast() method from rpc client.
    This method don't block to wait for return value from RPC caller.

    Change-Id: Ie119693854aa283b863a1eac2bdae3330c2b6a9d
    Closes-Bug: #1850864
    (cherry picked from commit 1a686fb401eca1843b81292fc88c13a2e6fd274d)

tags: added: in-stable-train

Reviewed: https://review.opendev.org/694561
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0b1aac544bd9dd0f379d6ae37cd30ce07759dbde
Submitter: Zuul
Branch: stable/queens

commit 0b1aac544bd9dd0f379d6ae37cd30ce07759dbde
Author: Slawek Kaplonski <email address hidden>
Date: Thu Oct 31 23:12:03 2019 +0100

    Switch to use cast method in dhcp_ready_on_ports method

    When DHCP agent reports to the neutron-server which ports are
    ready, it was using call() method from rpc client.
    That caused blocking dhcp agent's dhcp_ready_ports_loop thread which
    blocks to send info about other ready ports if there are any.

    Method call() should be used when RPC caller returns a value to the
    caller but that's not the case here. On neutron server side this
    RPC method is only calling provisioning_complete() method to
    finish provisioning of ports. And is not returning anything.

    So to make sending dhcp ready ports to neutron-server much faster
    this patch switch to use cast() method from rpc client.
    This method don't block to wait for return value from RPC caller.

    Change-Id: Ie119693854aa283b863a1eac2bdae3330c2b6a9d
    Closes-Bug: #1850864
    (cherry picked from commit 1a686fb401eca1843b81292fc88c13a2e6fd274d)

tags: added: in-stable-queens
tags: added: in-stable-rocky

Reviewed: https://review.opendev.org/694560
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8c2b944dc08377fa84d1264ce1b24fbbada44c52
Submitter: Zuul
Branch: stable/rocky

commit 8c2b944dc08377fa84d1264ce1b24fbbada44c52
Author: Slawek Kaplonski <email address hidden>
Date: Thu Oct 31 23:12:03 2019 +0100

    Switch to use cast method in dhcp_ready_on_ports method

    When DHCP agent reports to the neutron-server which ports are
    ready, it was using call() method from rpc client.
    That caused blocking dhcp agent's dhcp_ready_ports_loop thread which
    blocks to send info about other ready ports if there are any.

    Method call() should be used when RPC caller returns a value to the
    caller but that's not the case here. On neutron server side this
    RPC method is only calling provisioning_complete() method to
    finish provisioning of ports. And is not returning anything.

    So to make sending dhcp ready ports to neutron-server much faster
    this patch switch to use cast() method from rpc client.
    This method don't block to wait for return value from RPC caller.

    Change-Id: Ie119693854aa283b863a1eac2bdae3330c2b6a9d
    Closes-Bug: #1850864
    (cherry picked from commit 1a686fb401eca1843b81292fc88c13a2e6fd274d)

tags: added: in-stable-stein

Reviewed: https://review.opendev.org/694559
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=36977d5d0ce337bf0bcdeaee49ce6e9723a3810c
Submitter: Zuul
Branch: stable/stein

commit 36977d5d0ce337bf0bcdeaee49ce6e9723a3810c
Author: Slawek Kaplonski <email address hidden>
Date: Thu Oct 31 23:12:03 2019 +0100

    Switch to use cast method in dhcp_ready_on_ports method

    When DHCP agent reports to the neutron-server which ports are
    ready, it was using call() method from rpc client.
    That caused blocking dhcp agent's dhcp_ready_ports_loop thread which
    blocks to send info about other ready ports if there are any.

    Method call() should be used when RPC caller returns a value to the
    caller but that's not the case here. On neutron server side this
    RPC method is only calling provisioning_complete() method to
    finish provisioning of ports. And is not returning anything.

    So to make sending dhcp ready ports to neutron-server much faster
    this patch switch to use cast() method from rpc client.
    This method don't block to wait for return value from RPC caller.

    Change-Id: Ie119693854aa283b863a1eac2bdae3330c2b6a9d
    Closes-Bug: #1850864
    (cherry picked from commit 1a686fb401eca1843b81292fc88c13a2e6fd274d)

This issue was fixed in the openstack/neutron 13.0.6 release.

This issue was fixed in the openstack/neutron 14.0.4 release.

This issue was fixed in the openstack/neutron 15.0.1 release.

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers