DHCP agent should prioritize new ports when sending RPC messages to server

[Brian Haley]

When a port is provisioned in the dhcp-agent, for example via a port_create, it will just be added to the dhcp_ready_ports set and sent to neutron-server in _dhcp_ready_ports_loop() by popping elements off the list. So although it was prioritized when it was received, it is not prioritized when sent to the server to clear the provisioning block.

It seems like these ports should be sent first, then others behind it if there is still room in the RPC message. This could just be done with a second set() perhaps, unless we want to make it more complicated by using the priority sent from the server to place ports in different queues.

This should decrease the time it takes to clear the port provisioning block when an agent is restarted and gets a port_create message, as it would help even if it was sent with PRIORITY_PORT_CREATE_HIGH to a single agent, since the one it chose could still be in the middle of a full sync.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/709824

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/714193

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/709824
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=113dfac6083c2bda36f5186e123e4e5c82fa8097
Submitter: Zuul
Branch: master

commit 113dfac6083c2bda36f5186e123e4e5c82fa8097
Author: Brian Haley <email address hidden>
Date: Tue Feb 25 15:01:47 2020 -0500

    Prioritize port create and update ready messages

    The DHCP agent prioritizes RPC messages based on the
    priority field send from neutron-server, but then groups
    them all in the same dhcp_ready_ports set when sending
    them back to the server to clear the provisioning block(s).

    Priority should be given to new and changed ports, since
    those are most likely to be associated with new instances
    which can fail to boot if they are not handled quickly when
    the agent is very busy, for example, right after it was
    restarted.

    Change-Id: Ib5074abadd7189bb4bdd5e46c677f1bfb071221e
    Closes-bug: #1864675

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/714193
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f31ae53dd5cee1ebaec26dc563e8a3562f897b49
Submitter: Zuul
Branch: master

commit f31ae53dd5cee1ebaec26dc563e8a3562f897b49
Author: Brian Haley <email address hidden>
Date: Fri Mar 20 19:07:17 2020 +0000

    Revert "Switch to use cast method in dhcp_ready_on_ports method"

    This reverts commit 1a686fb401eca1843b81292fc88c13a2e6fd274d.

    With https://review.opendev.org/#/c/709824/ and this reverted, we
    have seen things work better in our large scale environment. Although
    cast() will return immediately, it can overwhelm the server by sending
    a lot of messages quickly, and won't necessarily prioritize the "new"
    ones, which are typically more important to mark as provisioned first.

    Change-Id: Ie61b222eec87c5613efc6a33553844c64a655a57
    Related-bug: #1864675

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/717335

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/717336

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/717337

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/717380

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717380
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a8e526bb0fb9e1e9a3f7a96ee895b883aaf2164d
Submitter: Zuul
Branch: stable/train

commit a8e526bb0fb9e1e9a3f7a96ee895b883aaf2164d
Author: Brian Haley <email address hidden>
Date: Fri Mar 20 19:07:17 2020 +0000

    Revert "Switch to use cast method in dhcp_ready_on_ports method"

    This reverts commit 1a686fb401eca1843b81292fc88c13a2e6fd274d.

    With https://review.opendev.org/#/c/709824/ and this reverted, we
    have seen things work better in our large scale environment. Although
    cast() will return immediately, it can overwhelm the server by sending
    a lot of messages quickly, and won't necessarily prioritize the "new"
    ones, which are typically more important to mark as provisioned first.

    Change-Id: Ie61b222eec87c5613efc6a33553844c64a655a57
    Related-bug: #1864675
    (cherry picked from commit f31ae53dd5cee1ebaec26dc563e8a3562f897b49)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717335
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1bf64726fa2e23fa8e8c565186cd5d9212ea1a1b
Submitter: Zuul
Branch: stable/train

commit 1bf64726fa2e23fa8e8c565186cd5d9212ea1a1b
Author: Brian Haley <email address hidden>
Date: Tue Feb 25 15:01:47 2020 -0500

    Prioritize port create and update ready messages

    The DHCP agent prioritizes RPC messages based on the
    priority field send from neutron-server, but then groups
    them all in the same dhcp_ready_ports set when sending
    them back to the server to clear the provisioning block(s).

    Priority should be given to new and changed ports, since
    those are most likely to be associated with new instances
    which can fail to boot if they are not handled quickly when
    the agent is very busy, for example, right after it was
    restarted.

    Conflicts:
        neutron/tests/unit/agent/dhcp/test_agent.py

    Change-Id: Ib5074abadd7189bb4bdd5e46c677f1bfb071221e
    Closes-bug: #1864675
    (cherry picked from commit 113dfac6083c2bda36f5186e123e4e5c82fa8097)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717336
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e2f01c65d91ab7b7837c7ab3118ec00b8310dc3c
Submitter: Zuul
Branch: stable/stein

commit e2f01c65d91ab7b7837c7ab3118ec00b8310dc3c
Author: Brian Haley <email address hidden>
Date: Tue Feb 25 15:01:47 2020 -0500

    Prioritize port create and update ready messages

    The DHCP agent prioritizes RPC messages based on the
    priority field send from neutron-server, but then groups
    them all in the same dhcp_ready_ports set when sending
    them back to the server to clear the provisioning block(s).

    Priority should be given to new and changed ports, since
    those are most likely to be associated with new instances
    which can fail to boot if they are not handled quickly when
    the agent is very busy, for example, right after it was
    restarted.

    Conflicts:
        neutron/tests/unit/agent/dhcp/test_agent.py

    Change-Id: Ib5074abadd7189bb4bdd5e46c677f1bfb071221e
    Closes-bug: #1864675
    (cherry picked from commit 113dfac6083c2bda36f5186e123e4e5c82fa8097)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717337
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=18c7e3ead7e385b83c516cbe4558ddad60fbd54e
Submitter: Zuul
Branch: stable/rocky

commit 18c7e3ead7e385b83c516cbe4558ddad60fbd54e
Author: Brian Haley <email address hidden>
Date: Tue Feb 25 15:01:47 2020 -0500

    Prioritize port create and update ready messages

    The DHCP agent prioritizes RPC messages based on the
    priority field send from neutron-server, but then groups
    them all in the same dhcp_ready_ports set when sending
    them back to the server to clear the provisioning block(s).

    Priority should be given to new and changed ports, since
    those are most likely to be associated with new instances
    which can fail to boot if they are not handled quickly when
    the agent is very busy, for example, right after it was
    restarted.

    Conflicts:
        neutron/tests/unit/agent/dhcp/test_agent.py

    Change-Id: Ib5074abadd7189bb4bdd5e46c677f1bfb071221e
    Closes-bug: #1864675
    (cherry picked from commit 113dfac6083c2bda36f5186e123e4e5c82fa8097)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/726172

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/726172
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f0606dbb7fa4024e9f25b9b6c73e33897b5087f9
Submitter: Zuul
Branch: stable/queens

commit f0606dbb7fa4024e9f25b9b6c73e33897b5087f9
Author: Brian Haley <email address hidden>
Date: Tue Feb 25 15:01:47 2020 -0500

    Prioritize port create and update ready messages

    The DHCP agent prioritizes RPC messages based on the
    priority field send from neutron-server, but then groups
    them all in the same dhcp_ready_ports set when sending
    them back to the server to clear the provisioning block(s).

    Priority should be given to new and changed ports, since
    those are most likely to be associated with new instances
    which can fail to boot if they are not handled quickly when
    the agent is very busy, for example, right after it was
    restarted.

    Conflicts:
        neutron/tests/unit/agent/dhcp/test_agent.py

    Change-Id: Ib5074abadd7189bb4bdd5e46c677f1bfb071221e
    Closes-bug: #1864675
    (cherry picked from commit 113dfac6083c2bda36f5186e123e4e5c82fa8097)
    (cherry picked from commit 18c7e3ead7e385b83c516cbe4558ddad60fbd54e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.