dhcp agent race condition between between network_create_end and port_delete_end RPC

[Allain Legacy]

A race condition exists in the DHCP agent between the network_create_end and port_delete_end RPC callbacks. If a DHCP agent is fetching network info [1] at the exact same time as a port is being deleted [2] it is possible that the port delete RPC arrives after the network info RPC completed on the server but before the result is processed on the agent. That leads to a condition where the agent has a port in its network cache that has already been deleted. That then has the potential to add duplicate entries in the dnsmasq host file for one IP address (two different MAC addresses).

Because this is a timing issue I am not able to create a standalone/isolated test case to show this behavior. In one of our QA test labs this happens ~20% of the time when a VM is deleted at the roughly the same time that a DHCP server is moved from one node to another.

This is happening because there is no synchronization between the port_delete_end and the network_create_end RPC event handlers. Since the port_delete_end RPC does not have any network_id information there is no way to synchronize the two operations in the agent. In our system we have addressed this by changing the *_delete_end RPC notifications coming from [3] to also include the network_id and then changing [2] to acquire _net_lock(network_id) before processing the port_delete_end notification. This approach addressed all occurrences of the issue. I have included a copy of our patch for reference here [4].

This an example of the series of logs generated by dnsmasq when a duplicate entry is added because of the stale port described above. dnsmasq then refuses to serve the IP address because of the duplicate and the VM never gets an IP address.

2017-11-06T10:24:57.000 compute-0 dnsmasq-dhcp[57037]: info read /var/run/neutron/dhcp/c3b9af9c-2e6b-42af-8fb7-d73b552fb24b/host
2017-11-06T10:24:57.000 compute-0 dnsmasq-dhcp[57037]: info read /var/run/neutron/dhcp/c3b9af9c-2e6b-42af-8fb7-d73b552fb24b/opts
2017-11-06T10:24:58.000 compute-0 dnsmasq[57037]: info read /var/run/neutron/dhcp/c3b9af9c-2e6b-42af-8fb7-d73b552fb24b/addn_hosts - 13 addresses
2017-11-06T10:24:58.000 compute-0 dnsmasq[57037]: err duplicate dhcp-host IP address 192.168.216.75 at line 13 of /var/run/neutron/dhcp/c3b9af9c-2e6b-42af-8fb7-d73b552fb24b/host
2017-11-06T10:24:58.000 compute-0 dnsmasq-dhcp[57037]: info read /var/run/neutron/dhcp/c3b9af9c-2e6b-42af-8fb7-d73b552fb24b/host
2017-11-06T10:24:58.000 compute-0 dnsmasq-dhcp[57037]: info read /var/run/neutron/dhcp/c3b9af9c-2e6b-42af-8fb7-d73b552fb24b/opts

[1] neutron.agent.dhcp.agent.DhcpPluginApi.get_network_info
[2] neutron.agent.dhcp.agent.DhcpAgent.port_delete_end
[3] neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api.DhcpAgentNotifyAPI.notify
[4] http://paste.openstack.org/show/626381/

[Brian Haley (brian-haley)]

Hi - I've looked at the patch and think it's probably close, it would just need to account for the case where there is no network_id in the message.

Can you send this out for review so we can gather more feedback?

[Allain Legacy (alegacy)]

sure, i can try to rework it a bit. The patch as attached is like that because in our system deployment we are guaranteed that the neutron-server node is always at the same version (or better) than the agent therefore the network_id is always present. I realize that is not the case for other users/installations.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/524711

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/524711
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fa78b580105111b1238e5b18d415b08e8fb35d97
Submitter: Zuul
Branch: master

commit fa78b580105111b1238e5b18d415b08e8fb35d97
Author: Allain Legacy <email address hidden>
Date: Fri Nov 10 10:44:50 2017 -0600

    dhcp: serializing port delete and network rpc calls

    The port delete events are not synchronized with network rpc events. This
    creates a condition which makes it possible for a port delete event to be
    processed just before a previously started network query completes.

    The problematic order of operations is as follows:

      1) a network is scheduled to an agent; a network rpc is sent to the
         agent

      2) the agent queries the network data from the server

      3) while that query is in progress a port on that network is deleted; a
         port rpc is sent to the agent

      4) that port delete rpc is received before the network query rpc
         completes

      5) the port delete results in no action because the port was not present
         on the agent

      6) the network query finishes and adds the port to the cache (even
         though the port has already been deleted)

      7) some time passes and a new port is configured with the same IP
         address as the port that was deleted in (3)

      8) the dhcp host file is corrupted with 2 entries for the same IP
         address.

      9) dhcp queries for the newest port is rejected because of the duplicate
         entry in the dhcp host file.

    The solution is to add the network_id to the port_delete_end rpc event
    so that the _net_lock(network_id) synchronization point can be acquired
    so that it is processed serially with other network related events.

    To ensure backwards compatibility with newer agents running against older
    servers the determination of which network_id value to use in the lock is
    handled using a utility that will fallback to the previous mode of operation
    whenever the network_id attribute is not present in the *_delete_end RPC
    events. That utility can be removed in the future when it is guaranteed
    that the network_id attribute will be present in RPC messages from the
    server.

    Closes-Bug: #1732456

    Change-Id: I735f8b1c9248b12e5feb6cbe970cf67f321e6ebc
    Signed-off-by: Allain Legacy <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/605562

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/605563

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605563
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4ca8baf6966657eff474bf88e1bd8c296e630a1e
Submitter: Zuul
Branch: stable/queens

commit 4ca8baf6966657eff474bf88e1bd8c296e630a1e
Author: Allain Legacy <email address hidden>
Date: Fri Nov 10 10:44:50 2017 -0600

    dhcp: serializing port delete and network rpc calls

    The port delete events are not synchronized with network rpc events. This
    creates a condition which makes it possible for a port delete event to be
    processed just before a previously started network query completes.

    The problematic order of operations is as follows:

      1) a network is scheduled to an agent; a network rpc is sent to the
         agent

      2) the agent queries the network data from the server

      3) while that query is in progress a port on that network is deleted; a
         port rpc is sent to the agent

      4) that port delete rpc is received before the network query rpc
         completes

      5) the port delete results in no action because the port was not present
         on the agent

      6) the network query finishes and adds the port to the cache (even
         though the port has already been deleted)

      7) some time passes and a new port is configured with the same IP
         address as the port that was deleted in (3)

      8) the dhcp host file is corrupted with 2 entries for the same IP
         address.

      9) dhcp queries for the newest port is rejected because of the duplicate
         entry in the dhcp host file.

    The solution is to add the network_id to the port_delete_end rpc event
    so that the _net_lock(network_id) synchronization point can be acquired
    so that it is processed serially with other network related events.

    To ensure backwards compatibility with newer agents running against older
    servers the determination of which network_id value to use in the lock is
    handled using a utility that will fallback to the previous mode of operation
    whenever the network_id attribute is not present in the *_delete_end RPC
    events. That utility can be removed in the future when it is guaranteed
    that the network_id attribute will be present in RPC messages from the
    server.

    Closes-Bug: #1732456

    Change-Id: I735f8b1c9248b12e5feb6cbe970cf67f321e6ebc
    Signed-off-by: Allain Legacy <email address hidden>
    (cherry picked from commit fa78b580105111b1238e5b18d415b08e8fb35d97)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605562
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d0931c4e55ea5cabedcee70ec13d76f0dd5cc714
Submitter: Zuul
Branch: stable/pike

commit d0931c4e55ea5cabedcee70ec13d76f0dd5cc714
Author: Allain Legacy <email address hidden>
Date: Fri Nov 10 10:44:50 2017 -0600

    dhcp: serializing port delete and network rpc calls

    The port delete events are not synchronized with network rpc events. This
    creates a condition which makes it possible for a port delete event to be
    processed just before a previously started network query completes.

    The problematic order of operations is as follows:

      1) a network is scheduled to an agent; a network rpc is sent to the
         agent

      2) the agent queries the network data from the server

      3) while that query is in progress a port on that network is deleted; a
         port rpc is sent to the agent

      4) that port delete rpc is received before the network query rpc
         completes

      5) the port delete results in no action because the port was not present
         on the agent

      6) the network query finishes and adds the port to the cache (even
         though the port has already been deleted)

      7) some time passes and a new port is configured with the same IP
         address as the port that was deleted in (3)

      8) the dhcp host file is corrupted with 2 entries for the same IP
         address.

      9) dhcp queries for the newest port is rejected because of the duplicate
         entry in the dhcp host file.

    The solution is to add the network_id to the port_delete_end rpc event
    so that the _net_lock(network_id) synchronization point can be acquired
    so that it is processed serially with other network related events.

    To ensure backwards compatibility with newer agents running against older
    servers the determination of which network_id value to use in the lock is
    handled using a utility that will fallback to the previous mode of operation
    whenever the network_id attribute is not present in the *_delete_end RPC
    events. That utility can be removed in the future when it is guaranteed
    that the network_id attribute will be present in RPC messages from the
    server.

    Closes-Bug: #1732456

    Change-Id: I735f8b1c9248b12e5feb6cbe970cf67f321e6ebc
    Signed-off-by: Allain Legacy <email address hidden>
    (cherry picked from commit fa78b580105111b1238e5b18d415b08e8fb35d97)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.5 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/620682

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620682
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3af1f487a3ff307fb8d7c4b6c5083b2a089e9744
Submitter: Zuul
Branch: stable/ocata

commit 3af1f487a3ff307fb8d7c4b6c5083b2a089e9744
Author: Allain Legacy <email address hidden>
Date: Fri Nov 10 10:44:50 2017 -0600

    dhcp: serializing port delete and network rpc calls

    The port delete events are not synchronized with network rpc events. This
    creates a condition which makes it possible for a port delete event to be
    processed just before a previously started network query completes.

    The problematic order of operations is as follows:

      1) a network is scheduled to an agent; a network rpc is sent to the
         agent

      2) the agent queries the network data from the server

      3) while that query is in progress a port on that network is deleted; a
         port rpc is sent to the agent

      4) that port delete rpc is received before the network query rpc
         completes

      5) the port delete results in no action because the port was not present
         on the agent

      6) the network query finishes and adds the port to the cache (even
         though the port has already been deleted)

      7) some time passes and a new port is configured with the same IP
         address as the port that was deleted in (3)

      8) the dhcp host file is corrupted with 2 entries for the same IP
         address.

      9) dhcp queries for the newest port is rejected because of the duplicate
         entry in the dhcp host file.

    The solution is to add the network_id to the port_delete_end rpc event
    so that the _net_lock(network_id) synchronization point can be acquired
    so that it is processed serially with other network related events.

    To ensure backwards compatibility with newer agents running against older
    servers the determination of which network_id value to use in the lock is
    handled using a utility that will fallback to the previous mode of operation
    whenever the network_id attribute is not present in the *_delete_end RPC
    events. That utility can be removed in the future when it is guaranteed
    that the network_id attribute will be present in RPC messages from the
    server.

    Closes-Bug: #1732456

    Change-Id: I735f8b1c9248b12e5feb6cbe970cf67f321e6ebc
    (cherry picked from commit fa78b580105111b1238e5b18d415b08e8fb35d97)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ocata-eol release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/802687

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/802687
Committed: https://opendev.org/openstack/neutron/commit/cbef6cda98f2100a4080465dc99b1e82530e2f01
Submitter: "Zuul (22348)"
Branch: master

commit cbef6cda98f2100a4080465dc99b1e82530e2f01
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Jul 28 11:26:27 2021 +0000

    Add "network_id" to "_after_router_interface_deleted" payload

    This parameter, sent by the DHCP agent, is needed to remove the
    workaround method "_get_network_lock_id".

    The removal of this method will be done in [1] in Y release.

    Related-Bug: #1732456

    [1]https://review.opendev.org/c/openstack/neutron/+/800967

    Change-Id: Ibd7fed33d314e901c69da33f42029f8ea67df98d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/800967
Committed: https://opendev.org/openstack/neutron/commit/18c959bd24d8cdec5e55edeb61603834f3f00179
Submitter: "Zuul (22348)"
Branch: master

commit 18c959bd24d8cdec5e55edeb61603834f3f00179
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jul 15 16:07:37 2021 +0000

    Remove "_get_network_lock_id" compatibility method

    This method was introduced in [1] because not all the object
    payloads (port, network and subnet) had the "network_id" parameter.
    This parameter is now delivered in all payloads and this method
    is not needed anymore.

    Note: This patch must be merged in X+, not before.

    [1]https://review.opendev.org/c/openstack/neutron/+/524711

    Related-Bug: #1732456

    Change-Id: I92f1bb97552dd7af99dcda71919ed137a99202a2