Multiple dhcp agents are scheduled to host one network automatically if multiple subnets are created at the same time

[Lujin Luo]

I have three all-in-one controller nodes deployed by DevStack with the latest codes. Neutron servers on these controllers are set behind Pacemaker and HAProxy to realize active/active HA. MariaDB Galera cluster is used as my database backend.

In neutron.conf, I have made the following changes:
dhcp_agents_per_network = 1
network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.ChanceScheduler

Since I only allow one dhcp agent per tenant on each controller, now I have three dhcp agents in total for a given tenant. After I created one network within this given tenant, before I add any subnets to this network, no dhcp agents would be scheduled to host this network. If I run multiple commands at the same time to add subnets to the network, we may end up with more than one dhcp agents hosting the network.

It is not easy to re-produce the bug. You might need to repeat the following steps multiple times.

How to reproduce:

Prerequisite
make the following changes in neutron.conf
[DEFAULT]
dhcp_agents_per_network = 1
network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.ChanceScheduler

Step 1: Confirm multiple dhcp agents are running
$ neutron agent-list --agent_type='DHCP agent'
my result is shown http://paste.openstack.org/show/483956/

Step 2: Create a network
$ neutron net-create net-dhcptest

Step 3: Create multiple subnets on the network at the same time
On controller1:
$ neutron subnet-create --name subnet-dhcptest-1 net-dhcptest 192.162.101.0/24
On controller2:
$ neutron subnet-create --name subnet-dhcptest-2 net-dhcptest 192.162.102.0/24

Step 4: Check which dhcp agent(s) is/are hosting the network
$ neutron dhcp-agent-list-hosting-net net-dhcptest
my result is shown http://paste.openstack.org/show/483958/

If you end up with only one dhcp agent, please delete the subnets and network. Then repeat Step 1-4 several times.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/288271

[Armando Migliaccio (armando-migliaccio)]

This bug is > 180 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Lujin Luo (luo-lujin)]

I will fix this one. Sorry for the late notice.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/288271
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Oleg Bondarev (obondarev)]

Increasing the priority as over-scheduling leads to extra IP address consumption by additional DHCP ports, which might be critical for small subnets.

[Bence Romsics (bence-romsics)]

Our users reported problems related to this bug so I worked on it and managed to reproduce it.

The basic trick is to inject a sufficiently long sleep at the right place in the code to make the race condition worse and easier to trigger. Here's how:

* You need a multi-host devstack: at least one all-in-one host (below: devstack0) and one compute host (below: devstack0a). Make sure both hosts run a fully functional dhcp-agent.

* Use ChanceScheduler for dhcp-agent scheduling. The problem is reproducible with other schedulers too, but it's much harder. With ChanceScheduler it's relatively easy to get multiple concurrent shcedulings produce different agents. With other almost deterministic schedulers like WeightScheduler this is much harder.

neutron.conf
[agent]
network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.ChanceScheduler

* Make sure neutron is able to process at least two API requests concurrently. For example by setting 'api_workers' to at least 2. The default value of 'api_workers' is okay on any multiprocessor system.

* Use the old neutron client for manual agent scheduling operations. I found multiple silent bugs in osc's agent scheduling operations - which I will report in other bug reports in the coming days.

neutron dhcp-agent-list-hosting-net NETWORK
neutron dhcp-agent-network-add AGENT NETWORK
neutron dhcp-agent-network-remove AGENT NETWORK

* The relevant neutron configuration in my devstack was this:

network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.ChanceScheduler
network_auto_schedule = True
allow_automatic_dhcp_failover = True
dhcp_agents_per_network = 1
enable_services_on_agents_with_admin_state_down = False

* Versions:

devstack 1f6bea17
neutron 10c5f451ce (plus two additional lines of code described below)

* Agent re-scheduling can be triggered by 'subnet create's or agents going to status=DOWN. Since the former is simpler to do we'll use that to trigger re-schedules here.

* Add a sleep() to the line before the call to bind() here:

https://opendev.org/openstack/neutron/src/commit/a309dee7c58ef43d5985fb2e84b839134bca6b9c/neutron/scheduler/base_scheduler.py#L52

Like this:

 50 chosen_agents = self.select(plugin, context, hostable_agents,
 51 hosted_agents, num_agents)
 52 import time
 53 time.sleep(30)
 54 # bind the resource to the agents
 55 self.resource_filter.bind(context, chosen_agents, resource['id'])

* Pick or create a network with a subnet, here we'll use the 'private' network pre-created by devstack.

* De-schedule all agents from the network. For example:

# check which agents host the network's dhcp:
neutron dhcp-agent-list-hosting-net private
# de-schedule them
neutron dhcp-agent-network-remove "$( openstack network agent list --agent-type dhcp --host devstack0 -f value -c ID )" private
neutron dhcp-agent-network-remove "$( openstack network agent list --agent-type dhcp --host devstack0a -f value -c ID )" private

Now this network has 0 dhcp servers which is below the targetted 1 dhcp server.

* Trigger two re-schedules concurrently - that is in quick succession compared ...

[Oleg Bondarev (obondarev)]

>>I don't understad why change #288271 does not make the original agent scheduling transactional but instead chooses to do a later cleanup of over-scheduled agents.

@bence-romsics not sure where change #288271 does a later cleanup of over-scheduled agents. Currently it uses binding index approach which has beed used for HA router scheduling for a while.

[Bence Romsics (bence-romsics)]

@Oleg: Sorry, I must have read some old patch set or I don't know where did I get that idea. I hope to spare some time this week to test #288271 with the reproduction above. Will comment here when I have the outcome.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/288271
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=69b3762dda47272513e02340e7942d5a39f825c5
Submitter: Zuul
Branch: master

commit 69b3762dda47272513e02340e7942d5a39f825c5
Author: Lujin <email address hidden>
Date: Thu Mar 16 15:42:47 2017 +0900

    Add binding_index to NetworkDhcpAgentBinding

    The patch proposes adding a new binding_index to the
    NetworkDhcpAgentBinding table, with an additional Unique
    Constraint that enforces a single <network_id, binding_index>
    per network.

    1. When a network is triggered to be auto-scheduled to DHCP
    agents, the number of DHCP agents is constrained by
    dhcp_agents_per_network in neutron.conf. This prevents
    too many DHCP agents from being scheduled in the first place.

    2. If users manually schedule a network to specific DHCP
    agents, the binding_index increments to show the number of
    DHCP agents hosting this network.

    Co-Authored-By: Oleg Bondarev <email address hidden>
    Change-Id: I1bc3f8b69c337f7c1cf7375509a0da61def9baf1
    Closes-Bug: #1535554

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/691641

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lajos Katona (<email address hidden>) on branch: master
Review: https://review.opendev.org/691641
Reason: The problematic patch is reverted:
https://review.opendev.org/691710

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/691852

[OpenStack Infra (hudson-openstack)]

Change abandoned by Oleg Bondarev (<email address hidden>) on branch: master
Review: https://review.opendev.org/691852
Reason: Not needed

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/683987
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0041860e97cbf34fd652224b3355818bf4a681f3
Submitter: Zuul
Branch: master

commit 0041860e97cbf34fd652224b3355818bf4a681f3
Author: elajkat <email address hidden>
Date: Fri Sep 6 16:06:32 2019 +0200

    HA race condition test for DHCP scheduling

    Add fullstack testcase for DHCP HA scheduling with fake scheduler
    that selects new agent with some extra sleep for every scheduling.

    Change-Id: I1045992ce9a18b37dd7dcdb46063698fad983932
    Related-Bug: #1535554

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.