dhcp-agent can not interact with neutron-server after neutron-server restarted

[LIU Yulong]

Recently we meet a strange issue between neutron-server and neutron-dhcp-agents. In a long run deployment, we just restart all neutron-servers, then we failed to boot VM. Yes, it is vif-plug-timeout!!! We noticed that the DHCP provisioningblock was not deleted.

Our operator gives the recover steps:
1. stop the network's scheduled dhcp-agents
2. delete the related Queues in rabbitMQ of these dhcp-agents
3. start dhcp-agents.

Then everything works fine.

So, do you have any idea about such issue?

[Brian Haley (brian-haley)]

Can you give any more info on the issue? Version being used? Traceback? etc.

I will try and recreate this on master, but cannot say I have seen this before.

[LIU Yulong (dragon889)]

These is no traceback no ERROR log at all. After restart neutron-servers, the agent list shows all dhcp-agents are active. That means some queues of the agents can work as normal, but some not.

"dhcp_ready_on_ports" is the final RPC call of the dhcp-agent to neutron-server which will remove the provisioningblock finally. The issue is that neutron-server can not get this message. Or the DHCP-agent can not send message to the related queue.

So, I guess this issue may reside in oslo-messaging or rabbitMQ itself.

Our rabbitMQ is a 3-nodes cluster. The rabbitMQ release version is rabbitmq-server-3.6.16-1.el7.

oslo-messaging is 8.1.4-1.el7.noarch.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]