Neutron server failed to send port notifications after dhcp agent restart

[Aviram Bar-Haim]

After restarting dhcp server, dnsmasq host file is not updated with new ports (so VMs fails to get IP).
from neutron server LOG:
2015-04-30 08:31:30.673 5320 WARNING neutron.scheduler.dhcp_agent_scheduler [req-40fc7074-aa77-4e4a-ad89-4897ecf2027e None] No more DHCP agents

After restarting neutron server, ports are being updated again and DHCP works.

Reproduced over Ubuntu, Neutron with VLAN and OVS agent.

Fuel ISO:
{"build_id": "2015-04-13_10-39-24", "ostf_sha": "c2a76a60ec4ebbd78e508216c2e12787bf25e423", "build_number": "169", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-13_10-39-24", "ostf_sha": "c2a76a60ec4ebbd78e508216c2e12787bf25e423", "build_number": "169", "api": "1.0", "nailgun_sha": "0435d55f252796e77791c62405af8f341338ae73", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c", "astute_sha": "d96a80b63198a578b2c159edbd76048819039eb0", "feature_groups": ["experimental"], "release": "6.1", "fuelmain_sha": "9bdbd5018ed8fb6ff8af5fadadc49969d7e58a13", "fuellib_sha": "7d1e708297bd8fb4619210779f93d00201c95bba"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "0435d55f252796e77791c62405af8f341338ae73", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c", "astute_sha": "d96a80b63198a578b2c159edbd76048819039eb0", "feature_groups": ["experimental"], "release": "6.1", "fuelmain_sha": "9bdbd5018ed8fb6ff8af5fadadc49969d7e58a13", "fuellib_sha": "7d1e708297bd8fb4619210779f93d00201c95bba"}

[Ilya Shakhat (shakhat)]

Please attach logs and/or diagnostic snapshot

[Aviram Bar-Haim (aviramb)]

Seems like there is an unmanaged neutron-dhcp-agent process running in background - is this intentional?
After changing configurations in /etc/neutron/dhcp_agent.ini, shout I restart neutron dhcp using neutron-dhcp-agent agent or with other HA tool?

After installation I see this:
[root@node-24 ~]# /etc/init.d/neutron-dhcp-agent status
neutron-dhcp-agent is stopped

[root@node-24 ~]# ps aux |grep neutron-dhcp-agent
neutron 32488 0.7 0.1 178076 39060 ? S 12:23 0:01 /usr/bin/python /usr/bin/neutron-dhcp-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/dhcp_agent.ini --log-file=/var/log/neutron/dhcp-agent.log
root 47056 0.0 0.0 103244 888 pts/0 S+ 12:26 0:00 grep neutron-dhcp-agent

[root@node-24 ~]# /etc/init.d/neutron-dhcp-agent start
Starting neutron-dhcp-agent: [ OK ]

[root@node-24 ~]# ps -ef |grep neutron-dhcp-agent
neutron 21183 1 31 13:23 ? 00:00:01 /usr/bin/python /usr/bin/neutron-dhcp-agent --log-file /var/log/neutron/dhcp-agent.log --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini
root 21501 38350 0 13:24 pts/0 00:00:00 grep neutron-dhcp-agent
neutron 32488 1 0 12:23 ? 00:00:07 /usr/bin/python /usr/bin/neutron-dhcp-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/dhcp_agent.ini --log-file=/var/log/neutron/dhcp-agent.log

[root@node-24 ~]# /etc/init.d/neutron-dhcp-agent stop
Stopping neutron-dhcp-agent: [ OK ]

[root@node-24 ~]# /etc/init.d/neutron-dhcp-agent status
neutron-dhcp-agent is stopped

[root@node-24 ~]# ps -ef |grep neutron-dhcp-agent
root 23059 38350 0 13:24 pts/0 00:00:00 grep neutron-dhcp-agent
neutron 32488 1 0 12:23 ? 00:00:07 /usr/bin/python /usr/bin/neutron-dhcp-agent --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/dhcp_agent.ini --log-file=/var/log/neutron/dhcp-agent.log

After I kill all neutron-dhcp-agent processes and then restart/start neutron-dhcp-agent, I (still) don't see any problems..

ISO:
{"build_id": "2015-04-29_07-55-19", "build_number": "361", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-04-29_07-55-19", "build_number": "361", "api": "1.0", "fuel-library_sha": "0e5b82d24853304befb22145ac4aaf3545d295e1", "nailgun_sha": "e660b1c09d7d4d07bdd48d424ce9aed3b6facd6e", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "8cd6cf575d3c101dee1032abb6877dfa8487e077", "astute_sha": "04ebab96d57b0e8acbf2d7f3ba05e4fbf31b741e", "fuel-ostf_sha": "b38602c841deaa03ddffc95c02f319360462cbe3", "release": "6.1", "fuelmain_sha": "ee112acfdd0f9017ef40be53e8e51bb5c429e97c"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "0e5b82d24853304befb22145ac4aaf3545d295e1", "nailgun_sha": "e660b1c09d7d4d07bdd48d424ce9aed3b6facd6e", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "8cd6cf575d3c101dee1032abb6877dfa8487e077", "astute_sha": "04ebab96d57b0e8acbf2d7f3ba05e4fbf31b741e",...

[Vladimir Kuklin (vkuklin)]

Aviram, this is not a bug at all - you should not use init scripts to restart pacemaker-controlled services. Please use corresponding pacemaker utilities, eg `crm resource restart p_neutron-dhcp-agent`