When instance is migrated to another compute node, it's dhcp lease is not removed from the first compute node even after instance termination.
If a new instance got the same IP which was present in the previous instance created on the the first compute node where dhcp lease for this IP remains, then the dnsmasq refuse DHCP request of the IP address for a new instance with different MAC.
Steps to reproduce:
Scenario:
1. Create cluster (CentOS, nova-network with Flat-DHCP , Ceph for images and volumes)
2. Add 1 node with controller and ceph OSD roles
3. Add 2 node with compute and ceph OSD roles
4. Deploy the cluster
5. Create a VM
6. Wait until the VM got IP address via DHCP (in VM console log)
7. Migrate the VM to another compute node.
8. Terminate the VM.
9. Repeat stages from 5 to 8 several times (in my case - 4..6 times was enough) until a new instance stops receiving IP address via DHCP.
10. Check dnsmasq-dhcp.log (/var/log/daemon.log on the compute node) for messages like :
=============================================
2014-11-09T20:28:29.671344+00:00 warning: not using configured address 10.0.0.2 because it is leased to fa:16:3e:65:70:be
This means that:
I. An instance was created on the compute node-1 and got a dhcp lease:
==== nova-dhcpbridge.log
2014-11-09 20:12:03.811 27360 DEBUG nova.dhcpbridge [-] Called 'add' for mac 'fa:16:3e:65:70:be' with ip '10.0.0.2' main /usr/lib/python2.6/site-packages/nova/cmd/dhcpbridge.py:135
II. When the instance was migrating from compute node-1 to node-3, 'dhcp_release' was not performed on compute node-1, please check the time range in the logs : 2014-11-09 20:14:36-37
==== Running.log (node-1)
2014-11-09T20:14:36.647588+00:00 debug: cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf conntrack -D -r 10.0.0.2
### But there is missing a command like: sudo nova-rootwrap /etc/nova/rootwrap.conf dhcp_release br100 10.0.0.2 fa:16:3e:65:70:be
III. On the compute node-3, DHCP lease was added and it was successfully removed when the instance was terminated:
==== Running.log (node-3)
2014-11-09T20:15:17.250243+00:00 debug: cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf dhcp_release br100 10.0.0.2 fa:16:3e:65:70:be
IV. When an another instance got the same address '10.0.0.2' and was created on node-1, it didn't get IP address via DHCP:
==== Running.log (node-1)
2014-11-09T20:28:29.671344+00:00 warning: not using configured address 10.0.0.2 because it is leased to fa:16:3e:65:70:be
api: '1.0'
astute_sha: 3c374c9f7bfbdbcd7ce2f716cd704e3044e6fb41
auth_required: true
build_id: 2014-11-07_21-36-13
build_number: '84'
feature_groups:
- mirantis
fuellib_sha: c7b71bd1ee939b5a634715ac7b13c2936ad93a5e
fuelmain_sha: 77f6a3a4d398c62d89b2831cef2c4b47c2b2085e
nailgun_sha: 8330f6221e190db87fc5baa735fa719c85a2e02d
ostf_sha: 9c6fadca272427bb933bc459e14bb1bad7f614aa
production: docker
release: '6.0'
release_versions:
2014.2-6.0:
VERSION:
api: '1.0'
astute_sha: 3c374c9f7bfbdbcd7ce2f716cd704e3044e6fb41
build_id: 2014-11-07_21-36-13
build_number: '84'
feature_groups:
- mirantis
fuellib_sha: c7b71bd1ee939b5a634715ac7b13c2936ad93a5e
fuelmain_sha: 77f6a3a4d398c62d89b2831cef2c4b47c2b2085e
nailgun_sha: 8330f6221e190db87fc5baa735fa719c85a2e02d
ostf_sha: 9c6fadca272427bb933bc459e14bb1bad7f614aa
production: docker
release: '6.0'
Ubuntu also affected.