VxLan tunnels IPs are not updated

[Nikita Koshikov]

This is MOS7.0+MU3

Steps to reproduce:
1)Deploy environment with tunnel segmentation
2)Make sure you have few computes
3)Delete compute from environment
4)Add the same computes(with the same name) to environment
5)Try to boot VM

Actual result - VM boot failed, in neutron logs there are messages, like:
<163>Jun 3 14:07:06 controller-1 neutron-server 2016-06-03 14:07:06.208 15217 ERROR oslo_messaging.rpc.dispatcher [
req-6de6e70a-f73a-4b65-bf61-b8fd6f1c69e8 ] Exception during message handling: Invalid input for operation: (u'Tunnel
 IP %(ip)s in use with host %(host)s', {'ip': u'10.200.240.11', 'host': u'uslv-pvrt-opn29.domain.tld'}).
2016-06-03 14:07:06.208 15217 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-06-03 14:07:06.208 15217 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_mess
aging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-06-03 14:07:06.208 15217 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-06-03 14:07:06.208 15217 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_mess
aging/rpc/dispatcher.py", line 186, in _dispatch
2016-06-03 14:07:06.208 15217 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-06-03 14:07:06.208 15217 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_mess
aging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-06-03 14:07:06.208 15217 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-06-03 14:07:06.208 15217 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/neutron/p
lugins/ml2/drivers/type_tunnel.py", line 248, in tunnel_sync
2016-06-03 14:07:06.208 15217 TRACE oslo_messaging.rpc.dispatcher raise exc.InvalidInput(error_message=msg)
2016-06-03 14:07:06.208 15217 TRACE oslo_messaging.rpc.dispatcher InvalidInput: Invalid input for operation: (u'Tunn
el IP %(ip)s in use with host %(host)s', {'ip': u'10.200.240.11', 'host': u'uslv-pvrt-opn29.domain.tld'}).
2016-06-03 14:07:06.208 15217 TRACE oslo_messaging.rpc.dispatcher
<163>Jun 3 14:07:06 controller-1 neutron-server 2016-06-03 14:07:06.209 15217 ERROR oslo_messaging._drivers.common
[req-6de6e70a-f73a-4b65-bf61-b8fd6f1c69e8 ] Returning exception Invalid input for operation: (u'Tunnel IP %(ip)s in
use with host %(host)s', {'ip': u'10.200.240.11', 'host': u'uslv-pvrt-opn29.domain.tld'}). to caller
<163>Jun 3 14:07:06 controller-1 neutron-server 2016-06-03 14:07:06.209 15217 ERROR oslo_messaging._drivers.common
[req-6de6e70a-f73a-4b65-bf61-b8fd6f1c69e8 ] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dis
t-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File
"/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\
n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n resu
lt = func(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/type_tunnel.py
", line 248, in tunnel_sync\n raise exc.InvalidInput(error_message=msg)\n', "InvalidInput: Invalid input for oper
ation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'10.200.240.11', 'host': u'uslv-pvrt-opn29.domain.tld'
}).\n"]

If we grab the data from mysql:
use neutron;
select * from ml2_vxlan_endpoints;
The stale data exists, with old IPs. Fuel serialize new IPs for computes, but they can't be updated.

We fix this by cleaning ml2_vxlan_endpoints table and restarting neutron-plugin-openvswitch-agent on all redeployed/added computes.

root@controller-1:~# dpkg -l | grep neutr
ii neutron-common 1:2015.1.1-1~u14.04+mos5355 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 1:2015.1.1-1~u14.04+mos5355 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 1:2015.1.1-1~u14.04+mos5355 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-metadata-agent 1:2015.1.1-1~u14.04+mos5355 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-plugin-ml2 1:2015.1.1-1~u14.04+mos5355 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-plugin-openvswitch-agent 1:2015.1.1-1~u14.04+mos5355 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-server 1:2015.1.1-1~u14.04+mos5355 all Neutron is a virtual network service for Openstack - server
ii python-neutron 1:2015.1.1-1~u14.04+mos5355 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 2015.1.1-1~u14.04+mos634 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutronclient 1:2.4.0-1~u14.04+mos474 all client - Neutron is a virtual network service for Openstack

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Nikita Koshikov (nkoshikov)]

I can confirm that fix from https://bugs.launchpad.net/neutron/+bug/1464178 works. I had to rebase it on Kilo's code base, but functionality is good.

How it was tested:

1) UPDATE ml2_vxlan_endpoints set ip_address='1.1.1.1' where host='x.x.x.x';
2) SELECT * from ml2_vxlan_endpoints where host='x.x.x.x';

restart neutron-openvswitch-agent on host x.x.x.x

3) SELECT * from ml2_vxlan_endpoints where host='x.x.x.x';
It shows now correct ip address.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Miguel Angel Ajo <email address hidden>
Review: https://review.fuel-infra.org/22123

[Ekaterina Shutova (eshutova)]

Verified on MOS 7.0 + MU6 updates.

Problem was reproduced with next scenario:
1)Deploy environment with tunnel segmentation
2)Make sure you have few computes
3)Delete compute from environment
4)Add one compute with different name
5)Add compute with the same name from step3 to environment
6)Check neutron logs
7)Boot VMs

Before there were errors in log:
/var/log/neutron/server.log:2016-10-03 13:47:20.537 23960 ERROR oslo_messaging._drivers.common [req-ac9476f9-8bdf-4912-a6be-3d174852c3c1 ] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/type_tunnel.py", line 248, in tunnel_sync\n raise exc.InvalidInput(error_message=msg)\n', "InvalidInput: Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.2.3', 'host': u'compute-2.test.domain.local'}).\n"]

With fix there is no errors, VMs are bootes successfully.