L2 gateway tunnel creation is failed when ODL is disconnected

[sunilcn]

Issue Description:
=================
When ODL is disconnected and created L2 gateway tunnel between DPN (i.e OVS) and TOR node then
after reconnecting the ODL, L2 tunnel is not established and reason for this l2-gateway-connection action remains in journal entry with "failed" state.

Configuration:
=============
Devstack enviornment-
 =>stable/pike version
 =>v2 driver is enabled

Openstack Controller& compute Node:10.164.224.235
Openstack Compute Node:10.164.224.230
ODL Controller Node:10.164.224.59 [ ODL-Oxyagen build]
TOR Node:10.164.224.59 [ simulated TOR switch]

Initial Condition:
==================
ODL is up and connected to Openstack control and compute node
VxLan Tunnel is up between Openstack Controller and Compute Node

Action:
=======
1. Create Network,Subnet and two neutron ports for TOR VMs

openstack network create mynet3
openstack subnet create --network mynet3 --subnet-range 16.16.16.0/24 mysubnet
openstack port create tor-port1 --network mynet3
openstack port create tor-port2 --network mynet3

openstack server create VM2 --flavor myhuge --image cirros-0.3.4-x86_64-uec --network net80 --availability-zone nova:controller-suneel

openstack server create VM3 --flavor myhuge --image cirros-0.3.4-x86_64-uec --network net80 --availability-zone nova:controller-suneel

2. On TOR node, launch VMs using following commands,

#./vm1 <mac address of tor-port1> <tap port id>
#./vm2 <mac address of tor-port2> <tap port id>

FYI,
a) the contents of vm1 file
sudo /usr/bin/qemu-system-x86_64 -name vm1 -hda disk.raw -m 1024 -cdrom /root/ZeroShell-3.5.0.iso -boot d -net nic -netdev type=tap,id=hostnet1,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown,ifname=tap$1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=$2,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off -nographic

b) the contents of vm2 file
sudo /usr/bin/qemu-system-x86_64 -name vm2 -hda disk.raw -m 1024 -cdrom /root/ZeroShell-3.5.0.iso -boot d -net nic -netdev type=tap,id=hostnet1,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown,ifname=tap$1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=$2,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off -nographic

3. Create L2 gateway tunnel between DPN and TOR

neutron l2-gateway-create --device name=br-tor,interface_names="tapf0b2e96c-4f;tapca5b83b8-70" gw1
         neutron l2-gateway-connection-create --default-segmentation-id 0 gw1 mynet3

4. on TOR VMs, get the IP for interface ETH00 using following command
>dhclient ETH00

5. Ping between VMs

6. Now delete the L2 gateway tunnel created above

neutron l2-gateway-connection-delete <id of connection>
neutron l2-gateway-delete <id of gateway>

7. Disconnect the ODL (i.e logout from karaf)

8. Recreate L2 gateway tunnel between DPN and TOR

neutron l2-gateway-create --device name=br-tor,interface_names="tapf0b2e96c-4f;tapca5b83b8-70" gw1
         neutron l2-gateway-connection-create --default-segmentation-id 0 gw1 mynet3

9. Reconnect the ODL (i.e restart the karaf)

10. Verify the sync between ODL data store and Openstack Database

Results:
=======
5. Ping is successful between TOR and DPN VM
6. Delete of tunnel is successful.
8. Creation of tunnel is successful on openstack and no error is seen.
9. ODL is connected back
10. only following l2-gateway create action is pushed to ODL and "neutron l2-gateway-connection" action is still present in journal entry with "failed" state.

neutron l2-gateway-create --device name=br-tor,interface_names="tapf0b2e96c-4f;tapca5b83b8-70" gw1

here is snap shot of journal entry for above action, this entry will remain in database forever.
[‎08-‎06-‎2018 11:01]

No Title
mysql> select * from opendaylightjournal;
+--------+----------------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+-------------+---------------------+---------------------+------------+
| seqnum | object_type | object_uuid | operation | data | state | retry_count | created_at | last_retried | version_id |
+--------+----------------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+-------------+---------------------+---------------------+------------+
| 9564 | l2gateway_connection | 2a33942b-2333-45c0-ac05-220d9637dc3c | create | }q(U
gateway_idqX$ d4b2efac-0ead-4759-9884-48967dffa73cqU
network_idqX$ be5b1686-895c-4037-8a25-055c944229d8qU tenant_idqX 8be57b61954d4fcb8c00ac25e2e65dfdqUsegmentation_idX 0Uidq U$2a33942b-2333-45c0-ac05-220d9637dc3cq
u. | failed | 5 | 2018-06-08 00:47:22 | 2018-06-08 00:48:04 | 13 |
+--------+----------------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+-------------+---------------------+---------------------+------------+
1 row in set (0.00 sec)

mysql> quit

Logs:
====
q-svc.log
openstack configuration files local.conf and neutron.conf