Comment 3 for bug 1689880

I just ran into this issue on a brand new ocata deployment made from scratch, followed official documentation. It is a bare min deployment for proof of concept.

https://docs.openstack.org/networking-ovn/ocata/install.html

1 - controller
2 - compute nodes

Using neutron-ovn-db-sync-util or "neutron_sync_mode = repair" breaks floating ips and gateways, even when deleting ovnsb_db.db first. The only fix is to do the following steps per external network that you want to fix:
1) disassociate all floating IPs (release not required)
2) clear router's external gateway
3) re-add router's external gateway - all vms now have full connectivity
4) re-attach floating ips

Things I noticed after neutron-ovn-db-sync-util is used or neutron-server is restarted with repair mode:
* Instances can still talk to each other and ping their gateways even, but no external access
* Even after things break I can sometimes still ping the router gateway IPs from other machines
* All of my associated floating IPs that stopped working show up in "ovn-nbctl show" as switch ports, but when I do the 4 steps above to fix everything they no longer show up - see below

floating IPS were 10.0.0.226 10.0.0.229, along with 2 vms without floating ips.

BROKEN:
[root@controller ~(keystone_admin)]$ ovn-nbctl show | grep 10.200.80
            addresses: ["fa:16:3e:b1:55:26 10.0.0.227"]
            addresses: ["fa:16:3e:9b:32:fe 10.0.0.226"]
            addresses: ["fa:16:3e:78:5a:04 10.0.0.229"]
            networks: ["10.200.0.227/24"]

WORKING:
[root@controller ~(keystone_admin)]$ ovn-nbctl show | grep 10.200.80
            addresses: ["fa:16:3e:b1:55:26 10.0.0.227"]
            networks: ["10.0.0.227/24"]

* When trying the repair in repair mode, I always get the same messages, they are never "fixed"

1st time
[root@controller openvswitch]# neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair
...
2017-10-04 18:06:26.350 6923 WARNING networking_ovn.ovn_db_sync [req-4cebd9e8-a4f1-438f-8d1e-370804eb9ca5 - - - - -] Router found in OVN but not in Neutron, router id=ogr-8e22bbc8-bd0c-4018-96de-ac289619ee91
2017-10-04 18:06:26.350 6923 WARNING networking_ovn.ovn_db_sync [req-4cebd9e8-a4f1-438f-8d1e-370804eb9ca5 - - - - -] Deleting the router ogr-8e22bbc8-bd0c-4018-96de-ac289619ee91 from OVN NB DB
2017-10-04 18:06:26.351 6923 WARNING networking_ovn.ovn_db_sync [req-4cebd9e8-a4f1-438f-8d1e-370804eb9ca5 - - - - -] Router found in OVN but not in Neutron, router id=ogr-2b37fdc2-64da-4521-9dc8-5bc5e5ea158d
2017-10-04 18:06:26.351 6923 WARNING networking_ovn.ovn_db_sync [req-4cebd9e8-a4f1-438f-8d1e-370804eb9ca5 - - - - -] Deleting the router ogr-2b37fdc2-64da-4521-9dc8-5bc5e5ea158d from OVN NB DB
2017-10-04 18:06:26.352 6923 WARNING networking_ovn.ovn_db_sync [req-4cebd9e8-a4f1-438f-8d1e-370804eb9ca5 - - - - -] Router found in OVN but not in Neutron, router id=ogr-ba2e6a92-76e8-44e1-9504-5839a3d0fe7f
2017-10-04 18:06:26.352 6923 WARNING networking_ovn.ovn_db_sync [req-4cebd9e8-a4f1-438f-8d1e-370804eb9ca5 - - - - -] Deleting the router ogr-ba2e6a92-76e8-44e1-9504-5839a3d0fe7f from OVN NB DB
2017-10-04 18:06:26.354 6923 INFO networking_ovn.cmd.neutron_ovn_db_sync_util [req-4cebd9e8-a4f1-438f-8d1e-370804eb9ca5 - - - - -] Sync completed

2nd time
[root@controller openvswitch]# neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair
...
2017-10-04 18:06:52.180 6942 INFO networking_ovn.cmd.neutron_ovn_db_sync_util [req-8d41840f-dcd3-426b-84a8-699c020b6f14 - - - - -] Syncing the networks and ports with mode : repair
2017-10-04 18:06:53.677 6942 WARNING networking_ovn.ovn_db_sync [req-8d41840f-dcd3-426b-84a8-699c020b6f14 - - - - -] Router found in OVN but not in Neutron, router id=ogr-8e22bbc8-bd0c-4018-96de-ac289619ee91
2017-10-04 18:06:53.678 6942 WARNING networking_ovn.ovn_db_sync [req-8d41840f-dcd3-426b-84a8-699c020b6f14 - - - - -] Deleting the router ogr-8e22bbc8-bd0c-4018-96de-ac289619ee91 from OVN NB DB
2017-10-04 18:06:53.678 6942 WARNING networking_ovn.ovn_db_sync [req-8d41840f-dcd3-426b-84a8-699c020b6f14 - - - - -] Router found in OVN but not in Neutron, router id=ogr-2b37fdc2-64da-4521-9dc8-5bc5e5ea158d
2017-10-04 18:06:53.679 6942 WARNING networking_ovn.ovn_db_sync [req-8d41840f-dcd3-426b-84a8-699c020b6f14 - - - - -] Deleting the router ogr-2b37fdc2-64da-4521-9dc8-5bc5e5ea158d from OVN NB DB
2017-10-04 18:06:53.679 6942 WARNING networking_ovn.ovn_db_sync [req-8d41840f-dcd3-426b-84a8-699c020b6f14 - - - - -] Router found in OVN but not in Neutron, router id=ogr-ba2e6a92-76e8-44e1-9504-5839a3d0fe7f
2017-10-04 18:06:53.679 6942 WARNING networking_ovn.ovn_db_sync [req-8d41840f-dcd3-426b-84a8-699c020b6f14 - - - - -] Deleting the router ogr-ba2e6a92-76e8-44e1-9504-5839a3d0fe7f from OVN NB DB
2017-10-04 18:06:53.681 6942 INFO networking_ovn.cmd.neutron_ovn_db_sync_util [req-8d41840f-dcd3-426b-84a8-699c020b6f14 - - - - -] Sync completed

* Lastly, I tried deleting the ovnnb_db.db and trying to sync, it finished syncing but doing anything in openstack caused neutron-server errors saying the switch/router/port IDs were not matching up in neutrons database and the ovnnb_db.db

Is it safe to build this cluster with neutron_sync_mode = off ?