[dvr] Routing fails after resetting compute nodes

[Georgy Dyuldin]

Detailed bug description:

There is no routing between two vms booted on different computes on different networks connnected with distributed router.

Steps to reproduce:

1. Deploy MOS 9.0 with 3 controllers, 2 computes, 1 cinder, with enabled DVR
2. Create net01, subnet net01__subnet for it
3. Create net02, subnet net02__subnet for it
4. Create router01 with type Distributed and connect it to external network
5. Add interfaces to the router01 with net01_subnet and net02_subnet
6. Boot vm_1 in the net01
7. Boot vm_2 in the net02 on different compute
8. Go to vm_1 and check ping vm_2
9. Reset computers on which vm_1 and vm_2 are
10. Wait some time while computers are resetting
11. Go to vm_2 and check ping vm_1

Expected results:

All steps should pass

Actual result:

There is no ping between vm_1 and vm_2 on last step (during about 15 minutes)

Reproducibility:

This test fails in more than 50% of cases

Description of the environment:

- Network model: VLAN+DVR, VxLAN+L2pop+DVR
- Version(from /api/v1/version) {"release": "9.0", "auth_required": true, "api": "1", "openstack_version": "mitaka-9.0", "feature_groups": []}
- Build id: #420 and earliest

Additional information:

Both vm are pingable from compute with "ip net e qdhcp-<net-uuid> ping <vm fixed ip>". Also i can connect to both with ssh (with ip net ... too).
Ping doesn't work in any direction (from vm1 to vm2 and from vm2 to vm1).
All neutron agents are alive:

root@node-2:~# neutron agent-list
+--------------------------------------+--------------------+--------------------------+-------------------+-------+----------------+---------------------------+
| id | agent_type | host | availability_zone | alive | admin_state_up | binary |
+--------------------------------------+--------------------+--------------------------+-------------------+-------+----------------+---------------------------+
| 033176c1-7701-4fc6-8ffe-cbb10a29c3be | Open vSwitch agent | node-4.test.domain.local | | :-) | True | neutron-openvswitch-agent |
| 08ec3c1e-b23d-4b5b-a1d2-5d776f97a2a9 | Metadata agent | node-5.test.domain.local | | :-) | True | neutron-metadata-agent |
| 17d5d009-2b3f-45e1-985c-4ee3bcb179e4 | DHCP agent | node-4.test.domain.local | nova | :-) | True | neutron-dhcp-agent |
| 288d1be3-5b95-4c65-bd8d-a342ac2b3a26 | Open vSwitch agent | node-2.test.domain.local | | :-) | True | neutron-openvswitch-agent |
| 33461df5-70a2-498a-b07f-8d7f82dd08b8 | L3 agent | node-3.test.domain.local | nova | :-) | True | neutron-l3-agent |
| 39655594-f9ff-4e54-88a1-7cae9426e141 | L3 agent | node-5.test.domain.local | nova | :-) | True | neutron-l3-agent |
| 40fc1620-0eef-4e12-98a8-9be535e12f2c | Metadata agent | node-4.test.domain.local | | :-) | True | neutron-metadata-agent |
| 475336e6-ce8d-439b-8594-b96259a11d9e | DHCP agent | node-3.test.domain.local | nova | :-) | True | neutron-dhcp-agent |
| 4ff3ec23-640f-4bba-ac57-fc89451a98b4 | L3 agent | node-2.test.domain.local | nova | :-) | True | neutron-l3-agent |
| 8b80dd41-d35f-449f-80a9-20cb0b85682f | Open vSwitch agent | node-1.test.domain.local | | :-) | True | neutron-openvswitch-agent |
| 982edae5-4242-47e0-a7f7-98a3c1d45571 | Open vSwitch agent | node-5.test.domain.local | | :-) | True | neutron-openvswitch-agent |
| c709c81d-be1f-48ab-ad2c-440ebddd176e | Metadata agent | node-2.test.domain.local | | :-) | True | neutron-metadata-agent |
| ca6f4e89-3d64-470c-8bb6-c7b22e976157 | Metadata agent | node-3.test.domain.local | | :-) | True | neutron-metadata-agent |
| e43da897-512e-4d01-9bdb-ffdb1d54cb8f | DHCP agent | node-2.test.domain.local | nova | :-) | True | neutron-dhcp-agent |
| e80efd70-3109-4c0b-be7e-e9e93dc18147 | Open vSwitch agent | node-3.test.domain.local | | :-) | True | neutron-openvswitch-agent |
| f44ca8bf-ae34-4783-ae8c-44a837f77385 | L3 agent | node-1.test.domain.local | nova | :-) | True | neutron-l3-agent |
| f75857f2-d839-49a3-b818-b716d4bdae0b | L3 agent | node-4.test.domain.local | nova | :-) | True | neutron-l3-agent |
| fd53543c-5a6d-4093-86c1-fac7d6c588b0 | Metadata agent | node-1.test.domain.local | | :-) | True | neutron-metadata-agent |
+--------------------------------------+--------------------+--------------------------+-------------------+-------+----------------+---------------------------+

Router is in ACTIVE status:

root@node-2:~# neutron router-show a5658dd1-0d45-4818-9543-b303662dec5d
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| availability_zone_hints | |
| availability_zones | nova |
| description | |
| distributed | True |
| external_gateway_info | {"network_id": "7189500c-c9a9-46b9-9c0a-854ddded1937", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "d1e6bf6a-e3de-44b3-b554-05216e570dfa", "ip_address": "10.109.16.133"}]} |
| ha | False |
| id | a5658dd1-0d45-4818-9543-b303662dec5d |
| name | router01 |
| routes | |
| status | ACTIVE |
| tenant_id | aa717ab5c37640d18acc9490c57dacb7 |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[Oleg Bondarev (obondarev)]

For some reason router-distributed port was reattached to OVS:

2016-05-31 11:45:09.503 5652 DEBUG neutron.agent.linux.async_process [-] Output received from [ovsdb-client monitor Interface name,ofport,external_ids --format=json]: {"data":[["f0a84715-d2fa-419f-8944-bae9a466cf6b","delete","qr-5e420f53-c3",12,["map",[["attached-mac","fa:16:3e:ef:6e:df"],["iface-id","5e420f53-c356-4977-94fa-389db3f612c7"],["iface-status","active"]]]]],"headings":["row","action","name","ofport","external_ids"]} _read_stdout /usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py:236
2016-05-31 11:45:09.679 5652 DEBUG neutron.agent.linux.async_process [-] Output received from [ovsdb-client monitor Interface name,ofport,external_ids --format=json]: {"data":[["7c6bd375-2231-4e82-b230-3073919455e5","insert","qr-5e420f53-c3",["set",[]],["map",[["attached-mac","fa:16:3e:ef:6e:df"],["iface-id","5e420f53-c356-4977-94fa-389db3f612c7"],["iface-status","active"]]]]],"headings":["row","action","name","ofport","external_ids"]} _read_stdout /usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py:236
2016-05-31 11:45:09.693 5652 DEBUG neutron.agent.linux.async_process [-] Output received from [ovsdb-client monitor Interface name,ofport,external_ids --format=json]: {"data":[["7c6bd375-2231-4e82-b230-3073919455e5","old",null,["set",[]],null],["","new","qr-5e420f53-c3",4,["map",[["attached-mac","fa:16:3e:ef:6e:df"],["iface-id","5e420f53-c356-4977-94fa-389db3f612c7"],["iface-status","active"]]]]],"headings":["row","action","name","ofport","external_ids"]}

Due to bug https://bugs.launchpad.net/mos/+bug/1577721 (which is waiting to be merged to stable/mitaka and then synced to MOS) the port was added and then deleted:

2016-05-31 11:45:19.884 5652 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-9c4d2be4-2b85-4275-94ce-5a44bc5f99e9 - - - - -] Starting to process devices in:{'current': set([u'6154d8ea-4c24-4219-afd7-01afef7a3c26', u'f126806b-9b7a-49f8-8fab-d321cb8bd2cd', u'1e0be70a-9d69-4d2f-a1d3-bd2f7ea4d28c']), 'removed': set([u'5e420f53-c356-4977-94fa-389db3f612c7']), 'added': set([u'5e420f53-c356-4977-94fa-389db3f612c7', u'6154d8ea-4c24-4219-afd7-01afef7a3c26', u'1e0be70a-9d69-4d2f-a1d3-bd2f7ea4d28c', u'f126806b-9b7a-49f8-8fab-d321cb8bd2cd']), 'updated': set([u'5e420f53-c356-4977-94fa-389db3f612c7'])}
...
 Agent rpc_loop - iteration:1 completed. Processed ports statistics: {'regular': {'updated': 1, 'added': 1, 'removed': 1}}.

So this is duplicate of https://bugs.launchpad.net/mos/+bug/1577721. Please feel free to verify after https://bugs.launchpad.net/mos/+bug/1577721 is fixed