[L3 HA] Lost connection to external network after destroying controller

[Kristina Berezovskaia]

After destroying controller on env with L3 HA connectivity to external net has been lost: active became one agent and nat rules were created on anjther (which was standby)

Steps:
1) Create net1, subnet
2) Create net2, subnet
3) Create router, set gatawey and interfaces to both nets
4) boot vm1 in net1 ans associate floating
5) boot vm2 in net2
7) start ping vm1 from vm2 by floating
8) destroy primary controller (active l3 agent is on primary controller)
9) wait some time
10) Check ping
Expected result: ping is available
Current result: ping isn't available

Find on:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "264"
  build_id: "264"
  fuel-nailgun_sha: "0e09dce510927f2cc490b898e5fe3f813bd791be"
  python-fuelclient_sha: "f033192b84263f0e699458a4274289a5198ae7e4"
  fuel-agent_sha: "660c6514caa8f5fcd482f1cc4008a6028243e009"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "48fd58676debcc85951db68df6d77c22daa55e52"
  fuel-library_sha: "ab7e51f345ffb7c256e0f61addcf86553d7c3867"
  fuel-ostf_sha: "23b7ae2a1a57de5a3e1861ffb7805394ca339cc2"
  fuel-mirror_sha: "6534117233a5bdc51d7d47361bc7d511e4b11e6f"
  fuelmenu_sha: "fcb15df4fd1a790b17dd78cf675c11c279040941"
  shotgun_sha: "a0bd06508067935f2ae9be2523ed0d1717b995ce"
  network-checker_sha: "a3534f8885246afb15609c54f91d3b23d599a5b1"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "26adf12c320936a97a9b0a84169a6e58c530e848"
(3 controllers, 2 compute, neutron+vxlan+l3 ha)

Destroy 3 controller, active now is 4. Logs in attachment

[Kristina Berezovskaia (kkuznetsova)]

Also was reproduced on vxlan+l2 pop.
Internal traffic between both vms are also not available now
Attach logs

[Kristina Berezovskaia (kkuznetsova)]

The same situation is for destroying non primary controller for vxlan+l2pop (destroy non-primary work for vlan and vxlan). Attach logs

[Kristina Berezovskaia (kkuznetsova)]

The same situation is for reset primary controller for vxlan (reset primary work for vlan and vxlan+l2). Attach logs

[Elena Ezhova (eezhova)]

Unable to reproduce on

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "303"
  build_id: "303"
  fuel-nailgun_sha: "352548386007399a95a6f21b1fcd9c48a0726325"
  python-fuelclient_sha: "b2bbcdf1c0f38adb34cff01cb6040006911f2ea5"
  fuel-agent_sha: "49bb78675b749d15ae8f0f045dc2b0811777a9d6"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "c56dfde2da034151a7e707b381c4cf9d213b4ba2"
  fuel-library_sha: "14576a3dbb3be5e4013c306776dc2eaefe0c15e0"
  fuel-ostf_sha: "9910a4726cbd038c257582b429527e40c4c3cb20"
  fuel-mirror_sha: "dbbe9ddc2c8a336aa7ab62952761bd079e374d1d"
  fuelmenu_sha: "680b720291ff577f4c058cee25f85e563c96312e"
  shotgun_sha: "cacb93cbc28910ff0dc38f30a855efa9af50d8ce"
  network-checker_sha: "d443ef47abeda58d319bc8d33d5005dd09440a02"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "74e9affd54e5a31fd55ed75a3402940dd186a621"

[Elena Ezhova (eezhova)]

Managed to reproduce on the same env.

[Kristina Berezovskaia (kkuznetsova)]

After reproducing on #307 iso (vxlan+l2pop+neutron, 2 compute and 3 controller nodes)
We can see the same situation as in bug https://bugs.launchpad.net/mos/+bug/1525835

root@node-7:~# neutron port-show 7d0d0174-7fd5-4b5f-8574-8fed24031c7a
+-----------------------+--------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | node-6.domain.tld |
| binding:profile | {} |
| binding:vif_details | {} |
| binding:vif_type | binding_failed |
| binding:vnic_type | normal |
| device_id | 0d644243-17bd-400c-8826-71af2c24991d |
| device_owner | network:router_interface |
| dns_assignment | {"hostname": "host-10-1-2-1", "ip_address": "10.1.2.1", "fqdn": "host-10-1-2-1.openstacklocal."} |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "152ec49a-b25b-4c06-a63d-961d4a56bda7", "ip_address": "10.1.2.1"} |
| id | 7d0d0174-7fd5-4b5f-8574-8fed24031c7a |
| mac_address | fa:16:3e:7c:0e:b0 |
| name | |
| network_id | 8c18aee3-ef89-4059-822f-fc6b49d961a8 |
| port_security_enabled | False |
| security_groups | |
| status | ACTIVE |
| tenant_id | 95fcc8847cb645f7af8d94af392ddbcf ...