[MOS 8.0][Neutron][L3 HA] After resetting primary controller with active l3-agent the active l3-agent is not changed

[Alexander Gromov]

Environment:
MOS 8.0
3 controllers + 2 computes + VLAN L3 HA or VxLAN L2 POP L3 HA

Steps to reproduce:
1. Create network1, network2
2. Create router1 and connect it with network1, network2 and external net
3. Boot vm1 in network1
4. Boot vm2 in network2 and associate floating ip
5. Add rules for ping
6. Find node with active ha_state for router
7. If node from step 6 isn't primary controller, reschedule router1 to primary by banning all another and then clear them
8. Start ping vm2 from vm1 by floating ip
9. Reset primary controller
10. Stop ping

Agents before test:
+--------------------------------------+--------------------------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| b66c0707-1ab6-4e75-ad94-eab6f1134769 | node-3.test.domain.local | True | :-) | active |
| 759a3b60-645b-46df-bc05-43cefe8f4760 | node-5.test.domain.local | True | :-) | standby |
| 30e36618-dcae-44d9-89e8-1ce78c23be3f | node-4.test.domain.local | True | :-) | standby |
+--------------------------------------+--------------------------+----------------+-------+----------+

Expected results:
ping lost NO MORE than 10 packets
ANOTHER agent has ACTIVE ha_state, others (2) has STAND BY ha_state
+--------------------------------------+--------------------------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| b66c0707-1ab6-4e75-ad94-eab6f1134769 | node-3.test.domain.local | True | :-) | standby |
| 759a3b60-645b-46df-bc05-43cefe8f4760 | node-5.test.domain.local | True | :-) | standby |
| 30e36618-dcae-44d9-89e8-1ce78c23be3f | node-4.test.domain.local | True | :-) | active |
+--------------------------------------+--------------------------+----------------+-------+----------+

Actual result:
ping lost MORE than 10 packets
THE SAME l3-agent becomes active after some time, others (2) has STANDBY ha_state
+--------------------------------------+--------------------------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+--------------------------------------+--------------------------+----------------+-------+----------+
| b66c0707-1ab6-4e75-ad94-eab6f1134769 | node-3.test.domain.local | True | :-) | active |
| 759a3b60-645b-46df-bc05-43cefe8f4760 | node-5.test.domain.local | True | :-) | standby |
| 30e36618-dcae-44d9-89e8-1ce78c23be3f | node-4.test.domain.local | True | :-) | standby |
+--------------------------------------+--------------------------+----------------+-------+----------+

[Dmitry Klenov (dklenov)]

Neutron folks, please verify, if this one os a dup of https://bugs.launchpad.net/fuel/+bug/1572165.

[Vadim Rovachev (vrovachev)]

Need to increase timeout in tests.

[Alexander Gromov (agromov)]

Reproduced on CI again for the following configuration: 8GB RAM, 2 CPU (for all nodes), 3 controllers 2 computes-cinder, VxLAN L2POP L3HA

[Alexander Gromov (agromov)]

After resetting the same l3 agent was active.

[Ann Taraday (akamyshnikova)]

I looked into this issue. Although "neutron l3-agent-list-hosting-router router-id" shows standby for all agents, ping is available and if we check "cat /var/lib/neutron/ha_confs/<router-id>/state" we can see that another agent has become master. Also in neutron-keepalived-state-change.log there is log message that this agent became master http://paste.openstack.org/show/497923/.

In logs of l3-agents (and all other neutron logs) http://paste.openstack.org/show/497925/, so seems that was lost connection to rabbit, so state change of the agent was lost for l3-agent-list-hosting-router.

Workaround for tests check "cat /var/lib/neutron/ha_confs/<router-id>/state" for state change.

[Kristina Berezovskaia (kkuznetsova)]

This test worked correctly now
8-0: https://mirantis.testrail.com/index.php?/runs/view/21130&group_by=cases:section_id&group_order=asc
9-2: https://mirantis.testrail.com/index.php?/runs/view/30315&group_by=cases:section_id&group_order=asc