OSTF failed in test 'check l3-agent rescheduling after l3-agent dies'

[Dmitry Tyzhnenko]

Found on CI - http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.ha_neutron_destructive/41/testReport/(root)/neutron_l3_migration/

        Scenario:
            1. Create cluster. HA, Neutron with GRE segmentation
            2. Add 3 nodes with controller roles
            3. Add 2 nodes with compute roles
            4. Add 1 node with cinder role
            5. Create an instance with a key pair
            6. Manually reschedule router from primary controller
               to another one
            7. Stop l3-agent on new node with pcs
            8. Check l3-agent was rescheduled
            9. Check network connectivity from instance via
               dhcp namespace
            10. Run OSTF

Expected result:
   All steps passed w/o errors

Actual result:
  OSTF test failed

AssertionError: Failed 1 OSTF tests; should fail 0 tests. Names of failed tests: [{u'Check network connectivity from instance via floating IP (failure)': u'Subnet can not be created Please refer to OpenStack logs for more details.'}]

OSTF errors - http://paste.openstack.org/show/406268/

Neutron logs of router, network and subnet creation - http://paste.openstack.org/show/406282/

[Oleg Bondarev (obondarev)]

according to logs:
 - network was created at 07:46:33.720669+00:00, request was handled on node-5
 - request for subnet creation was at 07:46:33.727394+00:00 , request was handled on node-2

We think 7 milliseconds might not be enough for db sync between nodes.

[Kevin Benton (kevinbenton)]

does Galera return from a commit before data is synced between all active members?

[Alexander Ignatov (aignatov)]

Based on analysis of logs attached to bug we found that test failed because subnet was not created by unknown reasons;.

Tried to reproduce on virtual env approximately the same like was used in Jenkins system tests.
Unfortunately issue was not reproduced with the assumptions did in previous two comments and with the following scenario:

1. Create network
2. Create subnet in network
3. Delete network.
4. Repeat items 1-3 in cycle, 100 iterations are fine.

You can take script here: http://paste.openstack.org/show/412213/

Tested on build 139. Build info: http://paste.openstack.org/show/412371/

https://bugs.launchpad.net/mos/+bug/1483260

[Dmitry Tyzhnenko (dtyzhnenko)]

Move the bug to invalid due not produced for last month