Tempest test test_add_list_remove_router_on_l3_agent is Failed

[Sergii Turivnyi]

Detailed bug description:
Tempest test test_add_list_remove_router_on_l3_agent is Failed with Error:
   [ tempest.lib.exceptions.Conflict: An object with that identifier already exists]

Configuration:
    ISO: http://paste.openstack.org/show/495500/
    Compute - QEMU.
    Network - Neutron with VLAN segmentation, DVR.
    Storage Backends - Ceph RBD for volumes (Cinder), Ceph RBD for ephemeral volumes (Nova), Ceph RBD for images (Glance)
    RADOS
    Additional services - Install Ironic, Sahara

In tab Settings->Compute check Nova quotas

Nodes: controller, compute, ironic, ceph-osd

Steps to reproduce:
    1. Deploy ISO in configuration see (Detailed bug description)
    2. Navigate to controller node
    3. Install git (use apt-get install git)
    4. Clone script to deploy rally + tempest
       # git clone https://github.com/obutenko/mos-rally-verify.git
    5. Navigate to the https://github.com/obutenko/mos-rally-verify
    6. Execute necessary steps to deploy tempest
    7. Tun test in debug mode
        #rally --debug verify start --regex tempest.api.network.admin.test_l3_agent_scheduler.L3AgentSchedulerTestJSON

Expected results:
Test is passed

Actual result:
Test is Failed
(see comments)

Reproducibility:
See attachment

Workaround:
---

Impact:
---

Description of the environment:
See (Detailed bug description)
Snapshot: https://www.google.com.ua/?gfe_rd=cr&ei=KJwgV4z-DfGIyAWyxqkQ&gws_rd=ssl

Additional information:
Error Message

test failed

Stacktrace

Traceback (most recent call last):
  File "/home/rally/.rally/tempest/for-deployment-da64b6f0-10c4-461b-b408-6298750a57d8/tempest/api/network/admin/test_l3_agent_scheduler.py", line 115, in test_add_list_remove_router_on_l3_agent
    router_id=self.router['id'])
  File "/home/rally/.rally/tempest/for-deployment-da64b6f0-10c4-461b-b408-6298750a57d8/tempest/lib/services/network/agents_client.py", line 47, in create_router_on_l3_agent
    return self.create_resource(uri, kwargs)
  File "/home/rally/.rally/tempest/for-deployment-da64b6f0-10c4-461b-b408-6298750a57d8/tempest/lib/services/network/base.py", line 60, in create_resource
    resp, body = self.post(req_uri, req_post_data)
  File "/home/rally/.rally/tempest/for-deployment-da64b6f0-10c4-461b-b408-6298750a57d8/tempest/lib/common/rest_client.py", line 259, in post
    return self.request('POST', url, extra_headers, headers, body)
  File "/home/rally/.rally/tempest/for-deployment-da64b6f0-10c4-461b-b408-6298750a57d8/tempest/lib/common/rest_client.py", line 641, in request
    resp, resp_body)
  File "/home/rally/.rally/tempest/for-deployment-da64b6f0-10c4-461b-b408-6298750a57d8/tempest/lib/common/rest_client.py", line 709, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: An object with that identifier already exists
Details: {u'message': u'The router 745af174-dddb-4be9-af73-ee9c832edcc6 has been already hosted by the L3 Agent f1160694-eff0-461b-949e-477acc83ca55.', u'type': u'RouterHostedByL3Agent', u'detail': u''}

[Sergii Turivnyi (sturivnyi)]

Snapshot: https://drive.google.com/open?id=0B8hkiEm94sEtYWljVVZjY1owZG8

[Ann Taraday (akamyshnikova)]

Tests failed with error "The router 745af174-dddb-4be9-af73-ee9c832edcc6 has been already hosted by the L3 Agent f1160694-eff0-461b-949e-477acc83ca55." This means that test tried to create router with same id that router was already created. In neutron server and l3-agent logs there are no error messages, only DEBUG neutron.scheduler.l3_agent_scheduler [req-ca225f68-bc78-49c4-8585-2aad442a1a5f - - - - -] Router a6e91384-5d05-4df4-8350-b9cdd5ba353f has already been hosted by L3 agent 92fba2dc-92f1-417d-a8f4-8de3bde86629 _filter_unscheduled_routers /usr/lib/python2.7/dist-packages/neutron/scheduler/l3_agent_scheduler.py

I was able to reproduce the failed test on the env with DVR and VXLAN iso 189. Error - http://paste.openstack.org/show/495678/ neutron router list during test execution - http://paste.openstack.org/show/495677/, so this router was created and deleted successfully during the test. Consider this bug as a problem with tempest test.

[Timur Nurlygayanov (tnurlygayanov)]

Hi Yaroslav, it looks like Tempest issue. Could you please describe separate bug in Tempest and fix it or ask other engineers from MOSQA team fix it?

Thank you!

[Yaroslav Lobankov (ylobankov)]

Patch is on review https://review.openstack.org/#/c/322118/

[Oleksiy Butenko (obutenko)]

It's tempest bug(non-release), so change to medium for 9.0 and Won't Fix.

[Yaroslav Lobankov (ylobankov)]

It is not a bug in the product. It's a Tempest issue https://bugs.launchpad.net/tempest/+bug/1590049

[Timur Nurlygayanov (tnurlygayanov)]

Ok let's track the status in Tempest upstream project then.