test_add_list_remove_router_on_l3_agent fails with Conflict

[Sergey Kraynev]

Test tempest.api.network.admin.test_l3_agent_scheduler.L3AgentSchedulerTestXML.test_add_list_remove_router_on_l3_agent fails for job gate-tempest-dsvm-neutron-full with traceback:

ft335.1: tempest.api.network.admin.test_l3_agent_scheduler.L3AgentSchedulerTestXML.test_add_list_remove_router_on_l3_agent[gate,smoke]_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
2014-08-18 15:31:39,865 535 INFO [tempest.common.rest_client] Request (L3AgentSchedulerTestXML:test_add_list_remove_router_on_l3_agent): 200 POST http://127.0.0.1:5000/v2.0/tokens
2014-08-18 15:31:40,148 535 INFO [tempest.common.rest_client] Request (L3AgentSchedulerTestXML:test_add_list_remove_router_on_l3_agent): 201 POST http://127.0.0.1:9696/v2.0/routers 0.282s
2014-08-18 15:31:40,259 535 INFO [tempest.common.rest_client] Request (L3AgentSchedulerTestXML:test_add_list_remove_router_on_l3_agent): 409 POST http://127.0.0.1:9696/v2.0/agents/47dd83c6-f92d-40d9-8601-5a38b6b9eda0/l3-routers 0.109s
2014-08-18 15:31:40,714 535 INFO [tempest.common.rest_client] Request (L3AgentSchedulerTestXML:_run_cleanups): 204 DELETE http://127.0.0.1:9696/v2.0/routers/fd7d082c-71db-4fa0-bd0c-0b31acef9b1a 0.450s
}}}

Traceback (most recent call last):
  File "tempest/api/network/admin/test_l3_agent_scheduler.py", line 66, in test_add_list_remove_router_on_l3_agent
    self.agent['id'], router['router']['id'])
  File "tempest/services/network/xml/network_client.py", line 218, in add_router_to_l3_agent
    resp, body = self.post(uri, str(common.Document(router)))
  File "tempest/services/network/network_client_base.py", line 73, in post
    return self.rest_client.post(uri, body, headers)
  File "tempest/common/rest_client.py", line 219, in post
    return self.request('POST', url, extra_headers, headers, body)
  File "tempest/common/rest_client.py", line 431, in request
    resp, resp_body)
  File "tempest/common/rest_client.py", line 485, in _error_checker
    raise exceptions.Conflict(resp_body)
Conflict: An object with that identifier already exists
Details: {'message': 'The router fd7d082c-71db-4fa0-bd0c-0b31acef9b1a has been already hosted by the L3 Agent 47dd83c6-f92d-40d9-8601-5a38b6b9eda0.', 'type': 'RouterHostedByL3Agent', 'detail': {}}

There is log of test results: http://logs.openstack.org/43/97543/10/gate/gate-tempest-dsvm-neutron-full/8f09c74/logs/testr_results.html.gz

[Armando Migliaccio (armando-migliaccio)]

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiaW4gdGVzdF9hZGRfbGlzdF9yZW1vdmVfcm91dGVyX29uX2wzX2FnZW50XCIgQU5EIGZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIgQU5EIGJ1aWxkX25hbWU6XCJjaGVjay10ZW1wZXN0LWRzdm0tbmV1dHJvbi1mdWxsXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDg0NjM4ODY1MjUsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0=

[Armando Migliaccio (armando-migliaccio)]

This is interesting:

http://logs.openstack.org/43/97543/10/gate/gate-tempest-dsvm-neutron-full/8f09c74/logs/screen-q-svc.txt.gz#_2014-08-18_15_31_40_202

It looks like the agent's sync_routers comes before the add_to_agent operation. Since sync_routers triggers an automatic scheduling [1], this is why we get the conflict during the add operation.

[1] - https://github.com/openstack/neutron/blob/master/neutron/db/l3_rpc_base.py#L66

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/115383

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/115829

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/115829
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=968ab9006ecf3fe3f22c19ac219e7376eb718d77
Submitter: Jenkins
Branch: master

commit 968ab9006ecf3fe3f22c19ac219e7376eb718d77
Author: armando-migliaccio <email address hidden>
Date: Wed Aug 20 17:36:06 2014 -0700

    Minor refactoring for add_router_to_l3_agent

    This method is more complicated than it needs
    to be, and it makes it difficult to target
    fixes for it.

    Furthermore, this method calls into
    auto_schedule_routers, which duplicates some
    of the DB calls already made in the above
    mentioned method. This refactoring
    is done in preparation of the performance
    improvement.

    Partial-bug: #1356121
    Related-Bug: #1358636

    Change-Id: I9a0cfa41a5f067949b964d39157def55c40bf9af

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/115383
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=89e76a8afae19c5f66d532538607aedaac59722a
Submitter: Jenkins
Branch: master

commit 89e76a8afae19c5f66d532538607aedaac59722a
Author: Eugene Nikanorov <email address hidden>
Date: Tue Aug 19 23:01:11 2014 +0400

    Fix l3 agent scheduling logic to avoid unwanted failures

    In case router is being added to l3 agent which is already hosting
    the router it is fine to let such a request to succeed.
    This patch also adds a check for unnecessary scheduling that might happen
    twice in described case and lead to unwanted messages in the logs.

    Change-Id: Id104b36ba7e1e6f6a9378ee600c33e9962230521
    Closes-Bug: #1358636