Lock wait timeout traces for DVR routers

[Armando Migliaccio]

This has been observed here:

http://logs.openstack.org/80/113580/3/experimental/check-tempest-dsvm-neutron-dvr/a0e0c32/logs/screen-q-svc.txt.gz?level=TRACE#_2014-08-13_00_13_58_408

This is most likely caused by a long-running DB transaction, but this needs more triaging.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

This was also seen recently in our testing. I will upload the logs soon.

[Armando Migliaccio (armando-migliaccio)]

If the stacktrace is different from the one above, please do.

Thanks,
Armando

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/114691

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/115754

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/115829

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/115837

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114691
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2fc7fd6eb1b7e728b45e6a202cd9011dbb920950
Submitter: Jenkins
Branch: master

commit 2fc7fd6eb1b7e728b45e6a202cd9011dbb920950
Author: armando-migliaccio <email address hidden>
Date: Fri Aug 15 15:55:21 2014 -0700

    Minor refactoring of auto_schedule_routers

    The method is more complicated than it needs to be, and it makes it difficult
    to target fixes for it.

    This is done in preparation of fix for DB lock timeout errors observed while
    dealing with DVR routers.

    Test coverage is already provided, and more granular coverage is added to
    reflect the new structure being introduced.

    Partial-bug: #1356121

    Change-Id: Ifb7a742b64139f3a5d9b88c3c6261b1b890946f9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/115829
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=968ab9006ecf3fe3f22c19ac219e7376eb718d77
Submitter: Jenkins
Branch: master

commit 968ab9006ecf3fe3f22c19ac219e7376eb718d77
Author: armando-migliaccio <email address hidden>
Date: Wed Aug 20 17:36:06 2014 -0700

    Minor refactoring for add_router_to_l3_agent

    This method is more complicated than it needs
    to be, and it makes it difficult to target
    fixes for it.

    Furthermore, this method calls into
    auto_schedule_routers, which duplicates some
    of the DB calls already made in the above
    mentioned method. This refactoring
    is done in preparation of the performance
    improvement.

    Partial-bug: #1356121
    Related-Bug: #1358636

    Change-Id: I9a0cfa41a5f067949b964d39157def55c40bf9af

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/115837
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=76f4d7db9f44022667d347dfe0474a3abb3b1bb7
Submitter: Jenkins
Branch: master

commit 76f4d7db9f44022667d347dfe0474a3abb3b1bb7
Author: armando-migliaccio <email address hidden>
Date: Wed Aug 20 18:04:33 2014 -0700

    Do not use auto_schedule_routers to add router to agent

    auto_schedule_routers makes a number of DB calls that are
    unnecessary as they are already made during the validation
    phase of add_router_to_l3_agent. Once the validation is
    done, the only business left is to create the binding: this
    is what this patch does.

    Partial-bug: #1356121

    Change-Id: Ia9be998c2b94416bc46ef78415099f0099b08c2a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/115754
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1105d732b2cb6ec66d042c85968d47fe6d733f5f
Submitter: Jenkins
Branch: master

commit 1105d732b2cb6ec66d042c85968d47fe6d733f5f
Author: armando-migliaccio <email address hidden>
Date: Wed Aug 20 12:15:42 2014 -0700

    Avoid auto-scheduling for distributed routers

    The reason for this is twofold:

    - It may relieve contention on DB access while
      both servers and l3 agents are busy setting up
      and syncing routers down respectively.

    - It prevents accidental placement of namespaces
      during the L3 sync_routers process, as auto
      scheduling without taking into account the state
      of the L3 agents, as well as the state of the
      routers being processed, may overrule the placement
      decision made during router operations.

    Partial-bug: #1356121
    Partial-bug: #1359326

    Change-Id: Ia677ce212145d6cee65adeb1d8ae594e6ac5e34d

[Armando Migliaccio (armando-migliaccio)]

An all time query on logstash shows no occurrences on the dvr job. I'd close this one for now.

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiT3BlcmF0aW9uYWxFcnJvcjogKE9wZXJhdGlvbmFsRXJyb3IpICgxMjA1LCAnTG9jayB3YWl0IHRpbWVvdXQgZXhjZWVkZWQ7IHRyeSByZXN0YXJ0aW5nIHRyYW5zYWN0aW9uJylcIiBBTkQgbWVzc2FnZTpcImluIGJpbmRfcm91dGVyXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6ImFsbCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MTA1NjM1MDI2Nzh9