Functional test test_router_processing_pool_size failing

[Slawek Kaplonski]

Functional test neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_router_processing_pool_size is failing.

Test was introduced recently with patch https://review.openstack.org/#/c/633869/

Example of failure http://logs.openstack.org/42/619742/52/check/neutron-functional-python27/21f0e48/testr_results.html.gz

[Slawek Kaplonski (slaweq)]

Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%20132%2C%20in%20test_router_processing_pool_size%5C%22

[LIU Yulong (dragon889)]

Meets the following timeouts:
2019-02-16 09:32:00.948 5471 ERROR neutron.agent.linux.utils [req-de310718-cd58-4615-a81e-0c00d9b46336 - - - - -] Rootwrap error running command: ['ip', 'netns', 'exec', 'qrouter-4bce7d00-6ada-45f7-b974-75dd8b6a0815', 'conntrack', '-D', '-q', '19.4.4.4']: fixtures._fixtures.timeout.TimeoutException

2019-02-17 15:20:15.509 11450 ERROR neutron.agent.linux.utils [req-5f8eabb0-0f21-4bfa-bd47-01abc12962e6 - - - - -] Rootwrap error running command: ['kill', '-9', '9813']: fixtures._fixtures.timeout.TimeoutException

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/637544

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637544
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a10281bf236aa47ee32f247e0aa79f5fdd42348b
Submitter: Zuul
Branch: master

commit a10281bf236aa47ee32f247e0aa79f5fdd42348b
Author: LIU Yulong <email address hidden>
Date: Mon Feb 18 21:33:32 2019 +0800

    Call _safe_router_removed during pool resize testing

    Closes-Bug: #1816239
    Change-Id: Ie93d17ff8b5825e401e342d215db4bcfd7b1cd3e

[Slawek Kaplonski (slaweq)]

I think that this issue isn't fixed yet.

I saw couple of times neutron-functional tests failing with timeout. What I think is that this test test_router_processing_pool_size is "hanging" somehow because in each of those cases, there is no result of this test in job-output.txt file but there are logs in dsvm-functional-tests logs directory. For example:

http://logs.openstack.org/96/636296/5/gate/neutron-functional/eb1b8e3/logs/dsvm-functional-logs/neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_router_processing_pool_size.txt.gz

http://logs.openstack.org/92/636892/1/check/neutron-functional/6508cd2/logs/dsvm-functional-logs/neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_router_processing_pool_size.txt.gz

http://logs.openstack.org/61/626561/14/check/neutron-functional-python27/2649a9a/logs/dsvm-functional-logs/neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_router_processing_pool_size.txt.gz

[Slawek Kaplonski (slaweq)]

Another example in python 2.7 job: http://logs.openstack.org/35/638635/1/check/neutron-functional-python27/a1b1704/logs/testr_results.html.gz

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/639034

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/639034
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=70fb9c64792b946fb83333561b05b603a5b17ec9
Submitter: Zuul
Branch: master

commit 70fb9c64792b946fb83333561b05b603a5b17ec9
Author: LIU Yulong <email address hidden>
Date: Fri Feb 22 05:23:31 2019 +0800

    Mock router info functions for a functional case

    In functional test environment, seems L3 agent can not handle
    such 30+ routers in the test test_router_processing_pool_size.
    It still meets timeout for some processing procedure.
    For now, router initialize/process/delete is not our test purpose
    for this case, so we just mock them.

    Closes-Bug: #1816239
    Change-Id: I85dc6fd9d98a6a13bbf35ee2e67ce6f69be48dde

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0b3 development milestone.