networking-ovn-dsvm-functional-py27 job killed on timeout in stable/train branch

[Ihar Hrachyshka]

Examples:

https://zuul.opendev.org/t/openstack/build/a1d2c87b23e641539cff111abc8c5e4a/log/job-output.txt
https://5b948c35ca2c61dd95cb-b83277dfb85347aca874505fa2930911.ssl.cf2.rackcdn.com/862358/2/check/networking-ovn-dsvm-functional-py27/073519e/job-output.txt

The job takes more than 2h and is eventually killed.

Log example:

2022-11-20 01:04:17.245895 | primary | 2022-11-20 01:04:17.245 | {7} networking_ovn.tests.functional.test_ovn_db_sync.TestOvnNbSync.test_ovn_nb_sync_off [67.421953s] ... ok
2022-11-20 01:04:17.247309 | primary | 2022-11-20 01:04:17.246 | {0} networking_ovn.tests.functional.test_router.TestRouter.test_gateway_chassis_with_bridge_mappings [12.473030s] ... ok
2022-11-20 01:04:17.248841 | primary | 2022-11-20 01:04:17.248 | {2} networking_ovn.tests.functional.test_router.TestRouter.test_router_port_ipv6_ra_configs_ipv4 [6.018013s] ... ok
2022-11-20 01:04:17.250444 | primary | 2022-11-20 01:04:17.250 | {0} networking_ovn.tests.functional.test_router.TestRouter.test_router_port_ipv6_ra_configs_addr_mode_slaac [5.654530s] ... ok
2022-11-20 01:04:17.252024 | primary | 2022-11-20 01:04:17.251 | {2} networking_ovn.tests.functional.test_trunk_driver.TestOVNTrunkDriver.test_subport_add [4.991551s] ... ok
2022-11-20 01:04:17.255232 | primary | 2022-11-20 01:04:17.254 | {0} networking_ovn.tests.functional.test_trunk_driver.TestOVNTrunkDriver.test_trunk_create_with_subports [4.687337s] ... ok
2022-11-20 01:04:17.256586 | primary | 2022-11-20 01:04:17.256 | {6} networking_ovn.tests.functional.test_ovn_db_sync.TestOvnNbSync.test_ovn_nb_sync_repair [73.895267s] ... ok
2022-11-20 01:04:17.257905 | primary | 2022-11-20 01:04:17.257 | {4} networking_ovn.tests.functional.test_ovn_db_sync.TestOvnNbSyncOverSsl.test_ovn_nb_sync_off [64.915188s] ... ok
2022-11-20 01:04:17.263929 | primary | 2022-11-20 01:04:17.259 | {6} networking_ovn.tests.functional.test_ovsdb_monitor.TestNBDbMonitor.test_distributed_lock [3.913342s] ... ok
2022-11-20 01:04:17.265641 | primary | 2022-11-20 01:04:17.265 | {6} networking_ovn.tests.functional.test_ovsdb_monitor.TestNBDbMonitor.test_floatingip_mac_bindings [11.765796s] ... ok
2022-11-20 01:04:17.267119 | primary | 2022-11-20 01:04:17.266 | {6} networking_ovn.tests.functional.test_ovsdb_monitor.TestNBDbMonitorOverSsl.test_port_up_down_events [6.316748s] ... ok
2022-11-20 01:04:17.271742 | primary | 2022-11-20 01:04:17.271 | {6} networking_ovn.tests.functional.test_qos_driver.TestOVNQosDriver.test_port_qos_options_with_rule [3.441892s] ... ok
2022-11-20 02:51:16.344359 | primary | /home/zuul/workspace/devstack-gate/functions.sh: line 1009: 26167 Killed timeout -s 9 ${REMAINING_TIME}m bash -c "source $WORKSPACE/devstack-gate/functions.sh && $cmd"

Observation:

both failed runs were executing the following test cases just before the test runner froze:

2022-11-20 01:04:17.256586 | primary | 2022-11-20 01:04:17.256 | {6} networking_ovn.tests.functional.test_ovn_db_sync.TestOvnNbSync.test_ovn_nb_sync_repair [73.895267s] ... ok
2022-11-20 01:04:17.257905 | primary | 2022-11-20 01:04:17.257 | {4} networking_ovn.tests.functional.test_ovn_db_sync.TestOvnNbSyncOverSsl.test_ovn_nb_sync_off [64.915188s] ... ok

(I don't know if this is related though)

[Lajos Katona (lajos-katona)]

hmm, there are no logs saved for these jobs (I am not sure if it is due to the timeout, or it is normal).
Is this issue happen only with zuul?

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Patch: https://review.opendev.org/c/openstack/networking-ovn/+/866022

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/networking-ovn/+/866022
Committed: https://opendev.org/openstack/networking-ovn/commit/65f1e0aa9623500393c671899a5d78a108472e59
Submitter: "Zuul (22348)"
Branch: stable/train

commit 65f1e0aa9623500393c671899a5d78a108472e59
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Nov 29 16:07:43 2022 +0100

    Remove "networking-ovn-dsvm-functional-py27" job

    This job is currently failing. Because this branch is in EM and we
    don't have resources to fix this job now, we are removing it from
    the gate and check queues.

    This patch is closing this related bug. If in a future a new patch
    is sent to fix the job, the bug will be reopened.

    Closes-Bug: #1997262
    Change-Id: Ifc4fdaf22f9768e4bffd5c3b4f2917a092859c4b

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-ovn train-eol release.