tripleo

Bug #1939023
Comment #4

Comment 4 for bug 1939023

Revision history for this message

Marios Andreou (marios-b) wrote on 2021-08-10:

Based on comment #1 and attached screen shot this started ~3rd August. I compared 2 'good runs' one that took close to 3 hours from 2/3 August [1] and another recent one from yesterday 9th august [2]

From [1]

3 hrs 9 mins 1 sec 2021-08-02 22:22:27

Ran: 1416 tests in 3475.6904 sec.
- Passed: 1295
- Skipped: 121
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 0
Sum of execute time for each test: 8074.3680 sec.

- Worker 0 (428 tests) => 0:57:48.876694
- Worker 1 (334 tests) => 0:50:33.366087
- Worker 2 (367 tests) => 0:39:59.810439
- Worker 3 (287 tests) => 0:49:15.021685

From [2]
Ran: 1416 tests in 7587.3411 sec.
- Passed: 1295
- Skipped: 121
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 0
Sum of execute time for each test: 16297.8125 sec.
- Worker 0 (362 tests) => 1:59:08.312166
- Worker 1 (356 tests) => 1:08:44.041797
- Worker 2 (426 tests) => 2:06:16.196664
- Worker 3 (272 tests) => 1:21:51.781254

As can be seen in 2 the same tests take twice as long to complete. You can see more about the timings at the stackviz logs [3] ('good' ~1 hour tempest run) and [4] (bad ~2 hours tempest)

[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby/d385cc6/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

[2] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby/fb267a1/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

[3] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby/d385cc6/logs/stackviz/#/

[4] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby/fb267a1/logs/stackviz/#/

Based on comment #1 and attached screen shot this started ~3rd August. I compared 2 'good runs' one that took close to 3 hours from 2/3 August [1] and another recent one from yesterday 9th august  [2]

From [1]

3 hrs 9 mins 1 sec	2021-08-02 22:22:27

Ran: 1416 tests in 3475.6904 sec.
 - Passed: 1295
 - Skipped: 121
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 8074.3680 sec.

- Worker 0 (428 tests) => 0:57:48.876694
 - Worker 1 (334 tests) => 0:50:33.366087
 - Worker 2 (367 tests) => 0:39:59.810439
 - Worker 3 (287 tests) => 0:49:15.021685

From [2]
Ran: 1416 tests in 7587.3411 sec.
 - Passed: 1295
 - Skipped: 121
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 16297.8125 sec.
 - Worker 0 (362 tests) => 1:59:08.312166
 - Worker 1 (356 tests) => 1:08:44.041797
 - Worker 2 (426 tests) => 2:06:16.196664
 - Worker 3 (272 tests) => 1:21:51.781254

As can be seen in 2 the same tests take twice as long to complete. You can see more about the timings at the stackviz logs [3] ('good' ~1 hour tempest run) and [4] (bad ~2 hours tempest)