| 2020-05-29 15:19:47 |
Sampat Ponnaganti |
bug |
|
|
added bug |
| 2020-05-29 15:20:09 |
Sampat Ponnaganti |
tempest: assignee |
|
Sam Kumar (sp810x) |
|
| 2020-05-29 15:23:36 |
Sampat Ponnaganti |
description |
Neutron tempest Pagination tet cases fail if run in parallel.
The pagination test cases see items created by other test cases and hence expected vs actual differ and the test cases fail.
For Eg:
These two test cases fail:
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68]
Traceback
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68] Traceback: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 353, in test_list_pagination_page_reverse_with_href_links self._test_list_pagination_page_reverse_with_href_links() File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1346, in _test_list_pagination_page_reverse_with_href_links self.assertSameOrder(expected_resources, reversed(resources)) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1160, in assertSameOrder self.assertEqual(len(original), len(actual)) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 5 != 6
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links[id-dcd02a7a-f07e-4d5e-b0ca-b58e48927a9b]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 337, in test_list_pagination_with_href_links
self._test_list_pagination_with_href_links()
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1312, in _test_list_pagination_with_href_links
self._test_list_pagination_iteratively(self._list_all_with_hrefs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1241, in _test_list_pagination_iteratively
len(expected_resources), sort_args
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1302, in _list_all_with_hrefs
self.assertEqual(1, len(resources_))
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual
self.assertThat(observed, matcher, message)
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: 1 != 0
Reason for failure:
More neutron trunk ports are being returned than expected.
Code has to be fixed for 2 and 3 below such that trunks are returned only for one project (specify project_id in GET trunks call). A parallel test might be creating trunks so this test case is failing.
2. test_list_pagination_page_reverse_with_href_links
-> Expected returns trunks for project: 864acee2d6c64faa8750cfe53437a158
-> Actual paginated returns trunk for more projects: 89e63227c3b6405498f8fb1973cd055d and 864acee2d6c64faa8750cfe53437a158
3. test_list_pagination_with_href_links
Same issue as in 2.
Able to run the tests if run in serial mode: concurrency = 1
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [3.404399s] ... ok{0}
neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_href_links [9.412321s] ... ok
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_marker [4.566547s] ... ok{0}
neutron_tempest_plugin.api.test_ports.PortsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [4.636547s] ... ok
{0} neutron_tempest_plugin.api.test_subnets.SubnetsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [1.971421s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [0.881594s] ... ok
{0} neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links [2.173380s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_asc [0.180619s] ... ok
{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_desc [0.176105s] ... ok
======
Totals
======
Ran: 9 tests in 107.7512 sec.
Passed: 9
Skipped: 0
Expected Fail: 0
Unexpected Success: 0
Failed: 0
Sum of execute time for each test: 27.4029 sec.
==============
Worker Balance
==============
Worker 0 (9 tests) => 0:01:21.001099
Proposed solution:
Update pagination test cases to query neutron resource only for a specific project.
Open to further discussion or any other solution. |
Neutron tempest Pagination tet cases fail if run in parallel.
The pagination test cases see items created by other test cases and hence expected vs actual differ and the test cases fail.
For Eg:
###### These two test cases fail:
Test case 1:
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68]
```code
Traceback
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68] Traceback: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 353, in test_list_pagination_page_reverse_with_href_links self._test_list_pagination_page_reverse_with_href_links() File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1346, in _test_list_pagination_page_reverse_with_href_links self.assertSameOrder(expected_resources, reversed(resources)) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1160, in assertSameOrder self.assertEqual(len(original), len(actual)) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 5 != 6
```
Test case 2: neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links[id-dcd02a7a-f07e-4d5e-b0ca-b58e48927a9b]
```code
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 337, in test_list_pagination_with_href_links
self._test_list_pagination_with_href_links()
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1312, in _test_list_pagination_with_href_links
self._test_list_pagination_iteratively(self._list_all_with_hrefs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1241, in _test_list_pagination_iteratively
len(expected_resources), sort_args
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1302, in _list_all_with_hrefs
self.assertEqual(1, len(resources_))
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual
self.assertThat(observed, matcher, message)
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: 1 != 0
```
###### Reason for failure:
More neutron trunk ports are being returned than expected.
Code has to be fixed for 2 and 3 below such that trunks are returned only for one project (specify project_id in GET trunks call). A parallel test might be creating trunks so this test case is failing.
2. test_list_pagination_page_reverse_with_href_links
-> Expected returns trunks for project: 864acee2d6c64faa8750cfe53437a158
-> Actual paginated returns trunk for more projects: 89e63227c3b6405498f8fb1973cd055d and 864acee2d6c64faa8750cfe53437a158
3. test_list_pagination_with_href_links
Same issue as in 2.
###### Able to run the tests if run in serial mode: concurrency = 1
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [3.404399s] ... ok{0}
neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_href_links [9.412321s] ... ok
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_marker [4.566547s] ... ok{0}
neutron_tempest_plugin.api.test_ports.PortsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [4.636547s] ... ok
{0} neutron_tempest_plugin.api.test_subnets.SubnetsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [1.971421s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [0.881594s] ... ok
{0} neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links [2.173380s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_asc [0.180619s] ... ok
{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_desc [0.176105s] ... ok
======
Totals
======
Ran: 9 tests in 107.7512 sec.
Passed: 9
Skipped: 0
Expected Fail: 0
Unexpected Success: 0
Failed: 0
Sum of execute time for each test: 27.4029 sec.
==============
Worker Balance
==============
Worker 0 (9 tests) => 0:01:21.001099
##### Proposed solution:
Update pagination test cases to query neutron resource only for a specific project.
Open to further discussion or any other solution. |
|
| 2020-05-29 15:24:31 |
Sampat Ponnaganti |
description |
Neutron tempest Pagination tet cases fail if run in parallel.
The pagination test cases see items created by other test cases and hence expected vs actual differ and the test cases fail.
For Eg:
###### These two test cases fail:
Test case 1:
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68]
```code
Traceback
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68] Traceback: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 353, in test_list_pagination_page_reverse_with_href_links self._test_list_pagination_page_reverse_with_href_links() File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1346, in _test_list_pagination_page_reverse_with_href_links self.assertSameOrder(expected_resources, reversed(resources)) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1160, in assertSameOrder self.assertEqual(len(original), len(actual)) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 5 != 6
```
Test case 2: neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links[id-dcd02a7a-f07e-4d5e-b0ca-b58e48927a9b]
```code
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 337, in test_list_pagination_with_href_links
self._test_list_pagination_with_href_links()
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1312, in _test_list_pagination_with_href_links
self._test_list_pagination_iteratively(self._list_all_with_hrefs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1241, in _test_list_pagination_iteratively
len(expected_resources), sort_args
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1302, in _list_all_with_hrefs
self.assertEqual(1, len(resources_))
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual
self.assertThat(observed, matcher, message)
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: 1 != 0
```
###### Reason for failure:
More neutron trunk ports are being returned than expected.
Code has to be fixed for 2 and 3 below such that trunks are returned only for one project (specify project_id in GET trunks call). A parallel test might be creating trunks so this test case is failing.
2. test_list_pagination_page_reverse_with_href_links
-> Expected returns trunks for project: 864acee2d6c64faa8750cfe53437a158
-> Actual paginated returns trunk for more projects: 89e63227c3b6405498f8fb1973cd055d and 864acee2d6c64faa8750cfe53437a158
3. test_list_pagination_with_href_links
Same issue as in 2.
###### Able to run the tests if run in serial mode: concurrency = 1
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [3.404399s] ... ok{0}
neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_href_links [9.412321s] ... ok
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_marker [4.566547s] ... ok{0}
neutron_tempest_plugin.api.test_ports.PortsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [4.636547s] ... ok
{0} neutron_tempest_plugin.api.test_subnets.SubnetsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [1.971421s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [0.881594s] ... ok
{0} neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links [2.173380s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_asc [0.180619s] ... ok
{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_desc [0.176105s] ... ok
======
Totals
======
Ran: 9 tests in 107.7512 sec.
Passed: 9
Skipped: 0
Expected Fail: 0
Unexpected Success: 0
Failed: 0
Sum of execute time for each test: 27.4029 sec.
==============
Worker Balance
==============
Worker 0 (9 tests) => 0:01:21.001099
##### Proposed solution:
Update pagination test cases to query neutron resource only for a specific project.
Open to further discussion or any other solution. |
Neutron tempest Pagination tet cases fail if run in parallel.
##### Issue
The pagination test cases see items created by other test cases and hence expected vs actual differ and the test cases fail.
##### Proposed solution:
Update pagination test cases to query neutron resource only for a specific project.
Open to further discussion or any other solution.
For Eg:
###### These two test cases fail:
Test case 1:
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68]
```code
Traceback
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68] Traceback: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 353, in test_list_pagination_page_reverse_with_href_links self._test_list_pagination_page_reverse_with_href_links() File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1346, in _test_list_pagination_page_reverse_with_href_links self.assertSameOrder(expected_resources, reversed(resources)) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1160, in assertSameOrder self.assertEqual(len(original), len(actual)) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 5 != 6
```
Test case 2: neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links[id-dcd02a7a-f07e-4d5e-b0ca-b58e48927a9b]
```code
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 337, in test_list_pagination_with_href_links
self._test_list_pagination_with_href_links()
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1312, in _test_list_pagination_with_href_links
self._test_list_pagination_iteratively(self._list_all_with_hrefs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1241, in _test_list_pagination_iteratively
len(expected_resources), sort_args
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1302, in _list_all_with_hrefs
self.assertEqual(1, len(resources_))
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual
self.assertThat(observed, matcher, message)
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: 1 != 0
```
###### Reason for failure:
More neutron trunk ports are being returned than expected.
Code has to be fixed for 2 and 3 below such that trunks are returned only for one project (specify project_id in GET trunks call). A parallel test might be creating trunks so this test case is failing.
2. test_list_pagination_page_reverse_with_href_links
-> Expected returns trunks for project: 864acee2d6c64faa8750cfe53437a158
-> Actual paginated returns trunk for more projects: 89e63227c3b6405498f8fb1973cd055d and 864acee2d6c64faa8750cfe53437a158
3. test_list_pagination_with_href_links
Same issue as in 2.
###### Able to run the tests if run in serial mode: concurrency = 1
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [3.404399s] ... ok{0}
neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_href_links [9.412321s] ... ok
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_marker [4.566547s] ... ok{0}
neutron_tempest_plugin.api.test_ports.PortsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [4.636547s] ... ok
{0} neutron_tempest_plugin.api.test_subnets.SubnetsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [1.971421s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [0.881594s] ... ok
{0} neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links [2.173380s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_asc [0.180619s] ... ok
{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_desc [0.176105s] ... ok
======
Totals
======
Ran: 9 tests in 107.7512 sec.
Passed: 9
Skipped: 0
Expected Fail: 0
Unexpected Success: 0
Failed: 0
Sum of execute time for each test: 27.4029 sec.
==============
Worker Balance
==============
Worker 0 (9 tests) => 0:01:21.001099 |
|
| 2020-05-29 15:38:38 |
Sampat Ponnaganti |
description |
Neutron tempest Pagination tet cases fail if run in parallel.
##### Issue
The pagination test cases see items created by other test cases and hence expected vs actual differ and the test cases fail.
##### Proposed solution:
Update pagination test cases to query neutron resource only for a specific project.
Open to further discussion or any other solution.
For Eg:
###### These two test cases fail:
Test case 1:
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68]
```code
Traceback
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68] Traceback: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 353, in test_list_pagination_page_reverse_with_href_links self._test_list_pagination_page_reverse_with_href_links() File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1346, in _test_list_pagination_page_reverse_with_href_links self.assertSameOrder(expected_resources, reversed(resources)) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1160, in assertSameOrder self.assertEqual(len(original), len(actual)) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 5 != 6
```
Test case 2: neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links[id-dcd02a7a-f07e-4d5e-b0ca-b58e48927a9b]
```code
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 337, in test_list_pagination_with_href_links
self._test_list_pagination_with_href_links()
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1312, in _test_list_pagination_with_href_links
self._test_list_pagination_iteratively(self._list_all_with_hrefs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1241, in _test_list_pagination_iteratively
len(expected_resources), sort_args
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1302, in _list_all_with_hrefs
self.assertEqual(1, len(resources_))
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual
self.assertThat(observed, matcher, message)
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: 1 != 0
```
###### Reason for failure:
More neutron trunk ports are being returned than expected.
Code has to be fixed for 2 and 3 below such that trunks are returned only for one project (specify project_id in GET trunks call). A parallel test might be creating trunks so this test case is failing.
2. test_list_pagination_page_reverse_with_href_links
-> Expected returns trunks for project: 864acee2d6c64faa8750cfe53437a158
-> Actual paginated returns trunk for more projects: 89e63227c3b6405498f8fb1973cd055d and 864acee2d6c64faa8750cfe53437a158
3. test_list_pagination_with_href_links
Same issue as in 2.
###### Able to run the tests if run in serial mode: concurrency = 1
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [3.404399s] ... ok{0}
neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_href_links [9.412321s] ... ok
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_marker [4.566547s] ... ok{0}
neutron_tempest_plugin.api.test_ports.PortsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [4.636547s] ... ok
{0} neutron_tempest_plugin.api.test_subnets.SubnetsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [1.971421s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [0.881594s] ... ok
{0} neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links [2.173380s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_asc [0.180619s] ... ok
{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_desc [0.176105s] ... ok
======
Totals
======
Ran: 9 tests in 107.7512 sec.
Passed: 9
Skipped: 0
Expected Fail: 0
Unexpected Success: 0
Failed: 0
Sum of execute time for each test: 27.4029 sec.
==============
Worker Balance
==============
Worker 0 (9 tests) => 0:01:21.001099 |
Neutron tempest Pagination tet cases fail if run in parallel.
##### Issue
The pagination test cases see items created by other test cases and hence expected vs actual differ and the test cases fail.
##### Proposed solution:
1. Update pagination test cases to query neutron resource only for a specific project.
2. Check the project ID in expected test data and use that Project ID to match the results in Actual. Ignore any other Project ID
Open to further discussion or any other solution.
For Eg:
###### These two test cases fail:
Test case 1:
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68]
```code
Traceback
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68] Traceback: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 353, in test_list_pagination_page_reverse_with_href_links self._test_list_pagination_page_reverse_with_href_links() File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1346, in _test_list_pagination_page_reverse_with_href_links self.assertSameOrder(expected_resources, reversed(resources)) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1160, in assertSameOrder self.assertEqual(len(original), len(actual)) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 5 != 6
```
Test case 2: neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links[id-dcd02a7a-f07e-4d5e-b0ca-b58e48927a9b]
```code
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 337, in test_list_pagination_with_href_links
self._test_list_pagination_with_href_links()
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1312, in _test_list_pagination_with_href_links
self._test_list_pagination_iteratively(self._list_all_with_hrefs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1241, in _test_list_pagination_iteratively
len(expected_resources), sort_args
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1302, in _list_all_with_hrefs
self.assertEqual(1, len(resources_))
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual
self.assertThat(observed, matcher, message)
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: 1 != 0
```
###### Reason for failure:
More neutron trunk ports are being returned than expected.
Code has to be fixed for 2 and 3 below such that trunks are returned only for one project (specify project_id in GET trunks call). A parallel test might be creating trunks so this test case is failing.
2. test_list_pagination_page_reverse_with_href_links
-> Expected returns trunks for project: 864acee2d6c64faa8750cfe53437a158
-> Actual paginated returns trunk for more projects: 89e63227c3b6405498f8fb1973cd055d and 864acee2d6c64faa8750cfe53437a158
3. test_list_pagination_with_href_links
Same issue as in 2.
###### Able to run the tests if run in serial mode: concurrency = 1
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [3.404399s] ... ok{0}
neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_href_links [9.412321s] ... ok
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_marker [4.566547s] ... ok{0}
neutron_tempest_plugin.api.test_ports.PortsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [4.636547s] ... ok
{0} neutron_tempest_plugin.api.test_subnets.SubnetsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [1.971421s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [0.881594s] ... ok
{0} neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links [2.173380s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_asc [0.180619s] ... ok
{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_desc [0.176105s] ... ok
======
Totals
======
Ran: 9 tests in 107.7512 sec.
Passed: 9
Skipped: 0
Expected Fail: 0
Unexpected Success: 0
Failed: 0
Sum of execute time for each test: 27.4029 sec.
==============
Worker Balance
==============
Worker 0 (9 tests) => 0:01:21.001099 |
|
| 2020-05-29 15:39:02 |
Sampat Ponnaganti |
description |
Neutron tempest Pagination tet cases fail if run in parallel.
##### Issue
The pagination test cases see items created by other test cases and hence expected vs actual differ and the test cases fail.
##### Proposed solution:
1. Update pagination test cases to query neutron resource only for a specific project.
2. Check the project ID in expected test data and use that Project ID to match the results in Actual. Ignore any other Project ID
Open to further discussion or any other solution.
For Eg:
###### These two test cases fail:
Test case 1:
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68]
```code
Traceback
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68] Traceback: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 353, in test_list_pagination_page_reverse_with_href_links self._test_list_pagination_page_reverse_with_href_links() File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1346, in _test_list_pagination_page_reverse_with_href_links self.assertSameOrder(expected_resources, reversed(resources)) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1160, in assertSameOrder self.assertEqual(len(original), len(actual)) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 5 != 6
```
Test case 2: neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links[id-dcd02a7a-f07e-4d5e-b0ca-b58e48927a9b]
```code
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 337, in test_list_pagination_with_href_links
self._test_list_pagination_with_href_links()
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1312, in _test_list_pagination_with_href_links
self._test_list_pagination_iteratively(self._list_all_with_hrefs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1241, in _test_list_pagination_iteratively
len(expected_resources), sort_args
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1302, in _list_all_with_hrefs
self.assertEqual(1, len(resources_))
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual
self.assertThat(observed, matcher, message)
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: 1 != 0
```
###### Reason for failure:
More neutron trunk ports are being returned than expected.
Code has to be fixed for 2 and 3 below such that trunks are returned only for one project (specify project_id in GET trunks call). A parallel test might be creating trunks so this test case is failing.
2. test_list_pagination_page_reverse_with_href_links
-> Expected returns trunks for project: 864acee2d6c64faa8750cfe53437a158
-> Actual paginated returns trunk for more projects: 89e63227c3b6405498f8fb1973cd055d and 864acee2d6c64faa8750cfe53437a158
3. test_list_pagination_with_href_links
Same issue as in 2.
###### Able to run the tests if run in serial mode: concurrency = 1
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [3.404399s] ... ok{0}
neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_href_links [9.412321s] ... ok
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_marker [4.566547s] ... ok{0}
neutron_tempest_plugin.api.test_ports.PortsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [4.636547s] ... ok
{0} neutron_tempest_plugin.api.test_subnets.SubnetsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [1.971421s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [0.881594s] ... ok
{0} neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links [2.173380s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_asc [0.180619s] ... ok
{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_desc [0.176105s] ... ok
======
Totals
======
Ran: 9 tests in 107.7512 sec.
Passed: 9
Skipped: 0
Expected Fail: 0
Unexpected Success: 0
Failed: 0
Sum of execute time for each test: 27.4029 sec.
==============
Worker Balance
==============
Worker 0 (9 tests) => 0:01:21.001099 |
Neutron tempest Pagination tet cases fail if run in parallel.
##### Issue
The pagination test cases see items created by other test cases and hence expected vs actual differ and the test cases fail.
##### Proposed solution:
1. Update pagination test cases to query neutron resource only for a specific project.
OR
2. Check the project ID in expected test data and use that Project ID to match the results in Actual. Ignore any other Project ID
Open to further discussion or any other solution.
For Eg:
###### These two test cases fail:
Test case 1:
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68]
```code
Traceback
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links[id-b4293e59-d794-4a93-be09-38667199ef68] Traceback: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 353, in test_list_pagination_page_reverse_with_href_links self._test_list_pagination_page_reverse_with_href_links() File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1346, in _test_list_pagination_page_reverse_with_href_links self.assertSameOrder(expected_resources, reversed(resources)) File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1160, in assertSameOrder self.assertEqual(len(original), len(actual)) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 5 != 6
```
Test case 2: neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links[id-dcd02a7a-f07e-4d5e-b0ca-b58e48927a9b]
```code
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/test_trunk.py", line 337, in test_list_pagination_with_href_links
self._test_list_pagination_with_href_links()
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1132, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1123, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1312, in _test_list_pagination_with_href_links
self._test_list_pagination_iteratively(self._list_all_with_hrefs)
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1241, in _test_list_pagination_iteratively
len(expected_resources), sort_args
File "/usr/local/lib/python3.6/dist-packages/neutron_tempest_plugin/api/base.py", line 1302, in _list_all_with_hrefs
self.assertEqual(1, len(resources_))
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 411, in assertEqual
self.assertThat(observed, matcher, message)
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: 1 != 0
```
###### Reason for failure:
More neutron trunk ports are being returned than expected.
Code has to be fixed for 2 and 3 below such that trunks are returned only for one project (specify project_id in GET trunks call). A parallel test might be creating trunks so this test case is failing.
2. test_list_pagination_page_reverse_with_href_links
-> Expected returns trunks for project: 864acee2d6c64faa8750cfe53437a158
-> Actual paginated returns trunk for more projects: 89e63227c3b6405498f8fb1973cd055d and 864acee2d6c64faa8750cfe53437a158
3. test_list_pagination_with_href_links
Same issue as in 2.
###### Able to run the tests if run in serial mode: concurrency = 1
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [3.404399s] ... ok{0}
neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_href_links [9.412321s] ... ok
{0} neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_pagination_with_marker [4.566547s] ... ok{0}
neutron_tempest_plugin.api.test_ports.PortsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [4.636547s] ... ok
{0} neutron_tempest_plugin.api.test_subnets.SubnetsSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [1.971421s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_page_reverse_with_href_links [0.881594s] ... ok
{0} neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_pagination_with_href_links [2.173380s] ... ok{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_asc [0.180619s] ... ok
{0}
neutron_tempest_plugin.api.test_trunk.TrunksSearchCriteriaTest.test_list_sorts_desc [0.176105s] ... ok
======
Totals
======
Ran: 9 tests in 107.7512 sec.
Passed: 9
Skipped: 0
Expected Fail: 0
Unexpected Success: 0
Failed: 0
Sum of execute time for each test: 27.4029 sec.
==============
Worker Balance
==============
Worker 0 (9 tests) => 0:01:21.001099 |
|
| 2020-06-01 08:52:48 |
Martin Kopec |
affects |
tempest |
neutron |
|
