Activity log for bug #1806026

Date Who What changed Old value New value Message
2018-11-30 12:07:20 Paweł Zadrożny bug added bug
2018-11-30 12:07:28 Paweł Zadrożny information type Proprietary Public
2018-11-30 12:08:00 Paweł Zadrożny tags go-api
2018-12-04 11:56:52 Witaut Bajaryn summary Issue with resource cleanup on test tear down VMs remain in agent on test teardown
2018-12-04 11:59:58 Witaut Bajaryn description Issue with resource cleanup on test tear down `test_deny_all_ingress` test case passed, but fails when cleanup is run: ``` 2018-11-27 14:49:58,116 - INFO - ================================================================================ 2018-11-27 14:49:58,116 - INFO - STARTING TEST : test_deny_all_ingress 2018-11-27 14:49:58,117 - INFO - TEST DESCRIPTION : Verify a network policy with Ingress rule as deny all. Steps: 1. Verify that all pods are reachable from all namespaces 2. Create a Network policy on "default" namespace. 3. Verify that ingress is not allowed to pods of namespace "default" 4. Verify that egress is allowed from pods of namespace "default" 2018-11-27 14:50:01,190 - INFO - Initial checks done. Running the testcase now 2018-11-27 14:50:01,190 - INFO - 2018-11-27 14:50:03,536 - INFO - Ping check to IP 10.32.0.5 from pod ctest-pod-79545136 with expectation True passed 2018-11-27 14:50:05,675 - INFO - Ping check to IP 10.32.0.4 from pod ctest-pod-03844500 with expectation True passed 2018-11-27 14:50:07,817 - INFO - Ping check to IP 10.32.0.6 from pod ctest-pod-79545136 with expectation True passed 2018-11-27 14:50:07,823 - INFO - Creating Network Policy deny-all-ingress 2018-11-27 14:50:07,838 - INFO - Network policy found in k8s 2018-11-27 14:50:07,848 - INFO - Network Policy deny-all-ingress with uuid bb976cc0-f253-11e8-b32c-fa163e8145ef found in kube manager 2018-11-27 14:50:07,885 - INFO - Network policy with name default-policy-management:k8s-allowall found in agent 2018-11-27 14:50:07,886 - INFO - Network policy with name default-policy-management:k8s-allowall associated with default ks8 2018-11-27 14:50:07,914 - WARNING - Network policy with name deny-all-ingress not found in agent 2018-11-27 14:50:09,954 - WARNING - Network policy with name deny-all-ingress not found in agent 2018-11-27 14:50:12,000 - INFO - Network policy with name deny-all-ingress found in agent 2018-11-27 14:50:12,001 - INFO - Network Policy deny-all-ingress verification passed 2018-11-27 14:50:24,134 - INFO - Ping check to IP 10.32.0.5 from pod ctest-pod-79545136 with expectation False passed 2018-11-27 14:50:36,262 - INFO - Ping check to IP 10.32.0.4 from pod ctest-pod-03844500 with expectation False passed 2018-11-27 14:50:38,394 - INFO - Ping check to IP 10.32.0.6 from pod ctest-pod-79545136 with expectation True passed 2018-11-27 14:50:40,530 - INFO - Ping check to IP 10.32.0.6 from pod ctest-pod-16369978 with expectation True passed 2018-11-27 14:50:40,531 - INFO - Deleting Network Policy : deny-all-ingress 2018-11-27 14:50:42,832 - INFO - END TEST : test_deny_all_ingress : PASSED[0:00:44] 2018-11-27 14:50:42,832 - INFO - -------------------------------------------------------------------------------- �/�@s[�Y���~�@Tscripts.k8s_scripts.test_policy.TestNetworkPolicy.test_deny_all_ingress[openshift_1worker-00�'�B2018-11-27 14:50:42,840 - INFO - Deleting pod default:ctest-pod-11306003 2018-11-27 14:50:42,842 - INFO - Deleting pod non-default:ctest-pod-33201647 2018-11-27 14:50:42,843 - INFO - Deleting pod default:ctest-pod-79545136 2018-11-27 14:50:42,844 - INFO - Deleting pod default:ctest-pod-86017866 2018-11-27 14:50:42,845 - INFO - Deleting pod non-default:ctest-pod-03844500 2018-11-27 14:50:42,845 - INFO - Deleting pod non-default:ctest-pod-16369978 2018-11-27 14:50:42,849 - INFO - Deleting pod temp-ns:ctest-pod-32499572 2018-11-27 14:50:42,850 - INFO - Deleting pod temp-ns:ctest-pod-63278658 2018-11-27 14:50:42,851 - INFO - Deleting pod temp-ns:ctest-pod-34385238 2018-11-27 14:50:42,912 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,128 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,190 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,191 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,193 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,195 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,196 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,198 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,199 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:47,921 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,136 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,220 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,221 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,223 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,224 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,225 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,226 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,228 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:52,931 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,146 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,255 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,257 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,259 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,260 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,262 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,264 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,265 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:57,944 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,156 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,283 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,285 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,287 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,288 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,290 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,291 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,292 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:02,952 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,165 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,315 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,316 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,318 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,320 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,321 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,322 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,324 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:07,961 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,173 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,343 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,345 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,347 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,348 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,350 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,352 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,355 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:12,969 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,186 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,371 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,373 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,374 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,377 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,379 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,381 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,382 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:17,977 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,194 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,394 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,396 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,398 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,399 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,402 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,403 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,405 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:22,988 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,203 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,423 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,425 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,426 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,428 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,429 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,430 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,431 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,002 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,210 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,452 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,453 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,455 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,456 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,457 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,463 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,465 - INFO - Verified that pod ctest-pod-34385238 is removed in agent 2018-11-27 14:51:33,012 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-11306003 cleanup checks in agent failed <Greenlet at 0x7fbd80f53d70: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80ea7dd0>>> failed with AssertionError 2018-11-27 14:51:33,221 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-79545136 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd370: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e3c610>>> failed with AssertionError 2018-11-27 14:51:33,472 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-03844500 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd4b0: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e526d0>>> failed with AssertionError 2018-11-27 14:51:33,474 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-63278658 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd5f0: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e52850>>> failed with AssertionError 2018-11-27 14:51:33,476 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-32499572 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd690: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e52a10>>> failed with AssertionError 2018-11-27 14:51:33,478 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-86017866 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd230: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80f3f5d0>>> failed with AssertionError 2018-11-27 14:51:33,480 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-16369978 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd730: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e52d90>>> failed with AssertionError 2018-11-27 14:51:33,482 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-33201647 cleanup checks in agent failed <Greenlet at 0x7fbd80f53c30: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80ea7950>>> failed with AssertionError 2018-11-27 14:51:33,483 - INFO - Deleting namespace non-default 2018-11-27 14:51:39,540 - INFO - Deleting namespace temp-ns in serial_run_test serial_scripts/control_node_scaling/bgp_scale.py:823: SyntaxWarning: import * only allowed at module level def get_localhost_ip(): [localhost] local: rpm -q --queryformat '%{VERSION}-' contrail-test; rpm -q --queryformat '%{RELEASE}' contrail-test | awk -F'.' '{print $1}' Generated report_details* file: report_details_2018_11_27_14_48_25.ini could not change test cases names Buildfile: /contrail-test/build.xml ```
2018-12-04 12:00:10 Witaut Bajaryn description `test_deny_all_ingress` test case passed, but fails when cleanup is run: ``` 2018-11-27 14:49:58,116 - INFO - ================================================================================ 2018-11-27 14:49:58,116 - INFO - STARTING TEST : test_deny_all_ingress 2018-11-27 14:49:58,117 - INFO - TEST DESCRIPTION : Verify a network policy with Ingress rule as deny all. Steps: 1. Verify that all pods are reachable from all namespaces 2. Create a Network policy on "default" namespace. 3. Verify that ingress is not allowed to pods of namespace "default" 4. Verify that egress is allowed from pods of namespace "default" 2018-11-27 14:50:01,190 - INFO - Initial checks done. Running the testcase now 2018-11-27 14:50:01,190 - INFO - 2018-11-27 14:50:03,536 - INFO - Ping check to IP 10.32.0.5 from pod ctest-pod-79545136 with expectation True passed 2018-11-27 14:50:05,675 - INFO - Ping check to IP 10.32.0.4 from pod ctest-pod-03844500 with expectation True passed 2018-11-27 14:50:07,817 - INFO - Ping check to IP 10.32.0.6 from pod ctest-pod-79545136 with expectation True passed 2018-11-27 14:50:07,823 - INFO - Creating Network Policy deny-all-ingress 2018-11-27 14:50:07,838 - INFO - Network policy found in k8s 2018-11-27 14:50:07,848 - INFO - Network Policy deny-all-ingress with uuid bb976cc0-f253-11e8-b32c-fa163e8145ef found in kube manager 2018-11-27 14:50:07,885 - INFO - Network policy with name default-policy-management:k8s-allowall found in agent 2018-11-27 14:50:07,886 - INFO - Network policy with name default-policy-management:k8s-allowall associated with default ks8 2018-11-27 14:50:07,914 - WARNING - Network policy with name deny-all-ingress not found in agent 2018-11-27 14:50:09,954 - WARNING - Network policy with name deny-all-ingress not found in agent 2018-11-27 14:50:12,000 - INFO - Network policy with name deny-all-ingress found in agent 2018-11-27 14:50:12,001 - INFO - Network Policy deny-all-ingress verification passed 2018-11-27 14:50:24,134 - INFO - Ping check to IP 10.32.0.5 from pod ctest-pod-79545136 with expectation False passed 2018-11-27 14:50:36,262 - INFO - Ping check to IP 10.32.0.4 from pod ctest-pod-03844500 with expectation False passed 2018-11-27 14:50:38,394 - INFO - Ping check to IP 10.32.0.6 from pod ctest-pod-79545136 with expectation True passed 2018-11-27 14:50:40,530 - INFO - Ping check to IP 10.32.0.6 from pod ctest-pod-16369978 with expectation True passed 2018-11-27 14:50:40,531 - INFO - Deleting Network Policy : deny-all-ingress 2018-11-27 14:50:42,832 - INFO - END TEST : test_deny_all_ingress : PASSED[0:00:44] 2018-11-27 14:50:42,832 - INFO - -------------------------------------------------------------------------------- �/�@s[�Y���~�@Tscripts.k8s_scripts.test_policy.TestNetworkPolicy.test_deny_all_ingress[openshift_1worker-00�'�B2018-11-27 14:50:42,840 - INFO - Deleting pod default:ctest-pod-11306003 2018-11-27 14:50:42,842 - INFO - Deleting pod non-default:ctest-pod-33201647 2018-11-27 14:50:42,843 - INFO - Deleting pod default:ctest-pod-79545136 2018-11-27 14:50:42,844 - INFO - Deleting pod default:ctest-pod-86017866 2018-11-27 14:50:42,845 - INFO - Deleting pod non-default:ctest-pod-03844500 2018-11-27 14:50:42,845 - INFO - Deleting pod non-default:ctest-pod-16369978 2018-11-27 14:50:42,849 - INFO - Deleting pod temp-ns:ctest-pod-32499572 2018-11-27 14:50:42,850 - INFO - Deleting pod temp-ns:ctest-pod-63278658 2018-11-27 14:50:42,851 - INFO - Deleting pod temp-ns:ctest-pod-34385238 2018-11-27 14:50:42,912 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,128 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,190 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,191 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,193 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,195 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,196 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,198 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,199 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:47,921 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,136 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,220 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,221 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,223 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,224 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,225 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,226 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,228 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:52,931 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,146 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,255 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,257 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,259 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,260 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,262 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,264 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,265 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:57,944 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,156 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,283 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,285 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,287 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,288 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,290 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,291 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,292 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:02,952 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,165 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,315 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,316 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,318 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,320 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,321 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,322 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,324 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:07,961 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,173 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,343 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,345 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,347 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,348 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,350 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,352 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,355 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:12,969 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,186 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,371 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,373 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,374 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,377 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,379 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,381 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,382 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:17,977 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,194 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,394 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,396 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,398 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,399 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,402 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,403 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,405 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:22,988 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,203 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,423 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,425 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,426 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,428 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,429 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,430 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,431 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,002 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,210 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,452 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,453 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,455 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,456 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,457 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,463 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,465 - INFO - Verified that pod ctest-pod-34385238 is removed in agent 2018-11-27 14:51:33,012 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-11306003 cleanup checks in agent failed <Greenlet at 0x7fbd80f53d70: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80ea7dd0>>> failed with AssertionError 2018-11-27 14:51:33,221 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-79545136 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd370: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e3c610>>> failed with AssertionError 2018-11-27 14:51:33,472 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-03844500 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd4b0: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e526d0>>> failed with AssertionError 2018-11-27 14:51:33,474 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-63278658 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd5f0: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e52850>>> failed with AssertionError 2018-11-27 14:51:33,476 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-32499572 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd690: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e52a10>>> failed with AssertionError 2018-11-27 14:51:33,478 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-86017866 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd230: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80f3f5d0>>> failed with AssertionError 2018-11-27 14:51:33,480 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-16369978 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd730: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e52d90>>> failed with AssertionError 2018-11-27 14:51:33,482 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run result = self._run(*self.args, **self.kwargs) File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp self.delete() File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete assert self.verify_on_cleanup() File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-33201647 cleanup checks in agent failed <Greenlet at 0x7fbd80f53c30: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80ea7950>>> failed with AssertionError 2018-11-27 14:51:33,483 - INFO - Deleting namespace non-default 2018-11-27 14:51:39,540 - INFO - Deleting namespace temp-ns in serial_run_test serial_scripts/control_node_scaling/bgp_scale.py:823: SyntaxWarning: import * only allowed at module level def get_localhost_ip(): [localhost] local: rpm -q --queryformat '%{VERSION}-' contrail-test; rpm -q --queryformat '%{RELEASE}' contrail-test | awk -F'.' '{print $1}' Generated report_details* file: report_details_2018_11_27_14_48_25.ini could not change test cases names Buildfile: /contrail-test/build.xml ``` `test_deny_all_ingress` test case passes, but then fails when cleanup is run: ``` 2018-11-27 14:49:58,116 - INFO - ================================================================================ 2018-11-27 14:49:58,116 - INFO - STARTING TEST : test_deny_all_ingress 2018-11-27 14:49:58,117 - INFO - TEST DESCRIPTION :         Verify a network policy with Ingress rule as deny all.         Steps:         1. Verify that all pods are reachable from all namespaces         2. Create a Network policy on "default" namespace.         3. Verify that ingress is not allowed to pods of namespace "default"         4. Verify that egress is allowed from pods of namespace "default" 2018-11-27 14:50:01,190 - INFO - Initial checks done. Running the testcase now 2018-11-27 14:50:01,190 - INFO - 2018-11-27 14:50:03,536 - INFO - Ping check to IP 10.32.0.5 from pod ctest-pod-79545136 with expectation True passed 2018-11-27 14:50:05,675 - INFO - Ping check to IP 10.32.0.4 from pod ctest-pod-03844500 with expectation True passed 2018-11-27 14:50:07,817 - INFO - Ping check to IP 10.32.0.6 from pod ctest-pod-79545136 with expectation True passed 2018-11-27 14:50:07,823 - INFO - Creating Network Policy deny-all-ingress 2018-11-27 14:50:07,838 - INFO - Network policy found in k8s 2018-11-27 14:50:07,848 - INFO - Network Policy deny-all-ingress with uuid bb976cc0-f253-11e8-b32c-fa163e8145ef found in kube manager 2018-11-27 14:50:07,885 - INFO - Network policy with name default-policy-management:k8s-allowall found in agent 2018-11-27 14:50:07,886 - INFO - Network policy with name default-policy-management:k8s-allowall associated with default ks8 2018-11-27 14:50:07,914 - WARNING - Network policy with name deny-all-ingress not found in agent 2018-11-27 14:50:09,954 - WARNING - Network policy with name deny-all-ingress not found in agent 2018-11-27 14:50:12,000 - INFO - Network policy with name deny-all-ingress found in agent 2018-11-27 14:50:12,001 - INFO - Network Policy deny-all-ingress verification passed 2018-11-27 14:50:24,134 - INFO - Ping check to IP 10.32.0.5 from pod ctest-pod-79545136 with expectation False passed 2018-11-27 14:50:36,262 - INFO - Ping check to IP 10.32.0.4 from pod ctest-pod-03844500 with expectation False passed 2018-11-27 14:50:38,394 - INFO - Ping check to IP 10.32.0.6 from pod ctest-pod-79545136 with expectation True passed 2018-11-27 14:50:40,530 - INFO - Ping check to IP 10.32.0.6 from pod ctest-pod-16369978 with expectation True passed 2018-11-27 14:50:40,531 - INFO - Deleting Network Policy : deny-all-ingress 2018-11-27 14:50:42,832 - INFO - END TEST : test_deny_all_ingress : PASSED[0:00:44] 2018-11-27 14:50:42,832 - INFO - -------------------------------------------------------------------------------- �/�@s[�Y���~�@Tscripts.k8s_scripts.test_policy.TestNetworkPolicy.test_deny_all_ingress[openshift_1worker-00�'�B2018-11-27 14:50:42,840 - INFO - Deleting pod default:ctest-pod-11306003 2018-11-27 14:50:42,842 - INFO - Deleting pod non-default:ctest-pod-33201647 2018-11-27 14:50:42,843 - INFO - Deleting pod default:ctest-pod-79545136 2018-11-27 14:50:42,844 - INFO - Deleting pod default:ctest-pod-86017866 2018-11-27 14:50:42,845 - INFO - Deleting pod non-default:ctest-pod-03844500 2018-11-27 14:50:42,845 - INFO - Deleting pod non-default:ctest-pod-16369978 2018-11-27 14:50:42,849 - INFO - Deleting pod temp-ns:ctest-pod-32499572 2018-11-27 14:50:42,850 - INFO - Deleting pod temp-ns:ctest-pod-63278658 2018-11-27 14:50:42,851 - INFO - Deleting pod temp-ns:ctest-pod-34385238 2018-11-27 14:50:42,912 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,128 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,190 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,191 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,193 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,195 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,196 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,198 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:43,199 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:47,921 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,136 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,220 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,221 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,223 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,224 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,225 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,226 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:48,228 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:52,931 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,146 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,255 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,257 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,259 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,260 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,262 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,264 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:53,265 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:57,944 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,156 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,283 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,285 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,287 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,288 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,290 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,291 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:50:58,292 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:02,952 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,165 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,315 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,316 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,318 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,320 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,321 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,322 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:03,324 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:07,961 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,173 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,343 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,345 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,347 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,348 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,350 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,352 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:08,355 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:12,969 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,186 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,371 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,373 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,374 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,377 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,379 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,381 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:13,382 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:17,977 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,194 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,394 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,396 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,398 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,399 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,402 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,403 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:18,405 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:22,988 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,203 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,423 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,425 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,426 - WARNING - Pod uuid 9898e467-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,428 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,429 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,430 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:23,431 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,002 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,210 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,452 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,453 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,455 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,456 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,457 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,463 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list 2018-11-27 14:51:28,465 - INFO - Verified that pod ctest-pod-34385238 is removed in agent 2018-11-27 14:51:33,012 - WARNING - Pod uuid 95479399-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last):   File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run     result = self._run(*self.args, **self.kwargs)   File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp     self.delete()   File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete     assert self.verify_on_cleanup()   File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup     ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-11306003 cleanup checks in agent failed <Greenlet at 0x7fbd80f53d70: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80ea7dd0>>> failed with AssertionError 2018-11-27 14:51:33,221 - WARNING - Pod uuid 9593bcca-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last):   File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run     result = self._run(*self.args, **self.kwargs)   File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp     self.delete()   File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete     assert self.verify_on_cleanup()   File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup     ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-79545136 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd370: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e3c610>>> failed with AssertionError 2018-11-27 14:51:33,472 - WARNING - Pod uuid 96433da3-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last):   File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run     result = self._run(*self.args, **self.kwargs)   File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp     self.delete()   File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete     assert self.verify_on_cleanup()   File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup     ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-03844500 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd4b0: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e526d0>>> failed with AssertionError 2018-11-27 14:51:33,474 - WARNING - Pod uuid 9828cef0-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last):   File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run     result = self._run(*self.args, **self.kwargs)   File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp     self.delete()   File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete     assert self.verify_on_cleanup()   File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup     ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-63278658 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd5f0: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e52850>>> failed with AssertionError 2018-11-27 14:51:33,476 - WARNING - Pod uuid 97470c83-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last):   File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run     result = self._run(*self.args, **self.kwargs)   File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp     self.delete()   File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete     assert self.verify_on_cleanup()   File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup     ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-32499572 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd690: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e52a10>>> failed with AssertionError 2018-11-27 14:51:33,478 - WARNING - Pod uuid 960e23f5-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last):   File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run     result = self._run(*self.args, **self.kwargs)   File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp     self.delete()   File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete     assert self.verify_on_cleanup()   File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup     ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-86017866 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd230: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80f3f5d0>>> failed with AssertionError 2018-11-27 14:51:33,480 - WARNING - Pod uuid 968bd046-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last):   File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run     result = self._run(*self.args, **self.kwargs)   File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp     self.delete()   File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete     assert self.verify_on_cleanup()   File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup     ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-16369978 cleanup checks in agent failed <Greenlet at 0x7fbd80dbd730: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80e52d90>>> failed with AssertionError 2018-11-27 14:51:33,482 - WARNING - Pod uuid 95656a22-f253-11e8-b32c-fa163e8145ef is still seen in agent 192.168.0.19 VM list Traceback (most recent call last):   File "/usr/lib64/python2.7/site-packages/gevent/greenlet.py", line 534, in run     result = self._run(*self.args, **self.kwargs)   File "/contrail-test/fixtures/k8s/pod.py", line 82, in cleanUp     self.delete()   File "/contrail-test/fixtures/k8s/pod.py", line 145, in delete     assert self.verify_on_cleanup()   File "/contrail-test/fixtures/k8s/pod.py", line 152, in verify_on_cleanup     ' in agent failed' % (self.name)) AssertionError: Pod ctest-pod-33201647 cleanup checks in agent failed <Greenlet at 0x7fbd80f53c30: <bound method PodFixture.cleanUp of <k8s.pod.PodFixture object at 0x7fbd80ea7950>>> failed with AssertionError 2018-11-27 14:51:33,483 - INFO - Deleting namespace non-default 2018-11-27 14:51:39,540 - INFO - Deleting namespace temp-ns in serial_run_test serial_scripts/control_node_scaling/bgp_scale.py:823: SyntaxWarning: import * only allowed at module level   def get_localhost_ip(): [localhost] local: rpm -q --queryformat '%{VERSION}-' contrail-test; rpm -q --queryformat '%{RELEASE}' contrail-test | awk -F'.' '{print $1}' Generated report_details* file: report_details_2018_11_27_14_48_25.ini could not change test cases names Buildfile: /contrail-test/build.xml ```