1.19 List index out of range when trying to get phase for test_clusterip_service_endpoint

[Alexander Balderson]

Using hash ea33970, the same hash used to test the 1.19 release, test service endpoint is failing to find the phase for the hello-world container. I'm not seeing anything in the logs as to why that data would have been missing from the pod status, other than get po returning an empty list.

the testrun with crashdump can be found here:
https://solutions.qa.canonical.com/testruns/testRun/76eb0f01-f935-41a7-96b3-25836b2076ff

ERROR: py3: commands failed

2020-11-07-19:53:06 root ERROR [localhost] STDERR follows:
None
2020-11-07 19:53:06.597915 Traceback (most recent call last):
2020-11-07 19:53:06.597930 File "/home/ubuntu/k8s-validation/jobs/integration/test_service_endpoints.py", line 93, in test_clusterip_service_endpoint
2020-11-07 19:53:06.597943 await setup_svc("ClusterIP")
2020-11-07 19:53:06.597955 File "/home/ubuntu/k8s-validation/jobs/integration/test_service_endpoints.py", line 52, in setup_svc
2020-11-07 19:53:06.597968 await retry_async_with_timeout(is_pod_running, ())
2020-11-07 19:53:06.597980 File "/home/ubuntu/k8s-validation/jobs/integration/utils.py", line 198, in retry_async_with_timeout
2020-11-07 19:53:06.597993 if await func(*args):
2020-11-07 19:53:06.598006 File "/home/ubuntu/k8s-validation/jobs/integration/test_service_endpoints.py", line 20, in is_pod_running
2020-11-07 19:53:06.598019 phase = pod["items"][0]["status"]["phase"]
2020-11-07 19:53:06.598031 IndexError: list index out of range
2020-11-07 19:53:06.598043 ------------------------------ Captured log setup ------------------------------
2020-11-07 19:53:06.598056 WARNING juju.client.connection:connection.py:706 unknown facade CAASModelOperator
2020-11-07 19:53:06.598068 WARNING juju.client.connection:connection.py:730 unexpected facade CAASModelOperator found, unable to decipher version to use
2020-11-07 19:53:06.598081 WARNING juju.model:model.py:905 unknown delta type: id
2020-11-07 19:53:06.598093 ------------------------------ Captured log call -------------------------------
2020-11-07 19:53:06.598106 INFO sh.command:sh.py:638 <Command '/snap/bin/kubectl create deployment hello-world --image=gcr.io/google-samples/node-hello:1.0', pid 26266>: process started
2020-11-07 19:53:06.598118 INFO sh.command:sh.py:638 <Command '/snap/bin/kubectl set env deployment/hello-world PORT=50000', pid 26299>: process started
2020-11-07 19:53:06.598131 INFO sh.command:sh.py:638 <Command '/snap/bin/kubectl expose deployment hello-world --type=ClusterIP --name=hello-world --protocol=TCP --port=80 --target-port=50000', pid 26332>: process started
2020-11-07 19:53:06.598145 INFO sh.command:sh.py:638 <Command '/snap/bin/kubectl get po -o yaml --selector=app=hello-world', pid 26365>: process started
2020-11-07 19:53:06.598157 INFO sh.command:sh.py:638 <Command '/snap/bin/kubectl delete deployment hello-world', pid 26398>: process started
2020-11-07 19:53:06.598170 INFO sh.command:sh.py:638 <Command '/snap/bin/kubectl delete service hello-world', pid 26431>: process started
2020-11-07 19:53:06.598182 INFO sh.command:sh.py:638 <Command '/snap/bin/kubectl get po -o yaml --selector=app=hello-world', pid 26464>: process started

[George Kraft (cynerva)]

It looks like the is_pod_running function doesn't properly handle the case where the Pod doesn't exist yet. It's entirely possible that the Pod would not exist at that point, since the test does not create the Pod directly. The test creates a Deployment, and there may be a delay before Kubernetes gets around to creating the Pods that go with it.