test_clusterip_service_endpoint reports unauthorized when trying to get pods

[Alexander Balderson]

Testing on baremetal
we ran into a case where test_cluster_service_endpoint failed when running /snap/bin/kubectl get po ... because of an Unauthorized, and we must be logged into the server; full error at the bottom of the report.

This is weird because the command is running from the k8s units themselves using juju run. We have a bug on baremetal about k8s master failing to connect to itself, LP: #1929234 it could be the same issue, but it manifested strangely, and everything is active-idle in juju status.

I'm going to submit a patch to the test to print the unit names that are checking the pod for some extra information; but something went wrong with the k8s unit.

Test run at: https://solutions.qa.canonical.com/testruns/testRun/81564c2d-702c-4f27-8a1a-a35c783a4915
Crashdump at: https://oil-jenkins.canonical.com/artifacts/81564c2d-702c-4f27-8a1a-a35c783a4915/generated/generated/kubernetes/juju-crashdump-kubernetes-2021-08-03-22.19.27.tar.gz
Test result xml at: https://oil-jenkins.canonical.com/artifacts/81564c2d-702c-4f27-8a1a-a35c783a4915/generated/generated/kubernetes/k8s-suite/test_clusterip_service_endpoint-junit.xml
all tests which hit this bug can be found at: https://solutions.qa.canonical.com/bugs/bugs/bug/1938926

______________________ test_clusterip_service_endpoint ________________________
Traceback (most recent call last):
  File "/home/ubuntu/k8s-validation/jobs/integration/test_service_endpoints.py", line 125, in test_clusterip_service_endpoint
    await cleanup()
  File "/home/ubuntu/k8s-validation/jobs/integration/test_service_endpoints.py", line 69, in cleanup
    await retry_async_with_timeout(
  File "/home/ubuntu/k8s-validation/jobs/integration/utils.py", line 198, in retry_async_with_timeout
    if await func(*args):
  File "/home/ubuntu/k8s-validation/jobs/integration/test_service_endpoints.py", line 34, in is_pod_cleaned
    pod = get_pod_yaml()
  File "/home/ubuntu/k8s-validation/jobs/integration/test_service_endpoints.py", line 9, in get_pod_yaml
    out = sh.kubectl.get("po", o="yaml", selector="app=hello-world")
  File "/home/ubuntu/k8s-validation/.tox/py3/lib/python3.8/site-packages/sh.py", line 1520, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/home/ubuntu/k8s-validation/.tox/py3/lib/python3.8/site-packages/sh.py", line 784, in __init__
    self.wait()
  File "/home/ubuntu/k8s-validation/.tox/py3/lib/python3.8/site-packages/sh.py", line 841, in wait
    self.handle_command_exit_code(exit_code)
  File "/home/ubuntu/k8s-validation/.tox/py3/lib/python3.8/site-packages/sh.py", line 865, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1:

  RAN: /snap/bin/kubectl get po -o yaml --selector=app=hello-world

  STDOUT:
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

  STDERR:
error: You must be logged in to the server (Unauthorized)