failed e2e conformance tests 1.29

[Anna Savchenko]

Summarizing 5 Failures:
  [FAIL] [sig-network] Services [It] should be able to change the type from NodePort to ExternalName [Conformance] [sig-network, Conformance]
  k8s.io/kubernetes/test/e2e/network/service.go:1565
  [FAIL] [sig-network] DNS [It] should provide DNS for services [Conformance] [sig-network, Conformance]
  k8s.io/kubernetes/test/e2e/network/dns_common.go:476
  [FAIL] [sig-network] Services [It] should be able to change the type from ClusterIP to ExternalName [Conformance] [sig-network, Conformance]
  k8s.io/kubernetes/test/e2e/network/service.go:1523
  [FAIL] [sig-network] DNS [It] should provide DNS for the cluster [Conformance] [sig-network, Conformance]
  k8s.io/kubernetes/test/e2e/network/dns_common.go:459
  [FAIL] [sig-network] DNS [It] should provide DNS for pods for Subdomain [Conformance] [sig-network, Conformance]
  k8s.io/kubernetes/test/e2e/network/dns_common.go:476

Looking at "should provide DNS for services" logs, looks like it cannot find the check results:

  Jun 19 14:22:03.778: INFO: Unable to read <email address hidden> from pod dns-9735/dns-test-c9a07dbc-88f9-4852-aa73-7754e00ba1b2: the server could not find the requested resource (get pods dns-test-c9a07dbc-88f9-4852-aa73-7754e00ba1b2)
  Jun 19 14:22:03.790: INFO: Lookups using dns-9735/dns-test-c9a07dbc-88f9-4852-aa73-7754e00ba1b2 failed for: [<email address hidden> <email address hidden> <email address hidden> <email address hidden>]

When I logged in to the querier container, indeed I did not see any files in results directory.
I tried manually running the check in "should provide DNS for services" and could generate the results.

$ kubectl exec -it dns-test-aa59d657-7a32-4f1e-82b9-658bf1720150 -c querier -n dns-3271 -- sh
/ # ls results/
/ # for i in `seq 1 600`; do check="$$(dig +notcp +noall +answer +search kubernetes.default.svc.cluster.local A)" && test -n "$$check" && echo OK > /<email address hidden>;check=
"$$(dig +tcp +noall +answer +search kubernetes.default.svc.cluster.local A)" && test -n "$$check" && echo OK > /<email address hidden>;sleep 1; done
command terminated with exit code 137
/ # ls results/
<email address hidden> <email address hidden>
/ # cat <email address hidden>
OK
/ # cat <email address hidden>
OK

I did not try to execute the other 4 tests manually.

Charmed Kubernetes - 1.29
Sonobuoy version - v0.57.1
Juju - 3.5.0

This is an air-gapped environment (CDK on OpenStack).

[Anna Savchenko (annsavchenko)]

The command I used to run the tests:

$ sonobuoy run --plugin-env=e2e.E2E_EXTRA_ARGS=--non-blocking-taints=node-role.kubernetes.io/control-plane --sonobuoy-image 10.131.38.71:5000/sonobuoy:v0.57.1 --kube-conformance-image 10.131.38.71:5000/conformance:v1.29.5 --systemd-logs-image 10.131.38.71:5000/heptio-images/sonobuoy-plugin-systemd-logs:latest --e2e-repo-config sonobuoy-image-config --e2e-focus "should provide DNS for the cluster|should be able to change the type from NodePort to ExternalName|should provide DNS for pods for Subdomain|should provide DNS for services|should be able to change the type from ClusterIP to ExternalName" --wait

[Anna Savchenko (annsavchenko)]

Subscribed Field High as this blocks the handover of the K8s Cluster.

[Anna Savchenko (annsavchenko)]

I'm not sure if the issue is with the missing files.
I tried to run the commands executed in the abovementioned e2e tests and noticed that the dig and nslookup commands failed.

It looks like the e2e test tries to resolve kubernetes.default.svc.cluster.local instead of kubernetes.default.svc.cluster.local. (note the dot)

nslookup fails with "server can't find nodeport-service.services-5134.svc.cluster.local.redacted.domain: REFUSED"

I can see that redacted.domain is in /etc/resolv.conf

/ # cat /etc/resolv.conf
search services-5134.svc.cluster.local svc.cluster.local cluster.local redacted.domain
nameserver 10.152.183.79
options ndots:5

So either resolving kubernetes.default.svc.cluster.local. (instead of kubernetes.default.svc.cluster.local) or setting ndots to 4 works.

$ kubectl exec -it execpods2flb -n services-5134 -- sh
/ # nslookup nodeport-service.services-5134.svc.cluster.local
Server: 10.152.183.79
Address: 10.152.183.79#53

** server can't find nodeport-service.services-5134.svc.cluster.local.redacted.domain: REFUSED

/ # nslookup nodeport-service.services-5134.svc.cluster.local.
Server: 10.152.183.79
Address: 10.152.183.79#53

nodeport-service.services-5134.svc.cluster.local canonical name = externalsvc.services-5134.svc.cluster.local.
Name: externalsvc.services-5134.svc.cluster.local
Address: 10.152.183.38