using ipip in place of vxlan causes loss of pod-pod communcation

[Jeff Hillman]

K8s 1.19
docker 18.6.3
latest/stable charms (except docker which is edge)

When using the attached bundle in either AWS or O7k, no communication appears to happen between pods or services.

Specifically, no internal DNS resolution works, even a telnet to port 53 of the coredns service fails from within a pod. cannot pull from an clusterIP service that is running a webserver.

Also of note, ingress doesn't properly route between pods. in that, if a ClusterIP ingress is created, for example, with webserver.<random-worker-ip>.xip.io and that worker IP isn't the host that the pod is running on, no traffic is returned on a curl. if the worker IP is set to that of which the pod is running on, then it will return.

during a live deploy, switching from IPIP=Alwasy to IPIP=Never, and then vxlan=Never to vxlan=Always, traffic immediately starts flowing. Both for DNS resolution and proper behavior of ClusterIP Ingress.

This is easily reproduceable and happens 100% of the time.

The image of hillmanj/net-tools:latest has network tooling to assist with testing/validation.