No easy way to retrieve pod fqdn

[David]

For stateful services it is important to expose the individual pod hostnames to clients. Pod IPs could change, while hostnames remain the same. At the same time, in order for pods hostnames to be used across models, they need to be fully qualified domain names:

Manual steps to get the pod hostname involve running network-get to retrieve the bind-address, and then running a reverse dns resolution via socket.getnameinfo():

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
test microk8s-localhost microk8s/localhost 2.7.3 unsupported 16:57:59+01:00

App Version Status Scale Charm Store Rev OS Address Notes
mongodb-k8s mongo:bionic active 1 mongodb-k8s jujucharms 26 kubernetes 10.152.183.207

Unit Workload Agent Address Ports Message
mongodb-k8s/1* active idle 10.1.81.163 27017/TCP ready

$ microk8s.kubectl -n test exec -it mongodb-k8s-operator-0 bash
root@mongodb-k8s-operator-0:/var/lib/juju# juju-run mongodb-k8s/1 'network-get mongo'
bind-addresses:
- macaddress: ""
  interfacename: ""
  addresses:
  - hostname: ""
    address: 10.1.81.163
    cidr: ""
egress-subnets:
- 10.152.183.207/32
ingress-addresses:
- 10.152.183.207
root@mongodb-k8s-operator-0:/var/lib/juju# python3
>>> import socket
>>> socket.getnameinfo(("10.1.81.163", 0), socket.NI_NAMEREQD)[0]
'mongodb-k8s-0.mongodb-k8s-endpoints.test.svc.cluster.local'

[Thomas Miller (tlmiller)]

Hi David,

Thanks for the bug report. This is correct we currently do no expose FQDN of pods through juju. This would obviously take the form of pod.namespace.svc.cluster-domain. Before we dig into this any further are you able to supply the exact use case of what is trying to be achieved with all the actors involved?

Ta
tlm

[Ian Booth (wallyworld)]

Normally you'd access pods via a service, not directly; this allows the pods to come and go but the service is the stable endpoint for those pods. As Tom says, let's explore the use case.

The hostname field info network-get is only filled out if Juju can detect the hostname for the interface and if it is resolvable.

As a short term workaround, you already have all the pieces needed to construct the FQDN if needed. The namespace is the same as the model name which is exposed via a hook env var JUJU_MODEL.
You can configure the pod to set an env var equal to the valueRef spec.nodeName to get the pod name.

[David (davigar15)]

Hello,

Currently Juju creates 2 services:
- <app-name>
- <app-name>-endpoints

The service <app-name> maps to the ClusterIP of the application, and the <app-name>-endpoints is the headless service from which you can access each specific pod.

Use case: I'm building a mongodb-k8s (=<app-name>) charm, which basically sets a replica set, so the mongo_uri exposed by the charm should look like this:

> mongo_uri="mongodb://<ip1>:27017,<ip2>:27017,.../?replicaset=rs0"

The best way to effectively do that, would be if I could have available the FQDNs for each pod, which is formed like this:

<ip1>: <app-name>-0.<app-name>-endpoints --> mongodb-k8s-0.mongodb-k8s-endpoints
<ip2>: <app-name>-1.<app-name>-endpoints --> mongodb-k8s-1.mongodb-k8s-endpoints

This will work within the same model, but having the FQDN will provide the support for CMR within the same Kubernetes cluster.

[Ian Booth (wallyworld)]

Thanks for the extra info - very useful :-)
I'd suggest that it's not best practice to access pods directly, especially outside the namespace
hosting them. The idiomatic k8s way is to do this via services which sit in front of the pods in question, and use selectors to associate themselves with the relevant pod(s).

Juju currently only supports setting up a single service to sit in front of all the application pods as a whole. We'll need to do some feature work to support anything different.

[David (davigar15)]

Hello Ian,

Sorry I haven't been active here for a while. I should mention two things more:

Currently, to get the FQDN of the pods I execute the following function (new operator framework):

```
    def get_workload_fqdn(self):
        try:
            pod_addr = self.model.get_binding(self._relation_name).network.bind_address
            addr = socket.getnameinfo((str(pod_addr), 0), socket.NI_NAMEREQD)[0]
            return addr
        except Exception:
            return
```

The list of FQDNS that I'm getting are the following:

```
{
   "mongodb-k8s/3":"mongodb-k8s-1.mongodb-k8s-endpoints.test.svc.cluster.local",
   "mongodb-k8s/2":"10-1-47-36.mongodb-k8s.test.svc.cluster.local",
   "mongodb-k8s/4":"mongodb-k8s-2.mongodb-k8s-endpoints.test.svc.cluster.local",
   "mongodb-k8s/5":"mongodb-k8s-3.mongodb-k8s-endpoints.test.svc.cluster.local"
}
```

There seems to be a racing condition in the first pod, which is not resolved with headless service name. ("10-1-47-36.mongodb-k8s.test.svc.cluster.local")

Also, the answer for your previous message, about not to be the practice to access the pods directly, I would say that it depends on the usecase.

In mongodb for example, if I set a replicaset, I need to provide the MONGO_URI with the list of units, for what I would need the units hostnames

[John A Meinel (jameinel)]

Going through the documentation from MongoDB for running on K8s, AFAICT they do suggest accessing each node directly:
https://docs.mongodb.com/kubernetes-operator/master/tutorial/deploy-replica-set/

I know the client driver has long supported connecting to multiple, as well as discovering new hosts dynamically (connect to one of them, and discover the rest).

So it does seem like a case where they don't want to be hidden behind a service abstraction. It does mean they are sensitive to pods getting rescheduled, but I guess if you use a StatefulSet then their FQDN will be stable.

It seems like we could expose as 'hostname' the FQDN for stateful sets (I'm not sure we want to do it with a normal deployment.)

[David (davigar15)]

In a normal deployment may be not.

Regarding reaching the replica-sets directly, the problem is that if the IPs change, we need to provide the new IPs through the relations, and the charms that consume MongoDB will experience a rollback. If this can be avoided, it would be awesome

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.