Charms in the Operator Framework get a default Kubernetes service port of 65535

[Connor Chamberlain]

Juju 2.9.7 fixed LP1922133 [0], which gave juju applications access to their cluster ip, rather than a set of individual unit addresses. When charms relate using cluster ip, no daemons can be found listening at <cluster-ip>:<port>, despite the daemons being fully functional.

This doesn't occur when relations provide unit addresses, as daemons can be found at <unit-ip>:<port>, as expected.

When looking at the kubernetes service descriptions, it appears that charms in the Operator Framework are assigned a port of 65535, which cannot be changed from within the charm.

[0] https://bugs.launchpad.net/juju/+bug/1922133

Edit: rewrote to encapsulate new information on this bug

[John A Meinel (jameinel)]

We need a little bit more context to understand what charms are being broken and what they are defining as "a vip".
Is this the Service definition for the overall application?
Is the bug that the daemon *should* be available at the "vip" but isn't being seen, or is it that the units want to be advertising the individual unit IPs rather than an overall application IP?

[Connor Chamberlain (lcvcode)]

By "vip" I mean the IP that's assigned to the overall application. I think they did not exist prior to Juju version 2.9.7.

The bug here is the first problem you described; that charms should be able to find daemons listening at other applications' IPs.

I have confirmed that the daemon on each unit is fully functional and can be found at that unit's address. But none of them can be found by way of the application IP.

For clarity about my deployments, I have two development charms that get related to each other. The first charm populates relation data with its IP, and the second charm will use that IP to attempt a connection. In the case of Juju versions lower than 2.9.7, the first charm provides a list of individual unit IPs. On later versions, it will provide the application IP. The second charm attempts the connection. When using the application IP, the second charm looks for a listening daemon at <application-ip>:<port> and finds nothing. When using a list of units' addresses, the second charm looks for listening daemons at <unit-0-ip>:<port>, <unit-1-ip>:<port>, or <unit-2-ip>:<port> and succeeds.

[Connor Chamberlain (lcvcode)]

It looks like Kubernetes may be setting service target ports to 65535 by default. This breaks networking when two Charmed Operator charms attempt to communicate using cluster IPs.

In my case, I'm deploying:
* Pulsar https://code.launchpad.net/~lcvcode/+git/pulsar
* Bookie https://code.launchpad.net/~lcvcode/+git/bookie

After deploying Pulsar, I can see:
```
kubectl describe service -n pulsar pulsar
Name: pulsar
Namespace: pulsar
...
IP: 10.152.183.12
IPs: 10.152.183.12
Port: placeholder 65535/TCP
TargetPort: 65535/TCP
Endpoints: 10.1.195.244:65535,10.1.195.245:65535,10.1.195.247:65535
...
```
TargetPort should be configurable from within a charm, otherwise other applications cannot communicate using cluster IP.

A workaround for this is to use each unit's IP address (ops.model.network.bind_address [0]) rather than the application's ingress address (ops.model.network.ingress_address [1]) when providing addresses in relation data. In this way, units can communicate directly using the configured ports.

[0] - https://operator-framework.readthedocs.io/en/latest/index.html?highlight=ops.model#ops.model.Network.bind_address
[1] - https://operator-framework.readthedocs.io/en/latest/index.html?highlight=ops.model#ops.model.Network.ingress_address