Juju should report open-port failures more visibly (than just controller logs)

[Ben Hoyt]

I was playing with open-port on a K8s sidecar charm, and apparently K8s doesn't support the ICMP protocol (VM charms do). However, open-port didn't return a failure, and even later there was no visible sign that something was wrong, except in the controller logs:

$ juju debug-log -m controller
...
controller-0: 15:00:50 ERROR juju.worker.dependency "caas-firewaller-embedded" manifold worker returned unexpected error: cannot update service port for application "database": protocol "icmp" for service "icmp" not valid
...

So I think there are two things we could do here to improve this:

1) Have the open-port / close-port hook tools reject protocol "icmp" early if it's a K8s sidecar charm.
2) In general, have a way for Juju to report firewaller worker errors more visibly. For example, setting a short error message in application status?

As it was, it took me a while to figure out what was going on, because "kubectl get svc" didn't show any of the ports opened (because K8s rejected the ICMP port, it rejected all the ports I asked for, so no changes were made -- that's probably good though that it does it atomically).

[Harry Pidcock (hpidcock)]

Thanks for the report. I think it makes sense for the hook tool to error, as a means to create robust charms.

[Caner Derici (cderici)]

Fix is up for 3.4+ https://github.com/juju/juju/pull/17597

Just for future reference; here's what's happening, the charm calls the open_port(), ops pick it up, calls the jujuc, uniter picks it up, calls the client, which calls the apiserver, then the open port change is committed to the state. After that somewhere sometime the firewaller worker gets the trigger from mongo and starts trying to realize that, and ultimately freaks out.

This fix intercepts this at the apiserver before we commit the change to the state, so 1) there's nothing any downstream workers should be confused about, 2) the error can be communicated back to the model nicely, 3) that business logic should be on the apiserver anyways.

For the 2nd suggestion, because of the distributed nature of Juju, it is very difficult to communicate error messages from a worker back to the source of a workflow. If a worker fails, while the dependency engine can bounce the worker and log the error on the controller, it doesn't have a logical link to follow back, i.e. the concerns of workers don't (and shouldn't) one-to-one match with any workflow that Juju expose. Moreover, using status messages is not a good option either because I believe it would lead to a massively tight coupling between workers and juju entities, as a worker can fail for multitude of reasons related to different entities involved in a workflow.