[feature] add ability to have multiple ingress controllers with separate ingress classes

[Jeff Hillman]

There is a customer need to have multiple ingress controllers when trying to guarantee traffic separation with multiple interfaces/ingress IPs.

Currently, the method to do this is to set ingress=false on the worker charm, and hand install the nginx-ingress service.

Doing the hand install also allows the ability to edit the manifest to change the IngressClass.

Having unique ingress classes allows an ingress definition for a deployment to select the controller that it wants to use, and thus have more granular control over ingress and how it is handled.

[Jeff Hillman (jhillman)]

Subscribed Field Medium

[Chris Sanders (chris.sanders)]

I'm unsubscribing field-medium as this doesn't meet the definition of a contribution that needs to be merged. Multiple ingress isn't intended to be covered by the current add-ons, this probably needs to be addressed with an ingress operator.