[Master/Worker/CMR - RFE] Calico should be clustered across multiple models using cross-subnet option

[Pedro Guimarães]

As per bug LP #1864467, to achieve a full k8s master/worker CMR, Calico charm must be adapted to allow different Calico applications on different models to cluster together.

Calico offers the option to run cross-subnet IPIP overlay [1]. A CMR between two different calico applications should end up with one single overlay network, where cross-subnet IPIP is enforced (at least).

The CMR between Calico applications should enable master nodes to reach POD and service networks on kubernetes workers.

Right now, the only option I see to do this integration and allow masters to reach for POD and service networks is to exploit BGP options on calico charm and connect both applications using BGP.

[1]https://docs.projectcalico.org/v3.5/usage/configuration/ip-in-ip

[George Kraft (cynerva)]

The IPIP CrossSubnet mode is already supported: https://github.com/charmed-kubernetes/layer-calico/blob/e114b47f5cb6a3e06ef70be590c01f11b0b8df74/config.yaml#L17

But that won't help if your Calico units are related to two different etcd clusters. If they can't access etcd, then they can't share routing information with eachother. They won't know how to forward traffic to eachother, IPIP or otherwise. At that point your only option is manual configuration of BGP, like you said.

Is there anything else you're proposing to change in the Calico charm? If not, I suggest we close this and continue discussion in https://bugs.launchpad.net/charm-etcd/+bug/1864470.