rfe: Each kuryr-daemon maintain a neutron port at host node

[yangjianfeng]

I propose that we create a special subnet for kuryr, this subnet connect to the pod_router, kuryr create a neutron port base on this subnet for each kuryr-daemon and kuryr-daemon maintain a tap device at the host node. By the tap device the host node can forward the traffic to the pods. Base on this, we can deploy kube-proxy and dns addon and use them to provide service implementation and dns server.

[yangjianfeng (yangjianfeng)]

For more details please refer to the below link:
https://paste.opendev.org/show/bcmCY9L2hXU4KKF7ANZR/

[Luis Tomas Bolivar (ltomasbo)]

I miss some information here. I suppose the "why" to do this is to be able to deploy kube-proxy instead of having to depend on octavia, right?

My question here is, how this affects E/W traffic? that traffic should not go out to the host? What changes are needed to do that?

Is this only for N/S traffic? In that case, how are you deciding on the entry point for the traffic? or this is replicated in all the nodes? How the external traffic (Floating IPs) are managed in this case?

On a side note, for the baremetal case that you are targetting, perhaps another option that may fit is the metallb integration

[yangjianfeng (yangjianfeng)]

Yep, Octavia is a heavy service, the amphora backend LB will consume plenty of resources.

The key issue of how to use the raw kube-proxy with kuryr is the connectivity of host node and pod, right?

As the network topology graph [1], I add a neutron port (tap device) to host node, then the host node can be treated as a common leaf device (same level as pod). So, the tap device is necessary, the traffic between host node and pod can be seen as E/W traffic. In addition to, it's important to note that all ip addresses of the host nodes can not overlapping with all pods's.

For N/S traffic, We can leave it entirely to kube-proxy to handle, means the cluster ip allocated by kube-proxy, kuryr needn't to care. Kuryr only process the traffic of pods and host nodes.

For metallb, as far as I understand it just implement loadbalancer service. It can be as a supplement to manage the external traffic.

[1] https://paste.opendev.org/show/bcmCY9L2hXU4KKF7ANZR/

[Luis Tomas Bolivar (ltomasbo)]

Still not clear to me. Currently, devstack is setting up a port in br-int so that traffic can be send from the host to the pod network, which I suppose is something similar to what you are proposing here.

My main concern/doubt here is that I fail to see how this works for E/W traffic (not node to pod, but pod to pod). If we have a pod (let's say subnet 10.0.1.0/24) trying to reach a Cluster IP service (172.30.0.0/16, still neutron private subnet, connected to the router-pod), and then its endpoints (in a third network 10.0.2.0/24), I don't see where kubeproxy applies the iptables rules, as the traffic is not suppose to leave the ovs/ovn overlay

[yangjianfeng (yangjianfeng)]

Well, devstack just only used to deploy development environment. We need a production ready solution. I think I need to commit a devref in order to illustrate this more details and discuss it.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kuryr-kubernetes/+/833961