Cross-node traffic does not work with Cilium CNI
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Cilium Charm |
Fix Released
|
Medium
|
Mateo Florido | ||
Bug Description
TL;DR
When Charmed K8s is deployed with Cilium, there are some problems with the traffic flow between the different node. In the snippet below, there are three pods, two of them residing on the same node; so communication is possible between those two - but if one'd try to reach out to another (3rd one, residing on ANOTHER node), then there's no traffic flow:
######## Get all pods
ubuntu@
E0418 15:34:49.240814 176349 memcache.go:287] couldn't get resource list for metrics.
E0418 15:34:49.244867 176349 memcache.go:121] couldn't get resource list for metrics.
E0418 15:34:49.249119 176349 memcache.go:121] couldn't get resource list for metrics.
E0418 15:34:49.252581 176349 memcache.go:121] couldn't get resource list for metrics.
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-748c667d9
nginx-748c667d9
nginx-748c667d9
ubuntu@
E0418 15:34:52.247098 177214 memcache.go:287] couldn't get resource list for metrics.
E0418 15:34:52.256970 177214 memcache.go:121] couldn't get resource list for metrics.
E0418 15:34:52.262418 177214 memcache.go:121] couldn't get resource list for metrics.
######## Try to curl self
root@nginx-
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://
Commercial support is available at
<a href="http://
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
######## curl-ing ANOTHER pod on the SAME node
root@nginx-
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://
Commercial support is available at
<a href="http://
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@nginx-
####### curl-ing ANOTHER pod on ANOTHER node
root@nginx-
^C
########## additional info
juju status: https:/
bundle: http://
env: charmed k8s on top of AWS EC2 hosts; the same behaviour was also observed once bundle was deployed on top of OpenStack.
ubuntu@
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Hubble Relay: OK
\__/¯¯\__/ ClusterMesh: disabled
\__/
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
Containers: hubble-relay Running: 1
Cluster Pods: 10/10 managed by Cilium
Image versions cilium rocks.canonical
| Changed in charm-cilium: | |
| assignee: | nobody → Mateo Florido (mateoflorido) |
| Mateo Florido (mateoflorido) wrote | #1 |
| Changed in charm-cilium: | |
| importance: | Undecided → Medium |
| status: | New → Triaged |
| tags: | added: cdo-qa |
| Changed in charm-cilium: | |
| milestone: | none → 1.28 |
| Changed in charm-cilium: | |
| milestone: | 1.28 → 1.28+ck1 |
| Changed in charm-cilium: | |
| milestone: | 1.28+ck1 → 1.29 |
| Kevin W Monroe (kwmonroe) wrote | #2 |
| Changed in charm-cilium: | |
| status: | Triaged → Fix Committed |
| Changed in charm-cilium: | |
| status: | Fix Committed → Fix Released |
This issue is related to the bootstrap process of a new model with Juju on both AWS and O7k, although vSphere is currently unaffected. By default, Juju adds a new model with fan networking on AWS and O7k, the interfaces created for this purpose conflict with the Cilium VXLAN interface, resulting in the interface failing to start. To resolve this issue, modify the container-
Here are the steps to overcome this issue. Please note that these should be performed before creating the cluster.
1. Add a new model.
2. Set the model defaults for container-
3. Deploy CK+Cilium.
Furthermore, I noticed that the bundle uses CK 1.26. Please switch to the 1.27/edge channel, as this is the K8s version we used to test Cilium. It's also important to note that AWS has switched to the out-of-tree provider, so please refer to these two overlays to deploy it:
[AWS Overlay]
https:/
[AWS Storage]
https:/