kuryr-kubernetes

Network situation when combine kuryr-kubernetes and virtual-kubelet

Bug #1891144 reported by Lê Minh Quân
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kuryr-kubernetes
New
Undecided
Unassigned

Bug Description

I'm trying to deploy a model consist of VK(virtual-kubelet) and kubernetes kubelet runing on the same network

I am having a quite funny situation:

I have these network:

```
[root@controller ~(kubernetes)]$ openstack network list
+--------------------------------------+--------------------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+--------------------+--------------------------------------+
| 01b93608-1172-46db-86fb-9f355ce5a04a | services | 1b6dd175-5f90-4766-87e4-9595f6c37b0a |
| 050df0e8-5aec-477a-9365-90a6ed151159 | kuryr-net-f2e53628 | 845ba74f-8882-4567-ad64-00a950d8c3f6 |
| 95e7931a-d774-45b6-afaa-c382ef4a0a40 | pod | 125ac37f-bfaa-4c79-8050-552647bbe9ba |
| a154417c-7770-48bc-a024-6528d3b03aa6 | provider | 0e7ceb5c-2744-454c-88ed-2a9a4563a167 |
| a4322dbc-4ffa-4a38-895d-16a32ee1deb9 | selfservice | 70375905-5132-4060-9f8b-717646a278a0 |
| a47c405a-b7e3-43dd-a8a1-e3eef99c95a7 | LB-Manage-Net | 35ad39c4-4e19-4465-9488-5e8bee993f8c |
+--------------------------------------+--------------------+--------------------------------------+
```

well for short describes:

provider = external
subnet = 10.10.10.0/24 selfservice=internal network 1 for VM
subnet =10.1.0.0/16 pod=internal network for kubernetes pod (normal kubernetes)
subnet= 10.2.0.0/16 service=internal network for kubernetes service (normal kubernetes)

these pod and service network set to dhcp=no (I install kuryr-kubernetes and it work for normal pod)

But when I create VK pod in that network, it running but can't ping to that IP address although I can still ping to normal pod IP address

Any advices ?

here for examples:

```
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-virtual-77cf5845f5-4rfpb 1/1 Running 0 46s 192.168.122.123 virtual-kubelet <none> <none>
nginx-deployment-worker-57d9684bf8-zcr48 1/1 Running 0 16m 10.1.3.140 k8s-worker <none> <none>
```

When I curl these:
it normal:

```
[root@controller ~(kubernetes)]$ curl 192.168.122.123
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@controller ~(kubernetes)]$ curl 10.1.3.140
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
```

but when I use selfservice network to create VK pod:

```
root@k8s-master:~# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-virtual-77cf5845f5-862g8 1/1 Running 0 70s 10.10.10.48 virtual-kubelet <none> <none>
nginx-deployment-worker-57d9684bf8-zcr48 1/1 Running 0 19m 10.1.3.140 k8s-worker <none> <none>

[root@controller ~(kubernetes)]$ openstack capsule list
+--------------------------------------+---------------------------------------------------+---------+-------------+
| uuid | name | status | addresses |
+--------------------------------------+---------------------------------------------------+---------+-------------+
| b79348be-2c10-4e2a-a31c-c11bd1cd9ebd | default-nginx-deployment-virtual-77cf5845f5-862g8 | Running | 10.10.10.48 |
+--------------------------------------+---------------------------------------------------+---------+-------------+

[root@controller ~(kubernetes)]$ curl 10.10.10.48
curl: (56) Recv failure: Connection reset by peer
[root@controller ~(kubernetes)]$ ping 10.10.10.48
PING 10.10.10.48 (10.10.10.48) 56(84) bytes of data.
64 bytes from 10.10.10.48: icmp_seq=1 ttl=63 time=11.8 ms
64 bytes from 10.10.10.48: icmp_seq=2 ttl=63 time=0.898 ms
64 bytes from 10.10.10.48: icmp_seq=3 ttl=63 time=1.19 ms
64 bytes from 10.10.10.48: icmp_seq=4 ttl=63 time=1.17 ms
```

I can only ping it

but when it come to pod network:

```
root@k8s-master:~# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-virtual-77cf5845f5-45stx 1/1 Running 0 53s 10.1.3.255 virtual-kubelet <none> <none>
nginx-deployment-worker-57d9684bf8-zcr48 1/1 Running 0 23m 10.1.3.140 k8s-worker <none> <none>

[root@controller ~(kubernetes)]$ openstack capsule list
+--------------------------------------+---------------------------------------------------+---------+------------+
| uuid | name | status | addresses |
+--------------------------------------+---------------------------------------------------+---------+------------+
| 7bbe5e34-1327-45dc-9a41-7d9620840344 | default-nginx-deployment-virtual-77cf5845f5-45stx | Running | 10.1.3.255 |
+--------------------------------------+---------------------------------------------------+---------+------------+
[root@controller ~(kubernetes)]$ ping 10.1.3.255
PING 10.1.3.255 (10.1.3.255) 56(84) bytes of data.

^C
--- 10.1.3.255 ping statistics ---
44 packets transmitted, 0 received, 100% packet loss, time 44030ms

```

I cant even ping it, but still can curl (worker) pod on the same network:

```
[root@controller ~(kubernetes)]$ curl 10.1.3.140
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
```

Revision history for this message
Luis Tomas Bolivar (ltomasbo) wrote :

This seems related to the multinetwork and OpenStack security groups. Few questions:
- What driver are you using for pods isolation? Network Policies?
- Are all the networks connected to the same neutron router?
- Are the pods in the same namespace? Can you check SGs associated to the pod allow the traffic you need?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.