time for zun to create container and reach Running state is 19 times as long as directly using docker
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Zun |
Fix Released
|
High
|
hongbin | ||
| kuryr-libnetwork |
Fix Released
|
Undecided
|
hongbin | ||
Bug Description
My queens installation of zun seems to take a long time to create a container and get to a running state. I ran two tests. One test is the python script at the end of this email.
I ran that three times and below are the number of seconds that it took:
17.9 seconds
15.1 seconds
17.2 seconds
The average is 16.7 seconds.
The second test used docker directly without using zun. For that test I ran:
time bash -c 'docker run --detach cirros sleep 999d; docker ps -a --format "{{.Image}} {{.Command}} {{.Status}}"'
The output included this line which shows that it was running:
cirros "sleep 999d" Up Less than a second
I ran that on my zun compute node which means my zun test and docker-alone test used the same virtual machine which means they both ran on the same hardware in order to make it a fair comparison.
I ran that three times and below are the number of seconds that it took:
0.85 seconds
0.92 seconds
0.88 seconds
The average is .88 seconds.
So zun takes 16.7 / .88 = 19 times as long as docker alone to create and reach running state.
Is my test a fair comparison? Perhaps not since zun does more than docker-alone. For example, zun needs time to decide which compute node to run a container on. It also might need time for other things like maybe transferring a docker image to the compute node. I'm willing to wait a reasonable amount of time for those important tasks, but I am still surprised that they would take an extra 15.8 seconds.
Although this test creates only one container, my end goal for using zun is to automatically create hundreds of containers which means this amount of overhead is a burden. Therefore, I would appreciate if you can tell me if there is anything I can do to make it faster or if there is something the zun developers can do to improve the performance of zun.
Thanks.
Below are the logs from one of my runs of zun which I got from this command:
journalctl -u docker -u kuryr-libnetwork -u zun-compute -n 20
The lines which occurred after the test start time are:
Dec 20 11:46:44 pdv-packstack-
Dec 20 11:46:45 pdv-packstack-
Dec 20 11:46:52 pdv-packstack-
Dec 20 11:46:53 pdv-packstack-
Dec 20 11:46:56 pdv-packstack-
Dec 20 11:46:57 pdv-packstack-
Dec 20 11:46:57 pdv-packstack-
Dec 20 11:46:58 pdv-packstack-
Dec 20 11:46:58 pdv-packstack-
Dec 20 11:46:58 pdv-packstack-
Dec 20 11:46:58 pdv-packstack-
Dec 20 11:46:58 pdv-packstack-
Below is my script to time zun:
#!/usr/bin/env python
from keystoneauth1 import loading
from keystoneauth1 import session
from zunclient import client
import time
AUTH_URL='http://
USERNAME='internal'
PASSWORD='foo'
VERSION='1.12' # master -> 1.latest, stable/rocky -> 1.26, stable/queens -> 1.12
zun = client.
start_time = time.time()
print 'Requested creation of', \
zun.
while zun.containers.
pass
print 'seconds to start running =', time.time() - start_time
| hongbin (hongbin034) wrote | #1 |
| Changed in zun: | |
| assignee: | nobody → hongbin (hongbin034) |
| Jacob Burckhardt (jburckhardt) wrote | #2 |
| hongbin (hongbin034) wrote | #3 |
| Changed in zun: | |
| status: | New → Confirmed |
| importance: | Undecided → High |
| Changed in kuryr-libnetwork: | |
| assignee: | nobody → hongbin (hongbin034) |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to kuryr-libnetwork (master) | #4 |
| Changed in kuryr-libnetwork: | |
| status: | New → In Progress |
| OpenStack Infra (hudson-openstack) wrote | #5 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to kuryr-libnetwork (master) | #6 |
| OpenStack Infra (hudson-openstack) wrote | #7 |
| OpenStack Infra (hudson-openstack) wrote | #8 |
| OpenStack Infra (hudson-openstack) wrote | #9 |
| Changed in kuryr-libnetwork: | |
| status: | In Progress → Fix Released |
| Changed in zun: | |
| status: | Confirmed → In Progress |
| OpenStack Infra (hudson-openstack) wrote Fix merged to zun (master) | #10 |
| OpenStack Infra (hudson-openstack) wrote Fix included in openstack/kuryr-libnetwork 3.0.0 | #11 |
| Changed in zun: | |
| status: | In Progress → Fix Released |
Short answer:
In Rocky, we landed an optimization for Kuryr-libnetwork: https:/
Long answer:
Zun relies on Kuryr-libnetwork to provide the network connectivity. Kuryr-libnetwork is a Docker network plugin which is responsible for:
* Make API calls to Neutron to provision networking resources (i.e. security groups, ports).
* Do the port binding for the container. Generally speaking, it essentially creates a veth pair, plug one end to the Neutron-managed virtual switch and push the other end into the container.
If you are directly calling Docker API to create the containers, above steps are skipped so it is much faster.
We are trying to optimize Kuryr-libnetwork to cut the unnecessary API calls to Neutron. By looking forward, we are looking for an optimization in Neutron side: https:/
At the meanwhile, I will look into other possible ways to optimize the performance further. This is the area we are working hard to improve.