Build images freezes on infinite time

[Egor Kotko]

Steps to reproduce:
1) Download ccpinstaller and microservices
2) Install microservices:
#cd ~/fuel-ccp && pip install .
3) Create local registry:
#kubectl create -f ~/fuel-ccp-installer/registry/registry-pod.yaml
#kubectl create -f ~/fuel-ccp-installer/registry/service-registry.yaml
4) Build images and push its into local registry:
#mcp-microservices --images-base-distro debian --images-base-tag 8.4 --images-maintainer <email address hidden> --repositories-protocol https --repositories-port 443 --builder-push --registry-address 127.0.0.1:31500 --logfile /tmp/microservices_log --verbose --debug --registry-insecure --images-tag latest build

Expected result:
Built images

Actual result:
The command freezes on infinite state (near 14 hours).
The last log output is:
2016-07-20 18:26:28.261 3623 INFO microservices.build [-] glance-base: Running setup.py install for simplegeneric: finished with status 'done' (See the attached LOG file)

The list of images:
http://paste.openstack.org/show/538939/

The host state:
http://paste.openstack.org/show/538943/

[Egor Kotko (ykotko)]

log_strase.tar.gz - contains logs with strace.
As workaround was found parameter "--builder-workers 1"
for mcp-microservices ...

[Proskurin Kirill (kproskurin)]

How many cpu this environment has?

[Aleksander Mogylchenko (amogylchenko)]

multiprocessing.cpu_count() shows 2

[Sergey Reshetnyak (sreshetniak)]

Not reproduced on master with 2 workers.

[Egor Kotko (ykotko)]

Do you need the environment with reproduced bug?

[Dennis Dmitriev (ddmitriev)]

Confirmed.

1. Deploy k8s using kargo_deploy.sh (kubernetes v1.3.0)

2. Upload k8s_topology.yaml to the /home/vagrant from https://review.openstack.org/#/c/342175/22/mcp_tests/templates/k8s_templates/k8s_topology.yaml

3. Run the command (on node1):
ccp --images-base-distro debian --images-maintainer <email address hidden> --repositories-protocol https --repositories-port 443 --builder-push --registry-address 127.0.0.1:31500 --logfile /var/log/microservices.log --verbose --debug --builder-workers 1 --registry-insecure --images-tag test_latest --deploy-config ~/k8s_topology.yaml --images-namespace mcp build

Result:
1. Registry was started on node4:
root@node1:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
registry 1/1 Running 1 1h 10.233.67.130 node4

, port 31500 is available on localhost (with hypercube proxy):
root@node1:~# telnet 127.0.0.1 31500
Trying 127.0.0.1...
Connected to 127.0.0.1.

2. The command "ccp --images-base-distro ..." stucks on waiting for something (more than 2 hours at the moment):
root 29000 0.3 1.1 284836 72588 ? Sl 11:38 0:18 /usr/bin/python /usr/local/bin/ccp --images-base-distro debian ...

root@node1:~# strace -p 29000
strace: Process 29000 attached
select(0, NULL, NULL, NULL, {0, 9968}) = 0 (Timeout)
gettimeofday({1471439555, 96633}, NULL) = 0
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
gettimeofday({1471439555, 146955}, NULL) = 0
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
gettimeofday({1471439555, 197301}, NULL) = 0
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
gettimeofday({1471439555, 247599}, NULL) = 0
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
gettimeofday({1471439555, 297885}, NULL) = 0
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
...

[Sergey Reshetnyak (sreshetniak)]

Not reproduced last 3 month on CI and dev envs