ISO build: staging jobs failure due to: docker allocates port already in use

[Vitaly Gusev]

http://jenkins-product.srt.mirantis.net:8080/view/Staging%20Mirrors/job/6.1.staging.iso/229
http://jenkins-product.srt.mirantis.net:8080/view/Staging%20Mirrors/job/6.1.staging.iso/233

http://jenkins-product.srt.mirantis.net:8080/view/Staging%20Mirrors/job/6.1.test_staging_mirror/224/
http://jenkins-product.srt.mirantis.net:8080/view/Staging%20Mirrors/job/6.1.test_staging_mirror/228/

error reading information on service tuned: No such file or directory
error reading information on service avahi-daemon: No such file or directory
error reading information on service iscsi: No such file or directory
error reading information on service iscsid: No such file or directory
error reading information on service firstboot: No such file or directory
error reading information on service kdump: No such file or directory

Preparing bootloader config
error: package ruby is not installed

sudo docker rm -f "fuel-repo-container"
Error response from daemon: No such container: fuel-repo-container
2015/04/05 19:01:01 Error: failed to remove one or more containers
make: [/home/jenkins/workspace/tmp/6.1.staging.iso/build/docker/repo-container-up.done] Error 1 (ignored)
sudo docker -D run -d -p 80 -v /home/jenkins/workspace/tmp/6.1.staging.iso/local_mirror/centos:/var/www/html --name "fuel-repo-container" fuel/centos /usr/sbin/apachectl -DFOREGROUND
027f837698a8a7410c75e16bda22fe244caca7bbaa147ae96058b60f8a0ed662
[debug] commands.go:2229 End of CmdRun(), Waiting for hijack to finish.
2015/04/05 19:01:01 Error response from daemon: Cannot start container 027f837698a8a7410c75e16bda22fe244caca7bbaa147ae96058b60f8a0ed662: Error starting userland proxy: listen tcp 0.0.0.0:49162: bind: address already in use
make: *** [/home/jenkins/workspace/tmp/6.1.staging.iso/build/docker/repo-container-up.done] Error 1
Build step 'Execute shell' marked build as failure

[Matthew Mosesohn (raytrac3r)]

upstream commit fixes this: https://github.com/docker/docker/pull/9441
upstream bug: https://github.com/docker/docker/issues/8714
We should upgrade to later version of docker on build hosts, preferrably 1.5.0, but 1.4.1 will have this fix as wlel.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/170807

[Matthew Mosesohn (raytrac3r)]

Waiting on reviews. This solution seems to work just fine.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/170807
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=3bc810418c9b28246a0f0d8e9f9b29200c9fabf1
Submitter: Jenkins
Branch: master

commit 3bc810418c9b28246a0f0d8e9f9b29200c9fabf1
Author: Matthew Mosesohn <email address hidden>
Date: Mon Apr 6 13:27:37 2015 +0300

    Update lxc-docker to 1.5.0 for iso build

    lxc-docker-1.3.2 has upstream issue
    https://github.com/docker/docker/issues/8714
    which causes failures in allocating ports
    dynamically. Updating resolves this.

    Change-Id: I7fd22591a330ecb4ab0741af9f58fdd4dee54b18
    Closes-Bug: #1440663

[Vasyl Saienko (vsaienko)]

reproduced on http://jenkins-product.srt.mirantis.net:8080/job/7.0.staging.iso/150

[Vasyl Saienko (vsaienko)]

https://paste.mirantis.net/show/879/

[Sergey Kulanov (skulanov)]

Propose to set to non-critical but high, this is the first case for the last 4 months, checking...

[Bartłomiej Piotrowski (bpiotrowski)]

Apparently the problem hasn't been completely fixed upstream, looking at: https://github.com/docker/docker/issues/9047

I don't see what can we do about it. All further staging builds finished or failed for different reasons. One option is to retry the command that failed with port taken, but we won't probably see this failure anymore in 4 months.

[Bartłomiej Piotrowski (bpiotrowski)]

Lowering to medium as it's hard to reproduce.

[Matthew Mosesohn (raytrac3r)]

This hasn't reproduced in 8.0 yet. Moving to incomplete.

[Roman Prykhodchenko (romcheg)]

Marking as Invalid since it was Incomplete for longer than 3 weeks.