Problem with iso build on new server (ci-slave24) inf fuel-ci

[Artur Kaszuba]

We have a problem with iso build on server ci-slave24.fuel-infra.org, is it fresh server redeployed with build role. It ends with error:

Removing intermediate container 158ad1440cd0
Step 2 : RUN rm -rf /etc/yum.repos.d/*; echo -e "[nailgun]\nname=Nailgun Local Repo\nbaseurl=http://$(route -n | awk '/^0.0.0.0/ { print $2 }'):32769/os/x86_64/\ngpgcheck=0" > /etc/yum.repos.d/nailgun.repo; yum clean expire-cache ; yum update -y; yum --quiet install -y ruby21-nailgun-mcagents sysstat
 ---> Running in 15755450a1ad
Loaded plugins: fastestmirror, priorities
Cleaning repos: nailgun
0 metadata files removed
Loaded plugins: fastestmirror, priorities
Determining fastest mirrors
http://172.17.42.1:32769/os/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://172.17.42.1:32769/os/x86_64/repodata/repomd.xml: (28, 'connect() timed out!')
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: nailgun. Please verify its path and try again
http://172.17.42.1:32769/os/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://172.17.42.1:32769/os/x86_64/repodata/repomd.xml: (28, 'connect() timed out!')

There are also errors:
Error response from daemon: no such id: fuel-repo-container
time="2015-12-18T09:55:58Z" level=fatal msg="Error: failed to remove one or more containers"
make: [/home/jenkins/workspace/tmp/6.1-community.all/build/docker/repo-container-up.done] Error 1 (ignored)
sudo docker -D run -d -p 80 -v /home/jenkins/workspace/tmp/6.1-community.all/local_mirror/centos:/var/www/html --name "fuel-repo-container" fuel/centos /usr/sbin/apachectl -DFOREGROUND
ea636a881245183bb87c53b861ed06e8033ed4f5039093d696fdea430542c18c

Log: https://ci.fuel-infra.org/view/ISO/job/6.1-community.all/546/consoleFull

It looks like problem with local mirror or with docker container, but without access to server i cannot check it.

[Aleksandra Fedorova (bookwar)]

Build team and CI team has no access to the server. Devops team, please provide access under Jenkins user as to other external CI slaves

[Igor Shishkin (teran)]

Marking as medium according to our new policy.

[Igor Shishkin (teran)]

@Alexandra, it's not a CI slave but build slave with particular purpose.

[Igor Shishkin (teran)]

The issue is about firewall rules:

......
-A INPUT -m comment --comment "9999 - drop all" -j DROP
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o docker0 -m comment --comment "1000 - docker rule" -j DOCKER
-A FORWARD -o docker0 -m comment --comment "1001 - docker rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 -o docker0 -m comment --comment "1002 - docker rule" -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -m comment --comment "1003 - docker rule" -j ACCEPT
-A DOCKER -d 172.17.0.11/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
....

[Ihor Kalnytskyi (ikalnytskyi)]

Log for master:

https://ci.fuel-infra.org/view/ISO/job/8.0-community.all/176/console

[Mateusz Matuszkowiak (mmatuszkowiak)]

@Igor K., that build number 176 - went on different (build1 - not a new one) server.

[EnvInject] - Injecting contributions.
Building remotely on build1.fuel-infra.org (build) in workspace /home/jenkins/workspace/8.0-community.all
Wiping out workspace first.
Cloning the remote Git repository

Non related to this bug,

[Artur Kaszuba (akaszuba)]

On ci-slave24 the problem was caused by firewall, after change new build works:
https://ci.fuel-infra.org/view/ISO/job/6.1-community.all/551/console

It started to work after this change in iptables, this rule exists on old build1:
sudo iptables -I INPUT -i docker0 -j ACCEPT

Please update hiera or manifests to fix this problem.

[Mateusz Matuszkowiak (mmatuszkowiak)]

The required FW rule added to the hieras role.