fuel-library tests on ci-slave28 fail

[Aleksandra Fedorova]

https://ci.fuel-infra.org/job/7.0.fuel-library.pkgs.ubuntu.neutron_vlan_ha/buildTimeTrend

https://ci.fuel-infra.org/job/7.0.fuel-library.pkgs.ubuntu.neutron_vlan_ha/55/consoleFull

https://ci.fuel-infra.org/job/7.0.fuel-library.pkgs.ubuntu.neutron_vlan_ha/50/

https://ci.fuel-infra.org/job/7.0.fuel-library.pkgs.ubuntu.neutron_vlan_ha/47/

Please investigate.

Critical as it affects CI for 7.0

[Mateusz Matuszkowiak (mmatuszkowiak)]

builds failed with:
"Couldn't resolve host 'mirror.fuel-infra.org'"

but after logging to the srv:

root@ci-slave28:~# host mirror.fuel-infra.org
mirror.fuel-infra.org is an alias for seed.fuel-infra.org.
seed.fuel-infra.org has address 5.43.231.47
seed.fuel-infra.org has address 208.78.244.194
root@ci-slave28:~#

[Igor Shishkin (teran)]

Could be related to dhclient configuration refresh:

Sep 16 12:33:47 ci-slave28 dhclient: DHCPREQUEST of 5.43.231.40 on eth0 to 5.43.231.254 port 67 (xid=0x53258b20)
Sep 16 12:33:47 ci-slave28 dhclient: DHCPACK of 5.43.231.40 from 5.43.231.254
Sep 16 12:33:47 ci-slave28 dhclient: bound to 5.43.231.40 -- renewal in 2546 seconds.

[Igor Shishkin (teran)]

Let's set static configuration first.

[Igor Shishkin (teran)]

Decreasing to high since there are no blocks for teams or mass effect.

[Mateusz Matuszkowiak (mmatuszkowiak)]

Andrey have set static IP configuration on the mentioned hosts. Server is rebooted and working with static config. Setting to fix committed.

[Aleksandra Fedorova (bookwar)]

Happened again

https://ci.fuel-infra.org/job/7.0.fuel-library.pkgs.ubuntu.neutron_vlan_ha/62/

https://ci.fuel-infra.org/job/7.0.fuellib_review_pkgs_master_node/60/console

Please provide details regarding these issues.

7.0 CI is affected by this -> Critical

[Aleksandra Fedorova (bookwar)]

https://ci.fuel-infra.org/job/6.1.fuel-library.pkgs.master_node/207/

https://ci.fuel-infra.org/job/6.1.fuel-library.pkgs.ubuntu.ha_neutron_vlan/210/console

[Sergey Kulanov (skulanov)]

Running puppet manifest flushes firewall table rules for docker:

For now we have a single rule:
https://github.com/fuel-infra/puppet-manifests/blob/master/modules/fuel_project/manifests/jenkins/slave.pp#L835-L843

which is wrong.

At least the rules should be like here:
https://github.com/hesco/hesco-weave/blob/master/manifests/firewall/docker.pp
https://github.com/hesco/hesco-weave/blob/master/manifests/firewall/dnat_published_port.pp

As a simple fix just restart docker after running manifests

As a long live solution update firewall rules, but IMO better to restart docker from manifests after running firewall rules

[Andrey Nikitin (heos)]

Importance is decreased to High, because problem is not block, we are working on investigating of root cause.

[Aleksandra Fedorova (bookwar)]

Happened again https://ci.fuel-infra.org/job/fuellib_review_pkgs_master_node/4428/

[Andrey Nikitin (heos)]

The problem is reproduces on ci-slave21 - it fixed by restarting of the docker service on the server.

[Aleksandra Fedorova (bookwar)]

Restarting is not a fix.

Please describe how this problem appear? How to prevent it?

Does it happen only after reboot? - Then we probably need to fix startup sequence, maybe add additional docker restart in rc.local at least.

Just occasionally? Then how can we monitor it?

Everytime we apply puppet? Then let's fix puppet flow to restart docker at the end of each run.

[Andrey Nikitin (heos)]

Firewall rules are stored by using '/etc/init.d/iptables-persistent save' command.

It should be executed on all other ci-slaves, to fix the problem with Docker's firewall rules.

[Igor Shishkin (teran)]

@Andrey, what the status ?

[Andrey Nikitin (heos)]

@Igor, work in progress.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Andrey Nikitin <email address hidden>
Review: https://review.fuel-infra.org/12700

[Andrey Nikitin (heos)]

All firewall rules added into 'sandbox_slave' role on testing puppet master. Version of puppetlabs-firewall module is updated too on the testing.

[Andrey Nikitin (heos)]

Igor Belikov, please test the jobs on the Jenkins sandbox.

[Andrey Nikitin (heos)]

Root cause of the problem is missed firewall rules for docker. The following Docker-related rules should be in firewall:

*nat
:DOCKER - [0:0]
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

*filter
:DOCKER - [0:0]
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT

[Andrey Nikitin (heos)]

I've tested solution on Jenkins sandbox, it works:
https://jenkins-sandbox.infra.mirantis.net/job/7.0.fuel-library.pkgs.ubuntu.neutron_vlan_ha/1/console

But I've faced with another porblem with time synchronization by ntp on fuel master node. I think, that root cause of the problem - not the firewall rules, because that problem reproduced when all firewall rules are removed.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/12700
Submitter: Alexander Charykov <email address hidden>
Branch: master

Commit: 9fa698a70310f879a22121da6ccf6c92771e095e
Author: Andrey Nikitin <email address hidden>
Date: Thu Oct 15 16:28:41 2015

Updated version of puppetlabs-firewall module

Version of 'puppetlabs-firewall' module updated from 1.1.3 to 1.7.1.
It needs to have an ability to create a firewall rule with
inversion.

To fix the bug we should use 1.7.1 version of the module because 1.1.3 version
couldn't implement a firewall rule with inversion. For example, the rule:
-A FORWARD -i docker0 ! -o docker0 -m comment --comment "One more docker rule" -j ACCEPT.

Old version of the module is used non-correct pattern to parse such rules,
that's why old version is doesn't works.

Change-Id: Id50b92e8c9a998fad439bf22db624155146e6c15
Related-bug: #1496399

[Andrey Nikitin (heos)]

Solution of the bug was merged into production hiera in 'fuelci_slave' role.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Pawel Brzozowski <email address hidden>
Review: https://review.fuel-infra.org/13607

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13607
Submitter: Pawel Brzozowski <email address hidden>
Branch: master

Commit: 188612d5705d4acbe8f36b58d542829f1d8f18b6
Author: Pawel Brzozowski <email address hidden>
Date: Tue Nov 3 09:02:05 2015

iptables-persistent package

- related to https://trello.com/c/FkiXoEkw/1013-docker-vs-iptables-persistent-on-packtests

Change-Id: I46966934ff451fa868105bfb3690e09c979ead69
Partial-Bug: #1496399

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13610
Submitter: Alexander Charykov <email address hidden>
Branch: master

Commit: 0ee84e39fc30af13e066d7d35bffe6d24a2bae6f
Author: Pawel Brzozowski <email address hidden>
Date: Tue Nov 10 08:55:16 2015

iptables-persistent with upstart support

- added docker as prestarting dependency

Closes-Bug: #1496399
Change-Id: I00c1ad5d7be3e3efd5b9eb2c8f95a0ed41d171fc