"OSTF server is not available" after master node reboot.

[Dennis Dmitriev]

Jenkins job: http://jenkins-product.srt.mirantis.net:8080/view/6.0_swarm/job/6.0_fuelmain.system_test.ubuntu.thread_5/35/ ,
system test: 'ha_haproxy_termination'

After rebooting the master node, docker container 'ostf' was started, but DNAT rules wasn't added to iptables for accessing the container.

$ iptables -t nat -S # http://paste.openstack.org/show/137628/

{"build_id": "2014-11-22_22-01-00", "ostf_sha": "a35f516f1606b0d03d51ff63bfe3fbe23de4b622", "build_number": "129", "auth_required": true, "api": "1.0", "nailgun_sha": "7196c478bfe0b5cff97077c1829009b9dbc3ee92", "production": "docker", "fuelmain_sha": "0dd338b6280843b4c45b46635528a6c43939c772", "astute_sha": "c15623d05ccdf7ac10873e7a90df954de8726280", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-22_22-01-00", "ostf_sha": "a35f516f1606b0d03d51ff63bfe3fbe23de4b622", "build_number": "129", "api": "1.0", "nailgun_sha": "7196c478bfe0b5cff97077c1829009b9dbc3ee92", "production": "docker", "fuelmain_sha": "0dd338b6280843b4c45b46635528a6c43939c772", "astute_sha": "c15623d05ccdf7ac10873e7a90df954de8726280", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "2bfa9431a4839efae9c75d6b133df24c0f11c868"}}}, "fuellib_sha": "2bfa9431a4839efae9c75d6b133df24c0f11c868"}

[Łukasz Oleś (loles)]

Doesn't happen always. I didn't reproduce it

[Dennis Dmitriev (ddmitriev)]

Reproduced on http://jenkins-product.srt.mirantis.net:8080/view/5.1_swarm/job/5.1_fuelmain.system_test.ubuntu.upgrade_rollback/67/

./iptables-bad.log : http://paste.openstack.org/show/143985/
./iptables-good.log: http://paste.openstack.org/show/143986/

======= diff between `iptables -S` taken before and after restart containers 'ostf' and 'nginx':
[root@nailgun ~]# diff ./iptables-bad.log ./iptables-good.log
30a31,34
> -A FORWARD -d 172.17.0.5/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8080 -j ACCEPT
> -A FORWARD -d 172.17.0.5/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8000 -j ACCEPT
> -A FORWARD -d 172.17.0.7/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8777 -j ACCEPT
> -A FORWARD -d 172.17.0.7/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8777 -j ACCEPT
46d49
< -A FORWARD -d 172.17.0.7/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8777 -j ACCEPT
63a67,68
> -A FORWARD -i eth0 -o docker0 -p tcp -m state --state NEW -m tcp --dport 8000 -m comment --comment "nginx-tcp-8000-accept" -j ACCEPT
> -A FORWARD -i eth0 -o docker0 -p tcp -m state --state NEW -m tcp --dport 8080 -m comment --comment "nginx-tcp-8080-accept" -j ACCEPT

./iptables-nat-bad.log : http://paste.openstack.org/show/143987/
./iptables-nat-good.log : http://paste.openstack.org/show/143988/

======= diff between `iptables -t nat -S` taken before and after restart containers 'ostf' and 'nginx':
[root@nailgun ~]# diff ./iptables-nat-bad.log ./iptables-nat-good.log
5a6,7
> -A POSTROUTING -s 10.108.80.0/24 -p tcp -m tcp --dport 8080 -m comment --comment "nginx-tcp-8080-unmasquerade" -j ACCEPT
> -A POSTROUTING -s 10.108.80.0/24 -p tcp -m tcp --dport 8000 -m comment --comment "nginx-tcp-8000-unmasquerade" -j ACCEPT
24d25
< -A DOCKER -d 127.0.0.1/32 -p tcp -m tcp --dport 8777 -j DNAT --to-destination 172.17.0.7:8777
38a40,43
> -A DOCKER -d 10.108.80.2/32 -p tcp -m tcp --dport 8777 -j DNAT --to-destination 172.17.0.7:8777
> -A DOCKER -d 127.0.0.1/32 -p tcp -m tcp --dport 8777 -j DNAT --to-destination 172.17.0.7:8777
> -A DOCKER -p tcp -m tcp --dport 8000 -j DNAT --to-destination 172.17.0.5:8000
> -A DOCKER -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.17.0.5:8080

[Bogdan Dobrelya (bogdando)]

Need reproducers for 5.1.x

[Stanislaw Bogatkin (sbogatkin)]

Matthew, can you update your status about this bug?

[Matthew Mosesohn (raytrac3r)]

When we implement new docker for 6.1, we will use host networking and these issues should disappear. It should land either today or tomorrow.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/155403

[OpenStack Infra (hudson-openstack)]

Change abandoned by Denis Meltsaykin (<email address hidden>) on branch: master
Review: https://review.openstack.org/155403
Reason: No need in case of new docker in 6.1

[Matthew Mosesohn (raytrac3r)]

It's a floating bug for 5.1.X and 6.0. It's just the fact that sometimes dnat rules from Docker don't get applied. It can be fixed by restarting container, but it is very rare and happens mostly just in virtual environments. This is already fixed in 6.1 with host networking feature.