pulling containers is failing the overcloud deployment

[wes hayutin]

http://logstash.openstack.org/#dashboard/file/logstash.json?query=(message%3A%5C%22Unable%20to%20find%20image%20'192.168.24.1%3A8787%2Ftripleomaster%2F%5C%22%20OR%20message%3A%5C%22Error%20while%20pulling%20image%3A%20Get%20http%3A%2F%2F192.168.24.1%3A8787%2Fv1%2Frepositories%2Ftripleomaster%2F%5C%22)%20AND%20tags%3Aconsole%20AND%20voting%3A1

Examples:
http://logs.openstack.org/83/616683/1/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/1dabc26/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-12_22_50_35

http://logs.openstack.org/83/616683/1/gate/tripleo-ci-centos-7-containers-multinode/e48c7ef/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-12_22_47_04

http://logs.openstack.org/00/613300/5/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bc825aa/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-13_01_41_32

This must be a infra issue, but not quite sure what yet. People need to see this is happening quite a bit so adding the alert tag and passing to ruck/rovers. It would be great if we could get a checked in elastic-recheck query checked in.

[Alex Schultz (alex-schultz)]

I think this might be related to the puppet 5.5.6 update. I'm seeing the iptables rules not being properly applied.

Under puppet 5.5.6 the iptables on the undercloud looks like:

# Generated by iptables-save v1.4.21 on Tue Nov 13 00:53:34 2018
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1:68]
:openstack-INPUT - [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "000 accept related established rules ipv4" -j ACCEPT
-A INPUT -p icmp -m state --state NEW -m comment --comment "001 accept all icmp ipv4" -j ACCEPT
-A INPUT -i lo -m state --state NEW -m comment --comment "002 accept all to lo interface ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 22 -m state --state NEW -m comment --comment "003 accept ssh ipv4" -j ACCEPT
-A INPUT -s 198.72.124.180/32 -j ACCEPT
-A INPUT -s 198.72.124.178/32 -j ACCEPT
-A INPUT -j openstack-INPUT
-A INPUT -m state --state NEW -m limit --limit 20/min --limit-burst 15 -m comment --comment "998 log all ipv4" -j LOG
-A openstack-INPUT -i lo -j ACCEPT
-A openstack-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A openstack-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A openstack-INPUT -p tcp -m state --state NEW -m tcp --dport 19885 -j ACCEPT
-A openstack-INPUT -s 172.24.4.0/23 -p udp -m udp --dport 69 -j ACCEPT
-A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 6385 -j ACCEPT
-A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 80 -j ACCEPT
-A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8000 -j ACCEPT
-A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8003 -j ACCEPT
-A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8004 -j ACCEPT
-A openstack-INPUT -m limit --limit 2/min -j LOG --log-prefix "iptables dropped: "
-A openstack-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Tue Nov 13 00:53:34 2018

But under 4.8.2, it's:
 Generated by iptables-save v1.4.21 on Mon Nov 12 19:27:19 2018
*nat
:PREROUTING ACCEPT [2:64]
:INPUT ACCEPT [2:64]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -d 169.254.169.254/32 -i br-ct...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/617441

[chandan kumar (chkumar246)]

We have reverted puppet-5 in RDO, https://review.rdoproject.org/r/#/c/17333/1