tripleo

Bug #1657108
Comment #1

Comment 1 for bug 1657108

Revision history for this message

Michele Baldessari (michele) wrote on 2017-01-18: Re: pacemaker cluster setup fails if the image has prepopulated firewall rules with no cluster/pcsd access

So I can confirm this theory. I have injected the following (standard?) iptables file in /etc/sysconfig:
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

On a deploy where pacemaker is included in controller0,1,2 and galera0. And where pacemaker-remote is added to remote-0 and rabbit-0 nodes I get the following situation:
A) controller0,1,2:
[root@overcloud-controller-0 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
6394 1363K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
  189 11340 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
    1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
  861 54822 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 7933 packets, 1231K bytes)
pkts bytes target prot opt in out source destination
[root@overcloud-controller-0 ~]# systemctl is-active iptables
active

So in this situation the command that sets up the cluster and which expects to be able to talk to the pcsd ports will fail:
2017-01-18 11:39:54 +0000 Puppet (debug): Executing: '/sbin/pcs cluster auth overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 overcloud-galera-0 -u hacluster -p a6KKVbqfDqep2zgL --force'

Interestingly we also have
B) overcloud-galera-0
[root@overcloud-galera-0 ~]# more /etc/sysconfig/iptables
# sample configuration for iptables service
# you can edit this manually or use system-config-firewall
# please do not ask us to add additional ports/services to this default configuration
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
[root@overcloud-galera-0 ~]# systemctl status iptables
● iptables.service - IPv4 firewall with iptables
Loaded: loaded (/usr/lib/systemd/system/iptables.service; disabled; vendor preset: disabled)
Active: inactive (dead)

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 1031 packets, 470K bytes)
pkts bytes target prot opt in out source destination

So I think that what is happening here is that tripleo firewall gets configured too late on the controllers and the stock iptables rule prevent pcs from working. Investigating...

On a deploy where pacemaker is included in controller0,1,2 and galera0. And where pacemaker-remote is added to remote-0 and rabbit-0 nodes I get the following situation:
A) controller0,1,2:
[root@overcloud-controller-0 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 6394 1363K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
  189 11340 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
  861 54822 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 7933 packets, 1231K bytes)
 pkts bytes target     prot opt in     out     source               destination         
[root@overcloud-controller-0 ~]# systemctl is-active iptables
active

Interestingly we also have
B) overcloud-galera-0
[root@overcloud-galera-0 ~]# more /etc/sysconfig/iptables
# sample configuration for iptables service
# you can edit this manually or use system-config-firewall
# please do not ask us to add additional ports/services to this default configuration
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
[root@overcloud-galera-0 ~]# systemctl status iptables
● iptables.service - IPv4 firewall with iptables
   Loaded: loaded (/usr/lib/systemd/system/iptables.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

C) overcloud-rabbit-0
[root@overcloud-rabbit-0 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  695 4107K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 000 accept related established rules ipv4 */ state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            /* 001 accept all icmp ipv4 */ state NEW
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0            /* 002 accept all to lo interface ipv4 */ state NEW
    1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22 /* 003 accept ssh ipv4 */ state NEW
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 123 /* 105 ntp ipv4 */ state NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 4369,5672,25672 /* 109 rabbitmq ipv4 */ state NEW
  168 15288 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 161 /* 127 snmp ipv4 */ state NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 3121 /* 130 pacemaker_remote tcp ipv4 */ state NEW
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 998 log all */ LOG flags 0 level 4
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 999 drop all ipv4 */ state NEW

Chain OUTPUT (policy ACCEPT 1031 packets, 470K bytes)
 pkts bytes target     prot opt in     out     source               destination

So I think that what is happening here is that tripleo firewall gets configured too late on the controllers and the stock iptables rule prevent pcs from working. Investigating...