tripleo

Overcloud deployment on pre-deployed servers fails due to missing SSH firewall rule

Bug #1839324 reported by Cédric Jeanneret on 2019-08-07

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	High	Cédric Jeanneret	tripleo train-3

Bug Description

Description of problem:
Deployment on pre-deployed servers fails due to missing SSH firewall rules.

After https://review.opendev.org/#/c/667295/ landed deployment on pre-deployed servers doesn't work anymore since the SSH firewall rules are not set anymore:

How reproducible:
100%

Steps to Reproduce:
1. Deploy Stein overcloud on pre-deployed servers

Actual results:
overcloud deployments fails because the firewall rules set by Director do not include SSH access hence ansible can't reach the nodes any longer.

Expected results:
Firewall rules to allow SSH access are set.

Additional info:

The resulting firewall rules on the server do not include SSH access:

[root@controller-0 ~]# Chain INPUT (policy ACCEPT)
target prot opt source ACCEPT all -- 0.0.0.0/0 ACCEPT icmp -- 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT udp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT udp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 172.17.1.0/24 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT udp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 ACCEPT udp -- 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 LOG all -- 0.0.0.0/0 DROP all -- 0.0.0.0/0 iptables -nL
destination
0.0.0.0/0 state RELATED,ESTABLISHED /* 000 accept related established rules ipv4 */
0.0.0.0/0 state NEW /* 001 accept all icmp ipv4 */
0.0.0.0/0 state NEW /* 002 accept all to lo interface ipv4 */
0.0.0.0/0 multiport dports 873,3123,3306,4444,4567,4568,9200 state NEW /* 104 mysql galera-bundle ipv4 */
0.0.0.0/0 multiport dports 1993 state NEW /* 107 haproxy stats ipv4 */
0.0.0.0/0 multiport dports 3124,6379,26379 state NEW /* 108 redis-bundle ipv4 */
0.0.0.0/0 multiport dports 3122,4369,5672,25672 state NEW /* 109 rabbitmq-bundle ipv4 */
0.0.0.0/0 multiport dports 6789,3300 state NEW /* 110 ceph_mon ipv4 */
0.0.0.0/0 multiport dports 5000,13000,35357 state NEW /* 111 keystone ipv4 */
0.0.0.0/0 multiport dports 9292,13292 state NEW /* 112 glance_api ipv4 */
0.0.0.0/0 multiport dports 6800:7300 state NEW /* 113 ceph_mgr ipv4 */
0.0.0.0/0 multiport dports 8774,13774 state NEW /* 113 nova_api ipv4 */
0.0.0.0/0 multiport dports 9696,13696 state NEW /* 114 neutron api ipv4 */
0.0.0.0/0 multiport dports 4789 state NEW /* 118 neutron vxlan networks ipv4 */
0.0.0.0/0 multiport dports 8776,13776 state NEW /* 119 cinder ipv4 */
0.0.0.0/0 multiport dports 6081 state NEW /* 119 neutron geneve networks ipv4 */
0.0.0.0/0 multiport dports 3260 state NEW /* 120 iscsi initiator ipv4 */
0.0.0.0/0 multiport dports 3125,6641,6642 state NEW /* 121 OVN DB server ports ipv4 */
0.0.0.0/0 multiport dports 11211 state NEW /* 121 memcached 172.17.1.0/24 ipv4 */
0.0.0.0/0 multiport dports 8080,13808 state NEW /* 122 swift proxy ipv4 */
0.0.0.0/0 multiport dports 873,6000,6001,6002 state NEW /* 123 swift storage ipv4 */
0.0.0.0/0 multiport dports 8004,13004 state NEW /* 125 heat_api ipv4 */
0.0.0.0/0 multiport dports 8000,13800 state NEW /* 125 heat_cfn ipv4 */
0.0.0.0/0 multiport dports 80,443 state NEW /* 126 horizon ipv4 */
0.0.0.0/0 multiport dports 8042,13042 state NEW /* 128 aodh-api ipv4 */
0.0.0.0/0 multiport dports 8041,13041 state NEW /* 129 gnocchi-api ipv4 */
0.0.0.0/0 multiport dports 2224,3121,21064 state NEW /* 130 pacemaker tcp ipv4 */
0.0.0.0/0 multiport dports 5405 state NEW /* 131 pacemaker udp ipv4 */
0.0.0.0/0 multiport dports 6080,13080 state NEW /* 137 nova_vnc_proxy ipv4 */
0.0.0.0/0 multiport dports 8778,13778 state NEW /* 138 nova_placement ipv4 */
0.0.0.0/0 multiport dports 8775,13775 state NEW /* 139 nova_metadata ipv4 */
0.0.0.0/0 multiport dports 8125 state NEW /* 140 gnocchi-statsd ipv4 */
0.0.0.0/0 multiport dports 8977,13977 state NEW /* 140 panko-api ipv4 */
0.0.0.0/0 state NEW limit: avg 20/min burst 15 /* 998 log all ipv4 */ LOG flags 0 level 4
0.0.0.0/0 state NEW /* 999 drop all ipv4 */

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-07: Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/675124

Changed in tripleo:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-10: Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/675124
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9e5efd591101928daa7337f4b7a4b076fab24ff8
Submitter: Zuul
Branch: master

commit 9e5efd591101928daa7337f4b7a4b076fab24ff8
Author: Cédric Jeanneret <email address hidden>
Date: Wed Aug 7 14:11:26 2019 +0200

Ensure we get a subnet for ctlplane

    It might happen the ServiceData['net_cidr_map']['ctlplane'] has a "null"
    value, preventing tripleo::firewall to set the proper firewall rule for
    ctlplane SSH access.

This patch creates a conditions ensuring we have a value for that
parameter, and if not, fetches the value out of Hiera.

Closes-Bug: #1839324
Change-Id: I84cdb47f40df546162f1244e99c2391fd71daad7

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-12: Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/675814

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-12: Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/675814
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2a684c0b888ca0cf5277d1e6b5d6d689dccb7496
Submitter: Zuul
Branch: stable/stein

commit 2a684c0b888ca0cf5277d1e6b5d6d689dccb7496
Author: Cédric Jeanneret <email address hidden>
Date: Wed Aug 7 14:11:26 2019 +0200

Ensure we get a subnet for ctlplane

    It might happen the ServiceData['net_cidr_map']['ctlplane'] has a "null"
    value, preventing tripleo::firewall to set the proper firewall rule for
    ctlplane SSH access.

This patch creates a conditions ensuring we have a value for that
parameter, and if not, fetches the value out of Hiera.

    Closes-Bug: #1839324
    Change-Id: I84cdb47f40df546162f1244e99c2391fd71daad7
    (cherry picked from commit 9e5efd591101928daa7337f4b7a4b076fab24ff8)

tags:

added: in-stable-stein

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-13: Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/676136

Revision history for this message

Cédric Jeanneret (cjeanner) wrote on 2019-08-14:

Update: the issue was caused by some out of dated doc.

The current "related fix" will push a validation that ensures we won't face this situation again.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-15: Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/676136
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=34f3cbde646cbf263b4c2db7f828ba2edb13e2bc
Submitter: Zuul
Branch: master

commit 34f3cbde646cbf263b4c2db7f828ba2edb13e2bc
Author: Cédric Jeanneret <email address hidden>
Date: Tue Aug 13 09:59:06 2019 +0200

Ensure we get at least one ctlplane subnet

    This will prevent situations where firewall rules are applied to the
    overcloud nodes without any tagged ctlplane subnet, leading to a lockout
    from the nodes, making the whole deploy failing (and node unreachable).

This is especially important for the deployed-server case.

Related-Bug: #1839324
Change-Id: Ib3eca07050474930bfe60d6db24ef1c683079a24

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-15: Related fix proposed to tripleo-heat-templates (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/676580

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-15: Related fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/676580
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8b8b6dc1822adfdfa8d9e5d84617a34cc723738c
Submitter: Zuul
Branch: stable/stein

commit 8b8b6dc1822adfdfa8d9e5d84617a34cc723738c
Author: Cédric Jeanneret <email address hidden>
Date: Tue Aug 13 09:59:06 2019 +0200

Ensure we get at least one ctlplane subnet

This is especially important for the deployed-server case.

    Related-Bug: #1839324
    Change-Id: Ib3eca07050474930bfe60d6db24ef1c683079a24
    (cherry picked from commit 34f3cbde646cbf263b4c2db7f828ba2edb13e2bc)