puppet-tripleo package is outdated and doesn't know of enabled_services hiera key

[Cédric Jeanneret]

Hello,

When following this doc[1], it appears the firewall isn't properly set: there are only a bunch of rules allowing ping, ESTABLISHED, and final DROP, and it lacks the ones allowing SSH access to the overcloud.
This leads to a timeout, since ansible can't access the host any more.

Cheers,

C.

[1] https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ansible_config_download.html#manual-config-download

[Cédric Jeanneret (cjeanner)]

Precisions:
- this also happens with the "more standard" deploy command
- it happens on a compute node, while the controller does have all the wanted rules
- the config-download/Compute/deployment-hieradata.j2.yaml.rendered has a lot of rules, but in then end only a bunch of them are applied
- this deploy involves multi-nic + network isolation
- apparently this is NOT related to https://bugzilla.redhat.com/show_bug.cgi?id=1724560

Here are the rules being applied on the compute:
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination
 621K 1747M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* 000 accept related established rules ipv4 */
    0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW /* 001 accept all icmp ipv4 */
    0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 state NEW /* 002 accept all to lo interface ipv4 */
    0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 state NEW limit: avg 20/min burst 15 /* 998 log all ipv4 */ LOG flags 0 level 4
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state NEW /* 999 drop all ipv4 */

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 567K packets, 29M bytes)
 pkts bytes target prot opt in out source destination

The first one explains why I'm not locked off when connected prior the rule application.

The attached file contains all the hieradata for the compute node.

I'm still digging in the logs in order to find something.

[Cédric Jeanneret (cjeanner)]

Puppet seems to be OK on the compute with it:

Jul 16 07:53:57 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[000 accept related established rules]/Firewall[000 accept related established rules ipv4]/ensure: created
Jul 16 07:53:57 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[000 accept related established rules]/Firewall[000 accept related established rules ipv6]/ensure: created
Jul 16 07:53:57 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[001 accept all icmp]/Firewall[001 accept all icmp ipv4]/ensure: created
Jul 16 07:53:58 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[001 accept all icmp]/Firewall[001 accept all icmp ipv6]/ensure: created
Jul 16 07:53:58 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[002 accept all to lo interface]/Firewall[002 accept all to lo interface ipv4]/ensure: created
Jul 16 07:53:58 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[002 accept all to lo interface]/Firewall[002 accept all to lo interface ipv6]/ensure: created
Jul 16 07:53:58 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[004 accept ipv6 dhcpv6]/Firewall[004 accept ipv6 dhcpv6 ipv6]/ensure: created
Jul 16 07:53:58 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[998 log all]/Firewall[998 log all ipv4]/ensure: created
Jul 16 07:53:58 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[998 log all]/Firewall[998 log all ipv6]/ensure: created
Jul 16 07:53:59 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all ipv4]/ensure: created
Jul 16 07:53:59 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all ipv6]/ensure: created
Jul 16 07:53:59 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Firewall::Linux::Redhat/File[/etc/sysconfig/iptables]/seluser: seluser changed 'unconfined_u' to 'system_u'
Jul 16 07:53:59 overcloud-0-novacompute-0 puppet-user[31594]: Notice: /Stage[main]/Firewall::Linux::Redhat/File[/etc/sysconfig/ip6tables]/seluser: seluser changed 'unconfined_u' to 'system_u'

No error related to firewalling so far. And we can see the lack of ssh access authorization :/. Still digging.

[Cédric Jeanneret (cjeanner)]

NOTE: controller is also affected - the ssh rule isn't present!

I'm wondering if this isn't linked to that change: https://review.opendev.org/#/c/669414/

[Cédric Jeanneret (cjeanner)]

Guess I just found the issue:
my overcloud nodes have puppet-tripleo-11.0.1-0.20190711120944.787387a.el7.noarch

This version doesn't use the new "enabled_services" hiera key, while the old "service_names" isn't set anymore.

[Cédric Jeanneret (cjeanner)]

In the end it was some weird mix-up in my env, with different repositories for both under and overcloud.