Compute deployment fails: Iptables-firewall module cannot prefetch openstack firewall rules

[Vladimir Kuklin]

Compute redeployment fails if we want to run puppet on the compute node already running virtual machines.
This seems to be due to our puppet iptables module is unable to parse openstack rules for VMs:

Error: Could not prefetch firewall provider 'iptables': undefined method `[]' for nil:NilClass
/etc/puppet/modules/firewall/lib/puppet/provider/firewall/iptables.rb:218:in `rule_to_hash'
/etc/puppet/modules/firewall/lib/puppet/provider/firewall/iptables.rb:214:in `each'
/etc/puppet/modules/firewall/lib/puppet/provider/firewall/iptables.rb:214:in `rule_to_hash'
/etc/puppet/modules/firewall/lib/puppet/provider/firewall/iptables.rb:138:in `instances'
/etc/puppet/modules/firewall/lib/puppet/provider/firewall/iptables.rb:133:in `each'
/etc/puppet/modules/firewall/lib/puppet/provider/firewall/iptables.rb:133:in `instances'
/etc/puppet/modules/firewall/lib/puppet/provider/firewall.rb:7:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `prefetch'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:167:in `prefetch_if_necessary'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:67:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `call'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:116:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `benchmark'
/usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
/usr/bin/puppet:4

-A INPUT -j neutron-openvswi-INPUT
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -p icmp -m comment --comment "000 accept all icmp requests" -j ACCEPT
-A INPUT -i lo -m comment --comment "001 accept all to lo interface" -j ACCEPT
-A INPUT -m comment --comment "002 accept related established rules" -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -s 10.20.0.2/32 -p tcp -m multiport --sports 4369,5672,41055,55672,61613 -m comment --comment "003 remote rabbitmq " -j ACCEPT
-A INPUT -p tcp -m multiport --sports 8140 -m comment --comment "004 remote puppet " -j ACCEPT
-A INPUT -p tcp -m multiport --ports 22 -m comment --comment "020 ssh" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 80,443 -m comment --comment "100 http" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 3306,3307,4567,4568 -m comment --comment "101 mysql" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 5000,35357 -m comment --comment "102 keystone" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 8080,6000,6001,6002 -m comment --comment "103 swift" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 9292,9191,8773 -m comment --comment "104 glance" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 8774,8775,8776,6080 -m comment --comment "105 nova " -j ACCEPT
-A INPUT -p tcp -m multiport --ports 4369,5672,5673,41055 -m comment --comment "106 rabbitmq " -j ACCEPT
-A INPUT -p tcp -m multiport --ports 11211 -m comment --comment "107 memcached tcp" -j ACCEPT
-A INPUT -p udp -m multiport --ports 11211 -m comment --comment "107 memcached udp" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 873 -m comment --comment "108 rsync" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 3260 -m comment --comment "109 iscsi " -j ACCEPT
-A INPUT -p tcp -m multiport --ports 9696 -m comment --comment "110 neutron " -j ACCEPT
-A INPUT -p udp -m multiport --ports 67 -m comment --comment "111 dhcp-server" -j ACCEPT
-A INPUT -p udp -m multiport --ports 53 -m comment --comment "111 dns-server" -j ACCEPT
-A INPUT -p udp -m multiport --ports 123 -m comment --comment "112 ntp-server" -j ACCEPT
-A INPUT -p udp -m multiport --ports 5404 -m comment --comment "113 corosync-input" -j ACCEPT
-A INPUT -p udp -m multiport --ports 5405 -m comment --comment "114 corosync-output" -j ACCEPT
-A INPUT -p udp -m multiport --ports 58882 -m comment --comment "115 openvswitch db" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 5666 -m comment --comment "116 nrpe-server" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 16509 -m comment --comment "117 libvirt" -j ACCEPT
-A INPUT -s 192.168.0.0/24 -p tcp -m multiport --ports 5900:6100 -m comment --comment "118 vnc ports" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 8777 -m comment --comment "119 ceilometer" -j ACCEPT
-A INPUT -p tcp -m comment --comment "999 drop all other requests" -j DROP
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-openvswi-OUTPUT
-A neutron-filter-top -j neutron-openvswi-local
-A neutron-openvswi-FORWARD -m physdev --physdev-out tap3b5357fb-de --physdev-is-bridged -j neutron-openvswi-sg-chain
-A neutron-openvswi-FORWARD -m physdev --physdev-in tap3b5357fb-de --physdev-is-bridged -j neutron-openvswi-sg-chain
-A neutron-openvswi-FORWARD -m physdev --physdev-out tap8c651d44-2d --physdev-is-bridged -j neutron-openvswi-sg-chain
-A neutron-openvswi-FORWARD -m physdev --physdev-in tap8c651d44-2d --physdev-is-bridged -j neutron-openvswi-sg-chain
-A neutron-openvswi-INPUT -m physdev --physdev-in tap3b5357fb-de --physdev-is-bridged -j neutron-openvswi-o3b5357fb-d
-A neutron-openvswi-INPUT -m physdev --physdev-in tap8c651d44-2d --physdev-is-bridged -j neutron-openvswi-o8c651d44-2
-A neutron-openvswi-i3b5357fb-d -m state --state INVALID -j DROP
-A neutron-openvswi-i3b5357fb-d -m state --state RELATED,ESTABLISHED -j RETURN
-A neutron-openvswi-i3b5357fb-d -p icmp -j RETURN
-A neutron-openvswi-i3b5357fb-d -s 192.168.111.15/32 -j RETURN
-A neutron-openvswi-i3b5357fb-d -p tcp -m tcp -m multiport --dports 1:65535 -j RETURN
-A neutron-openvswi-i3b5357fb-d -s 192.168.111.2/32 -p udp -m udp --sport 67 --dport 68 -j RETURN
-A neutron-openvswi-i3b5357fb-d -j neutron-openvswi-sg-fallback
-A neutron-openvswi-i8c651d44-2 -m state --state INVALID -j DROP
-A neutron-openvswi-i8c651d44-2 -m state --state RELATED,ESTABLISHED -j RETURN
-A neutron-openvswi-i8c651d44-2 -p icmp -j RETURN
-A neutron-openvswi-i8c651d44-2 -s 192.168.111.16/32 -j RETURN
-A neutron-openvswi-i8c651d44-2 -p tcp -m tcp -m multiport --dports 1:65535 -j RETURN
-A neutron-openvswi-i8c651d44-2 -s 192.168.111.2/32 -p udp -m udp --sport 67 --dport 68 -j RETURN
-A neutron-openvswi-i8c651d44-2 -j neutron-openvswi-sg-fallback
-A neutron-openvswi-o3b5357fb-d -p udp -m udp --sport 68 --dport 67 -j RETURN
-A neutron-openvswi-o3b5357fb-d -j neutron-openvswi-s3b5357fb-d
-A neutron-openvswi-o3b5357fb-d -p udp -m udp --sport 67 --dport 68 -j DROP
-A neutron-openvswi-o3b5357fb-d -m state --state INVALID -j DROP
-A neutron-openvswi-o3b5357fb-d -m state --state RELATED,ESTABLISHED -j RETURN
-A neutron-openvswi-o3b5357fb-d -j RETURN
-A neutron-openvswi-o3b5357fb-d -j neutron-openvswi-sg-fallback
-A neutron-openvswi-o8c651d44-2 -p udp -m udp --sport 68 --dport 67 -j RETURN
-A neutron-openvswi-o8c651d44-2 -j neutron-openvswi-s8c651d44-2
-A neutron-openvswi-o8c651d44-2 -p udp -m udp --sport 67 --dport 68 -j DROP
-A neutron-openvswi-o8c651d44-2 -m state --state INVALID -j DROP
-A neutron-openvswi-o8c651d44-2 -m state --state RELATED,ESTABLISHED -j RETURN
-A neutron-openvswi-o8c651d44-2 -j RETURN
-A neutron-openvswi-o8c651d44-2 -j neutron-openvswi-sg-fallback
-A neutron-openvswi-s3b5357fb-d -s 192.168.111.16/32 -m mac --mac-source FA:16:3E:B9:45:20 -j RETURN
-A neutron-openvswi-s3b5357fb-d -j DROP
-A neutron-openvswi-s8c651d44-2 -s 192.168.111.15/32 -m mac --mac-source FA:16:3E:69:38:01 -j RETURN
-A neutron-openvswi-s8c651d44-2 -j DROP
-A neutron-openvswi-sg-chain -m physdev --physdev-out tap3b5357fb-de --physdev-is-bridged -j neutron-openvswi-i3b5357fb-d
-A neutron-openvswi-sg-chain -m physdev --physdev-in tap3b5357fb-de --physdev-is-bridged -j neutron-openvswi-o3b5357fb-d
-A neutron-openvswi-sg-chain -m physdev --physdev-out tap8c651d44-2d --physdev-is-bridged -j neutron-openvswi-i8c651d44-2
-A neutron-openvswi-sg-chain -m physdev --physdev-in tap8c651d44-2d --physdev-is-bridged -j neutron-openvswi-o8c651d44-2
-A neutron-openvswi-sg-chain -j ACCEPT
-A neutron-openvswi-sg-fallback -j DROP

[Bogdan Dobrelya (bogdando)]

If you run firewall module tests, you could see there are plenty of failures (probably because of bad regexes)
http://pastebin.com/8WHTChnQ
I hope some of them (confine, ipv6) could be safely ignored, while others should be addressed.

[Vladimir Kuklin (vkuklin)]

moving to 5.0 as this affects idempotency of puppet run on compute nodes

[Tomasz Głuch (tomekg)]

I'm also affected but this bug, in Fuel 4.1, multi-node cluster on physical nodes.
As I was unable to redeploy environment containing compute-nodes with running instances, so I've analyzed this issue.
First successful approach was to completely disable firewall in Puppet, but after a while of digging I've found a direct cause of problem.

It is related to missing support for "mac" match in PuppetLabs' firewall module. Fuel contains older version of this module, but in the newest one this match is still missing. I've prepared fix to upstream release (1.0.2). (Attached).

[Dmitry Borodaenko (angdraug)]

A more generic fix is proposed here:
https://review.openstack.org/92167

The fix relies on the fact that Puppet always uses --comment when creating rules, so it is safe to assume that rules that don't have a comment should not be touched by Puppet.

[Tomasz Głuch (tomekg)]

I agree this is more generic, but IMHO workaround should check comment string for "^\d+" prefix existence at least, to avoid another hard to track issues (identifying and solving this issue took 3 weeks). Problem will occur again when Neutron or user adds rule with comment at random time in the future.

[Dmitry Ilyin (idv1985)]

Ok, perhaps it would be better not to filter out by comment...

I've added MAC support Tomasz provided and it works.
https://review.openstack.org/#/c/92167/

But there could be more errors with other unsupported matchers. Maybe it would be better to add --comment "\d+\s\S+" filter?

[Bogdan Dobrelya (bogdando)]

Nice catch with the MAC support, Tomasz.
Please consider to sync our firewall module with the 1.0.2 as well (https://review.openstack.org/#/c/92351/)
At least it would fix the broken specs for our module and prevent us from re-engineering the wheel

[Dmitry Ilyin (idv1985)]

Here is my latest patch with both mac support and comment based filter https://review.openstack.org/#/c/92167/6

[Bogdan Dobrelya (bogdando)]

Once we accept Dmitry's patch with MAC support, I will rebase dependent upstream sync patches https://review.openstack.org/#/c/92351/ and https://review.openstack.org/#/c/92361/ and I hope we will accept them as well.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/92167
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f6195b0a65810b5c10515ee981f174be00e92c4f
Submitter: Jenkins
Branch: master

commit f6195b0a65810b5c10515ee981f174be00e92c4f
Author: Dmitry Ilyin <email address hidden>
Date: Wed May 7 20:37:15 2014 +0400

    Add MAC match support to firewall

    * Fixes parsing errors and allows defining
      rules with MAC matches. Thanks to Tomasz Głuch
      for pointing the lack of MAC support out.

    * Notify neutron-ovs-agent on compute nodes
      to let it clen out saved rules from old and
      removed instances.

    Change-Id: Ib82bf37644aaab3b4e794934ddef40593a87fb7d
    Partial-Bug: 1308963

[Mike Scherbakov (mihgen)]

What should we do next with this? Patch is committed, but I'm not sure if it's full fix, however enough for 5.0 as far as I understand. Should we close this bug and open another one for 5.1?

[Meg McRoberts (dreidellhasa)]

Added to Fixed Issues in 5.0 Release Notes.

[Nastya Urlapova (aurlapova)]

Verified on
{
build_id: "2014-05-27_05-51-41",
mirantis: "yes",
build_number: "26",
ostf_sha: "a8b7660082a6f152794c610d6abe30d360fd577d",
nailgun_sha: "bd09f89ef56176f64ad5decd4128933c96cb20f4",
production: "docker",
api: "1.0",
fuelmain_sha: "505741e4f431f85a8d0252fc42754d10c0326c1a",
astute_sha: "a7eac46348dc77fc2723c6fcc3dbc66cc1a83152",
release: "5.0",
fuellib_sha: "2f79c0415159651fc1978d99bd791079d1ae4a06"
}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/96882

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/96882
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=343b3d2c486c9dac779458303d5a41094dbe60ff
Submitter: Jenkins
Branch: stable/4.1

commit 343b3d2c486c9dac779458303d5a41094dbe60ff
Author: Dmitry Ilyin <email address hidden>
Date: Wed May 7 20:37:15 2014 +0400

    Add MAC match support to firewall

    * Fixes parsing errors and allows defining
      rules with MAC matches. Thanks to Tomasz Głuch
      for pointing the lack of MAC support out.

    * Notify neutron-ovs-agent on compute nodes
      to let it clen out saved rules from old and
      removed instances.

    Change-Id: Ib82bf37644aaab3b4e794934ddef40593a87fb7d
    Partial-Bug: 1308963

[Meg McRoberts (dreidellhasa)]

Documented as fixed in 4.1.1