Firewall operations can fail due to another process holding the xtables lock

[Mark Goddard]

Ironic inspector manipulates iptables rules to blacklist the MAC addresses of active ironic nodes. The iptables commands are executed without the -w option, which means that if another process is using iptables at the same time (holding the xtables lock, to be precise), then the commands will fail.

This failure could result in a number of problems but would most likely cause inspector's DHCP server to hand out IP addresses to active instances.

On one system when polling 'iptables -L' I can see the discovery chain appearing and disappearing every few 30 seconds to a minute.

The failure can occur in any of the iptables commands, but here is the inspector log from one failure.

Aug 12 08:50:00 localhost ironic-discoverd[205160]: DEBUG:ironic_discoverd.firewall:Blacklisting active MAC's set([u'00:1e:67:a3:97:a8', u'00:1e:67:a3:97:e9', u'00:1e:67:a3:97:0d', u'00:1e:67:a3:97:c1', u'00:1e:67:a3:98:3e'])
Aug 12 08:50:00 localhost ironic-discoverd[205160]: DEBUG:ironic_discoverd.firewall:Running iptables ('-D', 'INPUT', '-i', 'eno1-disc', '-p', 'udp', '--dport', '67', '-j', 'discovery_temp')
Aug 12 08:50:00 localhost ironic-discoverd[205160]: DEBUG:ironic_discoverd.firewall:Running iptables ('-A', 'discovery_temp', '-j', 'ACCEPT')
Aug 12 08:50:00 localhost ironic-discoverd[205160]: DEBUG:ironic_discoverd.firewall:ignoring failed iptables ('-D', 'INPUT', '-i', 'eno1-disc', '-p', 'udp', '--dport', '67', '-j', 'discovery_temp'):
Aug 12 08:50:00 localhost ironic-discoverd[205160]: iptables: No chain/target/match by that name.
Aug 12 08:50:00 localhost ironic-discoverd[205160]: DEBUG:ironic_discoverd.firewall:Running iptables ('-F', 'discovery_temp')
Aug 12 08:50:00 localhost ironic-discoverd[205160]: DEBUG:ironic_discoverd.firewall:Running iptables ('-I', 'INPUT', '-i', 'eno1-disc', '-p', 'udp', '--dport', '67', '-j', 'discovery_temp')
Aug 12 08:50:00 localhost ironic-discoverd[205160]: DEBUG:ironic_discoverd.firewall:Running iptables ('-X', 'discovery_temp')
Aug 12 08:50:00 localhost ironic-discoverd[205160]: DEBUG:ironic_discoverd.firewall:ignoring failed iptables ('-X', 'discovery_temp'):
Aug 12 08:50:00 localhost ironic-discoverd[205160]: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
Aug 12 08:50:00 localhost ironic-discoverd[205160]: DEBUG:ironic_discoverd.firewall:Running iptables ('-N', 'discovery_temp')
Aug 12 08:50:00 localhost ironic-discoverd[205160]: DEBUG:ironic_discoverd.firewall:Running iptables ('-D', 'INPUT', '-i', 'eno1-disc', '-p', 'udp', '--dport', '67', '-j', 'discovery')
Aug 12 08:50:00 localhost ironic-discoverd[205160]: ERROR:ironic_discoverd.firewall:iptables ('-N', 'discovery_temp') failed:
Aug 12 08:50:00 localhost ironic-discoverd[205160]: iptables: Chain already exists.
Aug 12 08:50:00 localhost ironic-discoverd[205160]: ERROR:ironic_discoverd.main:Periodic update failed
Aug 12 08:50:00 localhost ironic-discoverd[205160]: Traceback (most recent call last):
Aug 12 08:50:00 localhost ironic-discoverd[205160]: File "/usr/lib/python2.7/site-packages/ironic_discoverd/main.py", line 114, in periodic_update
Aug 12 08:50:00 localhost ironic-discoverd[205160]: firewall.update_filters()
Aug 12 08:50:00 localhost ironic-discoverd[205160]: File "/usr/lib/python2.7/site-packages/ironic_discoverd/firewall.py", line 113, in update_filters
Aug 12 08:50:00 localhost ironic-discoverd[205160]: _iptables('-N', NEW_CHAIN)
Aug 12 08:50:00 localhost ironic-discoverd[205160]: File "/usr/lib/python2.7/site-packages/ironic_discoverd/firewall.py", line 39, in _iptables
Aug 12 08:50:00 localhost ironic-discoverd[205160]: subprocess.check_output(cmd, **kwargs)
Aug 12 08:50:00 localhost ironic-discoverd[205160]: File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
Aug 12 08:50:00 localhost ironic-discoverd[205160]: raise CalledProcessError(retcode, cmd, output=output)
Aug 12 08:50:00 localhost ironic-discoverd[205160]: CalledProcessError: Command '('iptables', '-N', 'discovery_temp')' returned non-zero exit status 1

[Mark Goddard (mgoddard)]

Seen on CentOS7.1 running Kilo / inspector 1.1.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/216726

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/216726
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=3f7054ed4de0da80320c55ec42b1464d88bceae8
Submitter: Jenkins
Branch: master

commit 3f7054ed4de0da80320c55ec42b1464d88bceae8
Author: Dmitry Tantsur <email address hidden>
Date: Tue Aug 25 16:12:42 2015 +0200

    Pass -w flag to iptables to make it wait for xtables lock

    Change-Id: I8969da5e58550f71dba79b458feb63ed28e0585f
    Closes-Bug: #1484110

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/1.1
Review: https://review.openstack.org/238015

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Tantsur (<email address hidden>) on branch: stable/1.1
Review: https://review.openstack.org/238015
Reason: hmmm, -w is not supported everywhere, IIRC

[Mark Goddard (mgoddard)]

The original fix checked for support of the -w option but the stable/1.1 backport did not include this. Is there a plan to use the original fix?

[Dmitry Tantsur (divius)]

Due to maintenance complexity, stable/1.1 is in really deep freeze for now, so I don't expect any more updates, except for CVE's