ip6tables-restore fails in neutron_openvswitch_agent

[Serguei Bezverkhi]

2016-08-22 11:54:58.697 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-baa3335b-0013-42dd-856a-64a5c2557a01', 'ip6tables-restore', '-n'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
2016-08-22 11:54:58.970 1 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: # Generated by iptables_manager

Usage: ip6tables-restore [-b] [-c] [-v] [-t] [-h]
           [ --binary ]
           [ --counters ]
           [ --verbose ]
           [ --test ]
           [ --help ]
           [ --noflush ]
          [ --modprobe=<command>]

It seems iptables-1.4.21-16.el7.x86_64 does not support '-n' option used in the command above.

[Brian Haley (brian-haley)]

I think the usage message was just never updated, the code in the upstream repo shows --no-flush/-n as supported, https://git.netfilter.org/iptables/tree/iptables/ip6tables-restore.c

I'll send out a patch to fix the usage message, which is orthagonal to this bug.

Can you give any more information from the logs, or a pointer to a test that is failing?

[Serguei Bezverkhi (sbezverk)]

Here you go, here is a complete log I see:

2016-08-22 12:58:20.300 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:140
2016-08-22 12:58:20.304 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-baa3335b-0013-42dd-856a-64a5c2557a01', 'ip6tables-save'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
2016-08-22 12:58:20.585 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:140
2016-08-22 12:58:20.588 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-baa3335b-0013-42dd-856a-64a5c2557a01', 'ip6tables-restore', '-n'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
2016-08-22 12:58:20.873 1 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: # Generated by iptables_manager
*filter
:FORWARD - [0:0]
:INPUT - [0:0]
:OUTPUT - [0:0]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
:neutron-l3-agent-scope - [0:0]
-I FORWARD 1 -j neutron-filter-top
-I FORWARD 2 -j neutron-l3-agent-FORWARD
-I INPUT 1 -j neutron-l3-agent-INPUT
-I OUTPUT 1 -j neutron-filter-top
-I OUTPUT 2 -j neutron-l3-agent-OUTPUT
-I neutron-filter-top 1 -j neutron-l3-agent-local
-I neutron-l3-agent-FORWARD 1 -j neutron-l3-agent-scope
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*mangle
:FORWARD - [0:0]
:INPUT - [0:0]
:OUTPUT - [0:0]
:POSTROUTING - [0:0]
:PREROUTING - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-scope - [0:0]
-I FORWARD 1 -j neutron-l3-agent-FORWARD
-I INPUT 1 -j neutron-l3-agent-INPUT
-I OUTPUT 1 -j neutron-l3-agent-OUTPUT
-I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING
-I PREROUTING 1 -j neutron-l3-agent-PREROUTING
-I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-scope
-I neutron-l3-agent-PREROUTING 2 -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*raw
:OUTPUT - [0:0]
:PREROUTING - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
-I OUTPUT 1 -j neutron-l3-agent-OUTPUT
-I PREROUTING 1 -j neutron-l3-agent-PREROUTING
COMMIT
# Completed by iptables_manager
; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
ip6tables-restore v1.4.21: ip6tables-restore: unable to initialize table 'filter'

Error occurred at line: 2
Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.

2016-08-22 12:58:20.874 1 ERROR neutron.agent.linux.iptables_manager [-] IPTablesManager.apply failed to apply the following set of iptables rules:
      1. # Generated by iptables_manage...

[Brian Haley (brian-haley)]

This looks like the issue:

Stdout: ; Stderr: RTNETLINK answers: Invalid argument
ip6tables-restore v1.4.21: ip6tables-restore: unable to initialize table 'filter'

You might need to manually load the module? It's there in my config:

$ lsmod | grep ip6table_filter
ip6table_filter 16384 0
ip6_tables 28672 1 ip6table_filter
x_tables 36864 8 ip6table_filter,xt_mark,ip_tables,xt_multiport,iptable_filter,ebtables,iptable_mangle,ip6_tables

[Serguei Bezverkhi (sbezverk)]

Looks like you are right, after loading ip6tables module, I do not see the reoccurring traceback.
Thank you very much!

[Brian Haley (brian-haley)]

iptables patch: http://marc.info/?l=netfilter-devel&m=147189719220901&w=2

[Vladislav Belogrudov (vlad-belogrudov)]

this bug still happens. Neutron openvswitch agent container tries to run ip6tables-restore and fails because there is no ip6table_filter module loaded. The module normally is loaded by the command itself. But inside the container we don't provide /lib/modules ... With proper host mount the error is gone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/474167
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=91789c4dfd749cb25188ea3c4c6208a18b599778
Submitter: Jenkins
Branch: master

commit 91789c4dfd749cb25188ea3c4c6208a18b599778
Author: Vladislav Belogrudov <email address hidden>
Date: Wed Jun 14 14:37:37 2017 +0300

    Add /lib/modules to neutron_openvswitch_agent

    On many systems IPv6 related modules are not loaded by default.
    Usually when one runs ip6tables-* commands required modules are
    probed. In neutron_openvswitch_agent container /lib/modules
    does not exist. The commands fail to process ip6 firewall
    rules as consequence.

    Change-Id: Ic4e72eb4f5304f013b7a09ddd31794cfafa67e0b
    Closes-Bug: #1615715

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 5.0.0.0b3 development milestone.