Duplicate IPtables rule detected warning message seen in L3 agent

[Swaminathan Vasudevan]

In recent L3 agent logs in the gate we have been seeing this warning message associated with the DVR router jobs.
Right now none of the jobs are failing, but we need to see why this warning message is showing up in the logs or it might be due to some hidden issues.

http://logs.openstack.org/89/255989/11/check/gate-tempest-dsvm-neutron-dvr/e3464a5/logs/screen-q-l3.txt.gz?level=WARNING#_2016-01-18_13_34_52_764

[Swaminathan Vasudevan (swaminathan-vasudevan)]

I see that there is already a patch to address this issue. But it seems that this issue was also seen on normal neutron jobs and not just associated with dvr routers.

https://review.openstack.org/#/c/255484/1

[Brian Haley (brian-haley)]

This is actually pretty easy to reproduce on a DVR setup:

- associate a floating IP
- dis-associated it
- re-associate it

Since the fip namespace doesn't get destroyed on the dis-associate, the next associated will call the namespace creation code, which will add the rule again.

I'll look into it some more.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Brian I think I found out the problem
fip_ns.create() is being called everytime when there is a fip associated, even though there is a fip namespace.

So the create should not be called for every fip association if the fip namespace exists.

[Cedric Brandily (cbrandily)]

Is it really a trouble to raise such warning? As the warning also solves the duplication trouble

[Brian Haley (brian-haley)]

I think the warning just shows the agent could be more efficient and not do unnecessary work, in this case it is an iptables-save/restore.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/270532

[Brian Haley (brian-haley)]

BTW, this isn't only an l3-agent phenomenon, saw this in an ovs agent log file:

2016-01-18 09:12:55.382 WARNING neutron.agent.linux.iptables_manager [req-22b0e72f-04e2-489d-876f-8d58c7c4bffb None None] Duplicate iptables rule detected. This may indicate a bug in the the iptables rule generation code. Line: -A neutron-openvswi-od288dd2f-b -j RETURN

So another change will need to fix that.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Makes sense.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270532
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=530c1d30875c0ca2936303ed6e396d83018aaed6
Submitter: Jenkins
Branch: master

commit 530c1d30875c0ca2936303ed6e396d83018aaed6
Author: Swaminathan Vasudevan <email address hidden>
Date: Wed Jan 20 13:35:52 2016 -0800

    DVR: Fix Duplicate IPtables rule detected warning message in l3agent

    Duplicate IPtables rule detected warning message is seen in the
    l3 agent logs for sometime.

    This will be seen when multiple floatingips are created on
    the same node for different routers or when a floatingip
    is disassociated and re-associated to a fixed-ip on the same node.

    The fip namespace is retained in the compute node even though
    the floatingip is disassociated, but when we try to re-associate
    or create a new floatingip the code in l3agent is trying to check,
    if this is the 'first' floatingip and if so tries to re-create the
    floatingip namespace and the rules within it.

    This happens because we are unsubscribing the fip namespace count
    for every associated routers that we are deleting.

    This duplicate call to create the fip namespace should be restricted
    if there is already a fip namespace in the compute node and the fip
    namespace should be unsubscribed only when the external network is
    removed before the actual fip namespace is deleted.

    The change proposed in this fix, will only unsubscribe the fip
    namespace before it is deleted.

    Closes-Bug: #1535928

    Change-Id: I24016382091cad485f65e7753972f4b71702ff9f

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b3 development milestone.