Duplicate IPtables rule detected warning message seen in L3 agent

Bug #1535928 reported by Swaminathan Vasudevan on 2016-01-19
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
Low
Swaminathan Vasudevan

Bug Description

In recent L3 agent logs in the gate we have been seeing this warning message associated with the DVR router jobs.
Right now none of the jobs are failing, but we need to see why this warning message is showing up in the logs or it might be due to some hidden issues.

http://logs.openstack.org/89/255989/11/check/gate-tempest-dsvm-neutron-dvr/e3464a5/logs/screen-q-l3.txt.gz?level=WARNING#_2016-01-18_13_34_52_764

I see that there is already a patch to address this issue. But it seems that this issue was also seen on normal neutron jobs and not just associated with dvr routers.

https://review.openstack.org/#/c/255484/1

summary: - Duplicate IPtables rule detected warning message seen in L3 agent for
- DVR Routers
+ Duplicate IPtables rule detected warning message seen in L3 agent
tags: added: neutron
tags: removed: neutron
Brian Haley (brian-haley) wrote :

This is actually pretty easy to reproduce on a DVR setup:

- associate a floating IP
- dis-associated it
- re-associate it

Since the fip namespace doesn't get destroyed on the dis-associate, the next associated will call the namespace creation code, which will add the rule again.

I'll look into it some more.

Brian I think I found out the problem
fip_ns.create() is being called everytime when there is a fip associated, even though there is a fip namespace.

So the create should not be called for every fip association if the fip namespace exists.

Cedric Brandily (cbrandily) wrote :

Is it really a trouble to raise such warning? As the warning also solves the duplication trouble

Changed in neutron:
status: New → Confirmed
Changed in neutron:
assignee: nobody → Swaminathan Vasudevan (swaminathan-vasudevan)
Brian Haley (brian-haley) wrote :

I think the warning just shows the agent could be more efficient and not do unnecessary work, in this case it is an iptables-save/restore.

Fix proposed to branch: master
Review: https://review.openstack.org/270532

Changed in neutron:
status: Confirmed → In Progress
Changed in neutron:
importance: Undecided → Low
Brian Haley (brian-haley) wrote :

BTW, this isn't only an l3-agent phenomenon, saw this in an ovs agent log file:

2016-01-18 09:12:55.382 WARNING neutron.agent.linux.iptables_manager [req-22b0e72f-04e2-489d-876f-8d58c7c4bffb None None] Duplicate iptables rule detected. This may indicate a bug in the the iptables rule generation code. Line: -A neutron-openvswi-od288dd2f-b -j RETURN

So another change will need to fix that.

Makes sense.

Reviewed: https://review.openstack.org/270532
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=530c1d30875c0ca2936303ed6e396d83018aaed6
Submitter: Jenkins
Branch: master

commit 530c1d30875c0ca2936303ed6e396d83018aaed6
Author: Swaminathan Vasudevan <email address hidden>
Date: Wed Jan 20 13:35:52 2016 -0800

    DVR: Fix Duplicate IPtables rule detected warning message in l3agent

    Duplicate IPtables rule detected warning message is seen in the
    l3 agent logs for sometime.

    This will be seen when multiple floatingips are created on
    the same node for different routers or when a floatingip
    is disassociated and re-associated to a fixed-ip on the same node.

    The fip namespace is retained in the compute node even though
    the floatingip is disassociated, but when we try to re-associate
    or create a new floatingip the code in l3agent is trying to check,
    if this is the 'first' floatingip and if so tries to re-create the
    floatingip namespace and the rules within it.

    This happens because we are unsubscribing the fip namespace count
    for every associated routers that we are deleting.

    This duplicate call to create the fip namespace should be restricted
    if there is already a fip namespace in the compute node and the fip
    namespace should be unsubscribed only when the external network is
    removed before the actual fip namespace is deleted.

    The change proposed in this fix, will only unsubscribe the fip
    namespace before it is deleted.

    Closes-Bug: #1535928

    Change-Id: I24016382091cad485f65e7753972f4b71702ff9f

Changed in neutron:
status: In Progress → Fix Released

This issue was fixed in the openstack/neutron 8.0.0.0b3 development milestone.

tags: added: liberty-backport-potential
tags: removed: liberty-backport-potential
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers