Comment 4 for bug 1632540

Revision history for this message
zhichao zhu (rtmdk) wrote : Re:[Bug 1632540] Re: l3-agent print the ERROR log in l3 log file continuously , finally fill file space, leading to crash the l3-agent service

yes,you get the setep to recreate this problem。 i try to solve this problem,but it's difficult to me .i will git commit the modification to gerrit about this bug ,please give some advice

At 2016-10-12 22:55:36, "Swaminathan Vasudevan" <email address hidden> wrote:
>Do you mean you can recreate this problem just by have a DVR+HA+L3 enabled and then kill the l3 agent and create a router while the L3 agent is down and restart the l3 agent.
>Then we should be able to see this problem. Is this right steps to reproduce?
>
>--
>You received this bug notification because you are subscribed to the bug
>report.
>https://bugs.launchpad.net/bugs/1632540
>
>Title:
> l3-agent print the ERROR log in l3 log file continuously ,finally fill
> file space,leading to crash the l3-agent service
>
>Status in neutron:
> New
>
>Bug description:
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent [req-5d499217-05b6-4a56-a3b7-5681adb53d6c - d2b95803757641b6bc55f6309c12c6e9 - - -] Failed to process compatible router 'da82aeb4-07a4-45ca-ae7a-570aec69df29'
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent Traceback (most recent call last):
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 438, in _process_router_if_compatible
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self._process_added_router(router)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 446, in _process_added_router
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent ri.process(self)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 488, in process
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process(agent)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py", line 30, in process
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process(agent)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 386, in process
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(HaRouter, self).process(agent)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 385, in call
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.logger(e)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.force_reraise()
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 382, in call
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 964, in process
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.process_address_scope()
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_router.py", line 239, in process_address_scope
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.snat_iptables_manager, ports_scopemark)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.gen.next()
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 461, in defer_apply
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent raise n_exc.IpTablesApplyException(msg)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent IpTablesApplyException: Failure applying iptables rules
>
> for example,this ERROR information will fill l3-agent log file
> continuously until solving the problem ,it will fill the log file
> space.
>
> because we resyc the ERROR update into the queue when the update is
> not been handle successfully.then the greenthread in l3-agent will
> deal with the update periodicly,so print the log periodicly, but the
> l3 agent has been deal with this update,wo should delete this update.
>
> we could disable l3-agent in a networknode in ha model, then create
> router,then restart the l3-agent ,we will see this problem。
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/neutron/+bug/1632540/+subscriptions