yes,you get the setep to recreate this problem。 i try to solve this problem,but it's difficult to me .i will git commit the modification to gerrit about this bug ,please give some advice
At 2016-10-12 22:55:36, "Swaminathan Vasudevan" <email address hidden> wrote:
>Do you mean you can recreate this problem just by have a DVR+HA+L3 enabled and then kill the l3 agent and create a router while the L3 agent is down and restart the l3 agent.
>Then we should be able to see this problem. Is this right steps to reproduce?
>
>--
>You received this bug notification because you are subscribed to the bug
>report.
>https://bugs.launchpad.net/bugs/1632540
>
>Title:
> l3-agent print the ERROR log in l3 log file continuously ,finally fill
> file space,leading to crash the l3-agent service
>
>Status in neutron:
> New
>
>Bug description:
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent [req-5d499217-05b6-4a56-a3b7-5681adb53d6c - d2b95803757641b6bc55f6309c12c6e9 - - -] Failed to process compatible router 'da82aeb4-07a4-45ca-ae7a-570aec69df29'
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent Traceback (most recent call last):
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 438, in _process_router_if_compatible
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self._process_added_router(router)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 446, in _process_added_router
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent ri.process(self)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 488, in process
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process(agent)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py", line 30, in process
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process(agent)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 386, in process
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(HaRouter, self).process(agent)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 385, in call
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.logger(e)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.force_reraise()
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 382, in call
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 964, in process
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.process_address_scope()
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_router.py", line 239, in process_address_scope
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.snat_iptables_manager, ports_scopemark)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.gen.next()
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 461, in defer_apply
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent raise n_exc.IpTablesApplyException(msg)
> 2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent IpTablesApplyException: Failure applying iptables rules
>
> for example,this ERROR information will fill l3-agent log file
> continuously until solving the problem ,it will fill the log file
> space.
>
> because we resyc the ERROR update into the queue when the update is
> not been handle successfully.then the greenthread in l3-agent will
> deal with the update periodicly,so print the log periodicly, but the
> l3 agent has been deal with this update,wo should delete this update.
>
> we could disable l3-agent in a networknode in ha model, then create
> router,then restart the l3-agent ,we will see this problem。
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/neutron/+bug/1632540/+subscriptions
yes,you get the setep to recreate this problem。 i try to solve this problem,but it's difficult to me .i will git commit the modification to gerrit about this bug ,please give some advice
At 2016-10-12 22:55:36, "Swaminathan Vasudevan" <email address hidden> wrote: /bugs.launchpad .net/bugs/ 1632540 agent.l3. agent [req-5d499217- 05b6-4a56- a3b7-5681adb53d 6c - d2b95803757641b 6bc55f6309c12c6 e9 - - -] Failed to process compatible router 'da82aeb4- 07a4-45ca- ae7a-570aec69df 29' agent.l3. agent Traceback (most recent call last): agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ agent.py" , line 501, in _process_ router_ update agent.l3. agent self._process_ router_ if_compatible( router) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ agent.py" , line 438, in _process_ router_ if_compatible agent.l3. agent self._process_ added_router( router) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ agent.py" , line 446, in _process_ added_router agent.l3. agent ri.process(self) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ dvr_local_ router. py", line 488, in process agent.l3. agent super(DvrLocalR outer, self).process( agent) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ dvr_router_ base.py" , line 30, in process agent.l3. agent super(DvrRouter Base, self).process( agent) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ ha_router. py", line 386, in process agent.l3. agent super(HaRouter, self).process( agent) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ common/ utils.py" , line 385, in call agent.l3. agent self.logger(e) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ oslo_utils/ excutils. py", line 220, in __exit__ agent.l3. agent self.force_ reraise( ) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ oslo_utils/ excutils. py", line 196, in force_reraise agent.l3. agent six.reraise( self.type_ , self.value, self.tb) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ common/ utils.py" , line 382, in call agent.l3. agent return func(*args, **kwargs) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ router_ info.py" , line 964, in process agent.l3. agent self.process_ address_ scope() agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ dvr_edge_ router. py", line 239, in process_ address_ scope agent.l3. agent self.snat_ iptables_ manager, ports_scopemark) agent.l3. agent File "/usr/lib64/ python2. 7/contextlib. py", line 24, in __exit__ agent.l3. agent self.gen.next() agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/linux/ iptables_ manager. py", line 461, in defer_apply agent.l3. agent raise n_exc.IpTablesA pplyException( msg) agent.l3. agent IpTablesApplyEx ception: Failure applying iptables rules /bugs.launchpad .net/neutron/ +bug/1632540/ +subscriptions
>Do you mean you can recreate this problem just by have a DVR+HA+L3 enabled and then kill the l3 agent and create a router while the L3 agent is down and restart the l3 agent.
>Then we should be able to see this problem. Is this right steps to reproduce?
>
>--
>You received this bug notification because you are subscribed to the bug
>report.
>https:/
>
>Title:
> l3-agent print the ERROR log in l3 log file continuously ,finally fill
> file space,leading to crash the l3-agent service
>
>Status in neutron:
> New
>
>Bug description:
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
> 2016-10-12 10:04:38.587 25667 ERROR neutron.
>
> for example,this ERROR information will fill l3-agent log file
> continuously until solving the problem ,it will fill the log file
> space.
>
> because we resyc the ERROR update into the queue when the update is
> not been handle successfully.then the greenthread in l3-agent will
> deal with the update periodicly,so print the log periodicly, but the
> l3 agent has been deal with this update,wo should delete this update.
>
> we could disable l3-agent in a networknode in ha model, then create
> router,then restart the l3-agent ,we will see this problem。
>
>To manage notifications about this bug go to:
>https:/