l3 agent print the ERROR log in l3 log file continuously ,finally fill file space,leading to crash the l3-agent service

[zhichao zhu]

2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent [req-5d499217-05b6-4a56-a3b7-5681adb53d6c - d2b95803757641b6bc55f6309c12c6e9 - - -] Failed to process compatible router 'da82aeb4-07a4-45ca-ae7a-570aec69df29'
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 438, in _process_router_if_compatible
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self._process_added_router(router)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 446, in _process_added_router
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent ri.process(self)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 488, in process
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process(agent)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py", line 30, in process
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process(agent)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 386, in process
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent super(HaRouter, self).process(agent)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 385, in call
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.logger(e)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.force_reraise()
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 382, in call
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 964, in process
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.process_address_scope()
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_router.py", line 239, in process_address_scope
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.snat_iptables_manager, ports_scopemark)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent self.gen.next()
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 461, in defer_apply
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent raise n_exc.IpTablesApplyException(msg)
2016-10-12 10:04:38.587 25667 ERROR neutron.agent.l3.agent IpTablesApplyException: Failure applying iptables rules

this ERROR information will fill l3-agent log file continuously until solving the problem ,it will fill the file space.

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

What actions lead to that failure ?

[Tristan Cacqueray (tristan-cacqueray)]

@zhichao zhu, can you provides details and steps to reproduce that bug report?

[Tristan Cacqueray (tristan-cacqueray)]

Adding ossg-coresec to check if this needs to be kept private. Since this doesn't seem to describe a practical vulnerability, if there are no objections, I'd like to open this bug and remove the OSSA task.

[Luke Hinds (lhinds)]

It would be good to know conditions that created the error, with some steps on how to recreate.

Also what sort of frequency does the stack trace occur at? If its happening every 0.5 seconds, its a lot more of a concern then it happening every 60 seconds. A fix is needed either way, but it increases the security impact.

Can the error be invoked with external input is another key factor.

Definitely we need more info if possible, as its just a single stack trace at the moment.

[Luke Hinds (lhinds)]

@Tristan, sorry, I did not answer your question. As it stands this looks like more of a bug for me, but its difficult to really make a call as its too light on details.

[Jeremy Stanley (fungi)]

Since this sounds like something likely mitigated by simple disk utilization monitoring and attentive operations staff, and because the original reporter is not responding to requests for additional information, I propose switching the report to public if nobody objects by Friday of this week (for broader visibility and an increased chance someone else will come along with more specific reproduction scenarios).

[Tristan Cacqueray (tristan-cacqueray)]

Switched to public security based on above comments.

[Jeremy Stanley (fungi)]

"Denial of service" conditions arising from unconstrained resource consumption by authenticated users is a grey area we struggle with classifying (and we don't even have confirmation yet that it _can_ be triggered intentionally by mere users of the environment). At some point, operators must have a means of identifying abuse by their users, locking them out and cleaning up the mess. In a "typical" production deployment servicing potentially risky users, how quickly can an abuser "fill up" your logs doing this? Will your monitoring system alert operations to the increase in activity and disk utilization in reasonable time for them to take mitigating action? Are deployments likely to include rate-limiting proxies which further throttle problem API calls such as these?

In most cases, we triage such reports as security hardening opportunities (class D in our taxonomy: https://security.openstack.org/vmt-process.html#incident-report-taxonomy ) and since this report is already public there's no harm in doing that for now while entertaining further discussion on whether it should be reclassed and any potential advisory issued.

[Brian Haley (brian-haley)]

Without more info I'm not sure what we can do to reduce these messages, or if there was just an underlying bug in the l3-agent that triggered this. Since it's been a number of years and we haven't this this issue reported by others, I am going to close it, but please re-open if you have more data on how to reproduce it.