watch_log_file = true badness

Bug #1583266 reported by Kevin Fox on 2016-05-18
This bug report is a duplicate of:  Bug #1583270: watch-file=True causes blocking. Edit Remove
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Undecided
Unassigned
oslo.log
Undecided
Unassigned

Bug Description

We had:
watch_log_file = true

Set on our neutron agents. We have DVR enabled and were seeing significant slowdowns on fip changes. Kevin Benton and I narrowed it down to watch_log_file = true. It seems to be using blocking calls that prevent eventlet from performing asyncio properly. Setting it to false immediately resolved the issue.

Can:
 1. The bug be fixed
 2. The gate checks be updated to set watch_log_file = true by default so that its tested in the future?

Please?

Further details:
Mitaka RDO on CentOS 7.2. DVR enabled, L3HA disabled.

Thanks,
Kevin

Assaf Muller (amuller) wrote :

I'm curious, was this enabled by a deployment tool or did you manually change it to True?

Kevin Fox (kevpn) wrote :

I think our team set it so that ELK would work more smoothly with the logs. We didn't notice the change caused issues until much later though.

tags: added: l3-dvr-backlog
Brian Haley (brian-haley) wrote :

So is this really a problem in Neutron, or more specifically DVR? It seems DVR is operating properly, and LOG.debug() is a pretty normal operation.

Perhaps this needs to be re-assigned to a different component, or at least have the l3-dvr-backlog tag removed since it probably affects everything, it's just exacerbated on DVR compute nodes.

Cedric Brandily (cbrandily) wrote :

This bug is an oslo.log bug not a neutron one as watch_log_file=True on linux platforms use oslo_log.watchers.FastWatchedFileHandler which depends on select.poll which is not eventlet-friendly

Changed in neutron:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers