watch_log_file = true badness

Bug #1583266 reported by Kevin Fox
This bug report is a duplicate of:  Bug #1583270: watch-file=True causes blocking. Edit Remove
This bug affects 1 person
Affects Status Importance Assigned to Milestone

Bug Description

We had:
watch_log_file = true

Set on our neutron agents. We have DVR enabled and were seeing significant slowdowns on fip changes. Kevin Benton and I narrowed it down to watch_log_file = true. It seems to be using blocking calls that prevent eventlet from performing asyncio properly. Setting it to false immediately resolved the issue.

 1. The bug be fixed
 2. The gate checks be updated to set watch_log_file = true by default so that its tested in the future?


Further details:
Mitaka RDO on CentOS 7.2. DVR enabled, L3HA disabled.


Revision history for this message
Assaf Muller (amuller) wrote :

I'm curious, was this enabled by a deployment tool or did you manually change it to True?

Revision history for this message
Kevin Fox (kevpn) wrote :

I think our team set it so that ELK would work more smoothly with the logs. We didn't notice the change caused issues until much later though.

tags: added: l3-dvr-backlog
Revision history for this message
Brian Haley (brian-haley) wrote :

So is this really a problem in Neutron, or more specifically DVR? It seems DVR is operating properly, and LOG.debug() is a pretty normal operation.

Perhaps this needs to be re-assigned to a different component, or at least have the l3-dvr-backlog tag removed since it probably affects everything, it's just exacerbated on DVR compute nodes.

Revision history for this message
Cedric Brandily (cbrandily) wrote :

This bug is an oslo.log bug not a neutron one as watch_log_file=True on linux platforms use oslo_log.watchers.FastWatchedFileHandler which depends on select.poll which is not eventlet-friendly

Changed in neutron:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers