watch_log_file = true badness

[Kevin Fox]

We had:
watch_log_file = true

Set on our neutron agents. We have DVR enabled and were seeing significant slowdowns on fip changes. Kevin Benton and I narrowed it down to watch_log_file = true. It seems to be using blocking calls that prevent eventlet from performing asyncio properly. Setting it to false immediately resolved the issue.

Can:
 1. The bug be fixed
 2. The gate checks be updated to set watch_log_file = true by default so that its tested in the future?

Please?

Further details:
Mitaka RDO on CentOS 7.2. DVR enabled, L3HA disabled.

Thanks,
Kevin

[Assaf Muller (amuller)]

I'm curious, was this enabled by a deployment tool or did you manually change it to True?

[Kevin Fox (kevpn)]

I think our team set it so that ELK would work more smoothly with the logs. We didn't notice the change caused issues until much later though.

[Brian Haley (brian-haley)]

So is this really a problem in Neutron, or more specifically DVR? It seems DVR is operating properly, and LOG.debug() is a pretty normal operation.

Perhaps this needs to be re-assigned to a different component, or at least have the l3-dvr-backlog tag removed since it probably affects everything, it's just exacerbated on DVR compute nodes.

[Cedric Brandily (cbrandily)]

This bug is an oslo.log bug not a neutron one as watch_log_file=True on linux platforms use oslo_log.watchers.FastWatchedFileHandler which depends on select.poll which is not eventlet-friendly