With hardware offloading enabled, OVS logs are spammed with netdev_offload_tc ERR messages

[Nikolay Vinogradov]

If neutron-openvswitch charm is deployed with enable-hardware-offload=true ovs-vswitchd.log are flooded with messages like those:

2020-09-27T21:55:40.460Z|28192|netdev_offload_tc(revalidator625)|ERR|Dropped 11 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate
2020-09-27T21:55:40.460Z|28193|netdev_offload_tc(revalidator625)|ERR|dump_create: failed to get ifindex for qr-66e28056-8f: Operation not supported
2020-09-27T21:55:41.461Z|28194|netdev_offload_tc(revalidator625)|ERR|Dropped 11 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate
2020-09-27T21:55:41.461Z|28195|netdev_offload_tc(revalidator625)|ERR|dump_create: failed to get ifindex for qr-66e28056-8f: Operation not supported
2020-09-27T21:55:42.461Z|28196|netdev_offload_tc(revalidator625)|ERR|Dropped 11 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate
2020-09-27T21:55:42.461Z|28197|netdev_offload_tc(revalidator625)|ERR|dump_create: failed to get ifindex for qr-66e28056-8f: Operation not supported
2020-09-27T21:55:43.462Z|28198|netdev_offload_tc(revalidator625)|ERR|Dropped 11 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate
2020-09-27T21:55:43.462Z|28199|netdev_offload_tc(revalidator625)|ERR|dump_create: failed to get ifindex for qr-66e28056-8f: Operation not supported
2020-09-27T21:55:44.464Z|28200|netdev_offload_tc(revalidator625)|ERR|Dropped 11 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate
2020-09-27T21:55:44.464Z|28201|netdev_offload_tc(revalidator625)|ERR|dump_create: failed to get ifindex for qr-66e28056-8f: Operation not supported

The messages are possibly harmful, but there are thousands of them per hour. They can be disabled by executing:

sudo ovs-appctl vlog/set netdev_offload_tc:OFF

[James Page (james-page)]

Moved to openvswitch package as this is not a charm issue.

[Trent Lloyd (lathiat)]

There is an indication in the below RHBZ this can actually prevent openvswitch from working properly as it loses too much CPU time to this processing in large environments (100s or 1000s of ports)

https://bugzilla.redhat.com/show_bug.cgi?id=1737982

Seems to be a rejected upstream patch here, unclear if one was later accepted, we shoudl check for it:
https://lists.linuxfoundation.org/pipermail/ovs-dev/2019-March/357348.html

And potentially prioritise a fix for this.

[James Page (james-page)]

Triaging as enough there to confirm this is a bug and Medium as there is a workaround.

[James Page (james-page)]

This was fixed in the 2.13 branch under commit 1028cb71d0 which makes the log level for these messages a lower priority.

[James Page (james-page)]

git tag --contains 1028cb71d0

v2.12.0
v2.12.1
v2.12.2
v2.12.3
v2.13.0
v2.13.1
v2.13.2
v2.13.3
v2.14.0
v2.14.1
v2.14.2
v2.15.0

[Erlon R. Cruz (sombrafam)]

Hi James,

The fix mentioned above fixes the issue for the 'dpif_netlink' module. The problem mentioned
here is for the netdev_offload_tc. The fix should be basically the same though.

Erlon