Comment 5 for bug 1536003

I am observing ovn controller pegging at 100% even when no new resources are being actively created. Once the existing ports reach 500, ovn controller spends most of the time in lflow_run routine.

The perf profile taken during the test (creating resources) and after test completion without deleting the resources are mostly identical. Ovn controller utilization only comes down as we start deleting resources.

 Perf during the test:
#
# Samples: 120K of event 'cycles'
# Event count (approx.): 59579583762
#
# Overhead Samples Command Shared Object Symbol
# ........ ............ .............. ............................. ............................................................
#
    24.33% 29205 ovn-controller /usr/local/bin/ovn-controller 0x66959 B [.] bitwise_rscan
    13.95% 16737 ovn-controller /usr/local/bin/ovn-controller 0xb0ad B [.] create_patch_port.isra.2
    10.25% 12302 ovn-controller /usr/local/bin/ovn-controller 0x5c9f8 B [.] smap_find__
     6.84% 8203 ovn-controller /usr/lib64/libc-2.17.so 0x133d50 B [.] __strncmp_sse42
     5.23% 6277 ovn-controller /usr/lib64/libc-2.17.so 0x7e29a B [.] _int_malloc
     2.91% 3496 ovn-controller /usr/local/bin/ovn-controller 0x1a8cf B [.] hash_bytes
     2.33% 2794 ovn-controller /usr/local/bin/ovn-controller 0x8b4c0 B [.] flow_wildcards_hash
     2.32% 2787 ovn-controller /usr/local/bin/ovn-controller 0x1f257 B [.] match_hash
     2.26% 2717 ovn-controller /usr/lib64/libc-2.17.so 0x15d250 B [.] __memcmp_sse4_1
     2.20% 2641 ovn-controller /usr/local/bin/ovn-controller 0xa69f B [.] ofctrl_add_flow
     2.18% 2615 ovn-controller /usr/lib64/libc-2.17.so 0x7c92b B [.] _int_free
     1.51% 1810 ovn-controller /usr/local/bin/ovn-controller 0x5cf42 B [.] smap_get_node
     1.35% 1618 ovn-controller /usr/local/bin/ovn-controller 0x14ae4 B [.] lex_token_parse
     1.11% 1329 ovn-controller /usr/lib64/libc-2.17.so 0x13200e B [.] __strcmp_sse42

Perf after the test, system is mostly idle except for oven controller using one hw thread:

# To display the perf.data header info, please use --header/--header-only options.
#
# Samples: 111K of event 'cycles'
# Event count (approx.): 55343064284
#
# Overhead Samples Command Shared Object Symbol
# ........ ............ .............. ............................. ..........................................................
#
    29.73% 33139 ovn-controller /usr/local/bin/ovn-controller 0x66959 B [.] bitwise_rscan
     9.39% 10468 ovn-controller /usr/local/bin/ovn-controller 0xb0af B [.] create_patch_port.isra.2
     7.16% 7987 ovn-controller /usr/local/bin/ovn-controller 0x5c9eb B [.] smap_find__
     6.26% 6983 ovn-controller /usr/lib64/libc-2.17.so 0x7dff8 B [.] _int_malloc
     4.82% 5379 ovn-controller /usr/lib64/libc-2.17.so 0x132de8 B [.] __strncmp_sse42
     2.81% 3131 ovn-controller /usr/local/bin/ovn-controller 0x8b493 B [.] flow_wildcards_hash
     2.76% 3080 ovn-controller /usr/local/bin/ovn-controller 0x1f26e B [.] match_hash
     2.48% 2763 ovn-controller /usr/lib64/libc-2.17.so 0x15d275 B [.] __memcmp_sse4_1
     2.44% 2725 ovn-controller /usr/local/bin/ovn-controller 0xa69f B [.] ofctrl_add_flow
     2.44% 2725 ovn-controller /usr/lib64/libc-2.17.so 0x7c982 B [.] _int_free
     2.24% 2501 ovn-controller /usr/local/bin/ovn-controller 0x1a8fd B [.] hash_bytes
     1.57% 1755 ovn-controller /usr/local/bin/ovn-controller 0x14ad0 B [.] lex_token_parse
     1.14% 1270 ovn-controller /usr/lib64/libc-2.17.so 0x14a0ce B [.] __memcpy_ssse3_back
     1.00% 1115 ovn-controller /usr/local/bin/ovn-controller 0x5cf42 B [.] smap_get_node

I am new to this code, perhaps I am not reading this correctly. but It appears to me that it is going through patch_run, building existing ports, checking db irrespective of any port updates.