networking-odl-functional-oxygen fails with neutron.common.utils.WaitTimeout: Timed out after 5 seconds

[Lajos Katona]

networking-odl-functional-oxygen randomly fails (there are cases when just a few tests and cases when half of the suit fails) with the following exception:
...
  File "networking_odl/tests/functional/base.py", line 117, in get_odl_resource
    utils.wait_until_true(no_journal_rows, 5, 0.5)
  File "/home/zuul/src/opendev.org/openstack/networking-odl/.tox/functional/local/lib/python2.7/site-packages/neutron/common/utils.py", line 702, in wait_until_true
    raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 5 seconds

Example:
http://logs.openstack.org/04/668904/1/check/networking-odl-functional-oxygen/06e6765/testr_results.html.gz

Logstash query:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22utils.wait_until_true%5C%5C(no_journal_rows%2C%205%2C%200.5%5C%22

[Lajos Katona (lajos-katona)]

An interesting log entry:
http://logs.openstack.org/04/668904/1/check/networking-odl-functional-oxygen/06e6765/controller/logs/functional-logs/networking_odl.tests.functional.test_odl_dhcp_driver.TestOdlDhcpDriver.test_subnet_update_from_disable_to_enable.txt.gz

By this the connection towards ODL is timed out.

2019-07-04 15:23:57.809 31587 ERROR networking_odl.common.client [req-11836557-737d-4a05-8659-1905c97a68bc - - - - -] REST request ( post ) to url ( security-groups ) is failed. Request body : [{u'security_group': {'project_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'name': u'default', 'security_group_rules': [{'project_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'port_range_max': None, 'remote_group_id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'port_range_min': None, 'protocol': None, 'direction': u'ingress', 'security_group_id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'tenant_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'remote_ip_prefix': None, 'id': 'a6cd8e0e-d2fe-4c24-b81f-2693529d41ea', 'ethertype': u'IPv4'}, {'project_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'port_range_max': None, 'remote_group_id': None, 'port_range_min': None, 'protocol': None, 'direction': u'egress', 'security_group_id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'tenant_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'remote_ip_prefix': None, 'id': '2e859b19-cbdf-4232-961b-8451de8253dc', 'ethertype': u'IPv4'}, {'project_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'port_range_max': None, 'remote_group_id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'port_range_min': None, 'protocol': None, 'direction': u'ingress', 'security_group_id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'tenant_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'remote_ip_prefix': None, 'id': '0e4582ba-75a1-4dae-bff8-f6de8ad5c413', 'ethertype': u'IPv6'}, {'project_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'port_range_max': None, 'remote_group_id': None, 'port_range_min': None, 'protocol': None, 'direction': u'egress', 'security_group_id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'tenant_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'remote_ip_prefix': None, 'id': '059a1481-8085-47a0-9cca-98086a4468ab', 'ethertype': u'IPv6'}], 'description': u'Default security group', 'id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'tenant_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b'}}] service: ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=8181): Read timed out. (read timeout=10)
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal [req-11836557-737d-4a05-8659-1905c97a68bc - - - - -] Error while processing (Entry ID: 1) - create security_group 3483f322-c3c1-4d76-ae55-8e4ca7018c93 (Time stamp: 63697850637.8): ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=8181): Read timed out. (read timeout=10)
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal Traceback (most recent call last):
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal File "networking_odl/journal/journal.py", line 280, in _sync_entry
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal self.client.sendjson(method, urlpath, to_send)
2019-07-04 15:23:57.810 315...