We have a downstream failure of neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest.test_duplicate_policy_error probably because http timeout is different on RHEL based boxes, set 120 seconds. The reason why it started failing in Stein is the recent bump in the time requiring to complete the retry DB mechanism: https://review.opendev.org/#/c/583527/5 The patch increases the required time for a reply about resource already existing to more than 160 seconds.
That is because Neutron server retries for every exception about existing entry from the DB layer:
https://opendev.org/openstack/neutron-lib/src/branch/master/neutron_lib/db/api.py#L119
Steps to reproduce:
net_id=$(openstack network create rbac_net | awk '/ id /{ print $4 }')
openstack network rbac create --type network --action access_as_shared --target-project admin $net_id
openstack network rbac create --type network --action access_as_shared --target-project admin $net_id
( yes, it's the same command twice)
I don't understand which race scenario the retry mechanism for resource create it tries to solve. However, I can think of a race scenario it introduces:
$ openstack network rbac delete 8a00a24e-182a-4e5e-8694-35e66635b581
$ openstack network delete $net_id
$ net_id=$(openstack network create rbac_net | awk '/ id /{ print $4 }')
$ rbac_id=$(openstack network rbac create --type network --action access_as_shared --target-project admin $net_id | awk '/ id /{ print $4 }')
$ openstack network rbac create --type network --action access_as_shared --target-project admin $net_id &
[1] 31383
$ sleep 10
$ openstack network rbac delete $rbac_id
$ fg
openstack network rbac create --type network --action access_as_shared --target-project admin $net_id
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | access_as_shared |
| id | 13c5b655-e1b5-4c72-a8af-1ed2f9ddcf89 |
| location | Munch({'cloud': '', 'region_name': 'regionOne', 'zone': None, 'project': Munch({'id': 'cdf84b19b71249ffaffde6627d06da12', 'name': 'admin', 'domain_id': None, 'domain_name': 'Default'})}) |
| name | None |
| object_id | 618108b7-d191-4459-b18a-30b7a65be005 |
| object_type | network |
| project_id | cdf84b19b71249ffaffde6627d06da12 |
| target_project_id | cdf84b19b71249ffaffde6627d06da12 |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
The result should be that second creation of existing resource should fail and there should not exist any rbac policy. However, the second creation succeeded and there does exist the policy, that should have been deleted.
I digged into why do we retry on DBDuplicateEntry exception and found https:/
Basically the reason is that if we generate entries, it may happen two processes concurrently generate the same resource and then we need to retry on the process that caught the duplicated entry.