For the sake of completeness:
we observed the problem in Stein and Xena, but the code indicates this issue exists in the subsequent versions as well.
Without being too specific, our environment spans up to several thousands agents. I don't have clear numbers about how many rules per group we have, but I would hazard a guess at at least 2.
With regard to the code-path triggering the issue:
When a security group is deleted, a notification is sent to all the agents triggering the callbacks of SecurityGroupServerAPIShim [1]. All the agents receiving the notification proceed with the SecurityGroupServerAPIShim._clear_child_sg_rules callback which itself is using
self.rcache.get_resources to look for SecurityGroupRules belonging to the deleted SecurityGroup [2].
Now, the first thing RemoteResourceCache.get_resources does is call _flood_cache_for_query [3] which looks in its own cache if it already queried neutron-rpc for these resources and if not, finally, performs the unwelcome bulk_pull.
Here's a proposed patch (I don't know why it doesn't appear in the bug's comments): https:/ /review. opendev. org/c/openstack /neutron/ +/883235
For the sake of completeness:
we observed the problem in Stein and Xena, but the code indicates this issue exists in the subsequent versions as well.
Without being too specific, our environment spans up to several thousands agents. I don't have clear numbers about how many rules per group we have, but I would hazard a guess at at least 2.
With regard to the code-path triggering the issue:
When a security group is deleted, a notification is sent to all the agents triggering the callbacks of SecurityGroupSe rverAPIShim [1]. All the agents receiving the notification proceed with the SecurityGroupSe rverAPIShim. _clear_ child_sg_ rules callback which itself is using get_resources to look for SecurityGroupRules belonging to the deleted SecurityGroup [2].
self.rcache.
Now, the first thing RemoteResourceC ache.get_ resources does is call _flood_ cache_for_ query [3] which looks in its own cache if it already queried neutron-rpc for these resources and if not, finally, performs the unwelcome bulk_pull.
[1] https:/ /github. com/openstack/ neutron/ blob/fd21c905ca 9016092d48d3f44 42bae6d4abb42e3 /neutron/ api/rpc/ handlers/ securitygroups_ rpc.py# L247 /github. com/openstack/ neutron/ blob/fd21c905ca 9016092d48d3f44 42bae6d4abb42e3 /neutron/ api/rpc/ handlers/ securitygroups_ rpc.py# L306 /github. com/openstack/ neutron/ blob/fd21c905ca 9016092d48d3f44 42bae6d4abb42e3 /neutron/ agent/resource_ cache.py# L127
[2] https:/
[3] https:/