bulk creation of security group rules fails StaleDataError

Bug #1805132 reported by Jesper Schmitz Mouridsen on 2018-11-26
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Low
Unassigned

Bug Description

Concurrently creating (many) security group rules in a security group, either posting concurrently directly to the api, or by using heat fails (on queens) with:

Retry wrapper got retriable exception: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. wrapped /usr/lib/python2.7/site-packages/neutron/db/api.py:126
2018-11-26 12:01:51.917 4605 ERROR oslo_db.api [req-459cb962-2cc8-403a-af68-411efb7eb39c 2fe7c01cdd724059bb17e4cf38bc44b5 d45bb2772ddd4eda8f9bec432238eda1 - default default] DB exceeded retry limit.: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.

StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.

I've written a small script to reproduce this.

https://gist.github.com/jsm222/2b7db15eaebe69111793359c9a585823

Example usages:
python secruletest.py concurrent --auth-url http://127.0.0.1:5000/v3 --username demo --project-name demo --domain-id default --rules-amount 200
or
python secruletest.py heat --rules-amount 200 --file-out /tmp/heat.yaml

The heat template is just written not applied, the first example makes a security group named NeutronSecurityGroupRuleTest (can be set with --security-group-name ) and begins posting rules concurrently.

It is reproducible on rdo packstack queens, and also fails on production environment (Ubuntu based queens)

Bence Romsics (bence-romsics) wrote :

Thank you for your bug report.

As a first step it seems I managed to reproduce the error by running your script in 'concurrent' mode. So I'm setting the status to Confirmed.

 $ sudo journalctl -u devstack@neutron-api --since '10 minutes ago' | egrep 'Retry wrapper got retriable exception' | wc -l
 0
 $python ~/bug-1805132.py concurrent --auth-url http://100.109.0.51/identity/v3 --username demo --project-name demo --domain-id default --rules-amount 50
 # creates 50 secgroup rules
 $ sudo journalctl -u devstack@neutron-api --since '10 minutes ago' | egrep 'Retry wrapper got retriable exception' | wc -l
 169

Changed in neutron:
status: New → Confirmed
importance: Undecided → Low
Bence Romsics (bence-romsics) wrote :

There are multiple previous bug reports of 'Retry wrapper got retriable exception' on the table 'standardattributes':

https://bugs.launchpad.net/neutron/+bug/1738337
https://bugs.launchpad.net/neutron/+bug/1716321

But so far I could not verify if this is a duplicate of one of them or not.

Ryan Tidwell (ryan-tidwell) wrote :

I have not been able to confirm that the issues listed above are all duplicates, but these all fail in a very similar way. It does make me wonder whether there is an underlying issue related to the handling updates to standard attributes that affects operations on all resources like networks, subnets, and ports.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers