bulk creation of security group rules fails StaleDataError

[Jesper Schmitz Mouridsen]

Concurrently creating (many) security group rules in a security group, either posting concurrently directly to the api, or by using heat fails (on queens) with:

Retry wrapper got retriable exception: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. wrapped /usr/lib/python2.7/site-packages/neutron/db/api.py:126
2018-11-26 12:01:51.917 4605 ERROR oslo_db.api [req-459cb962-2cc8-403a-af68-411efb7eb39c 2fe7c01cdd724059bb17e4cf38bc44b5 d45bb2772ddd4eda8f9bec432238eda1 - default default] DB exceeded retry limit.: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.

StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.

I've written a small script to reproduce this.

https://gist.github.com/jsm222/2b7db15eaebe69111793359c9a585823

Example usages:
python secruletest.py concurrent --auth-url http://127.0.0.1:5000/v3 --username demo --project-name demo --domain-id default --rules-amount 200
or
python secruletest.py heat --rules-amount 200 --file-out /tmp/heat.yaml

The heat template is just written not applied, the first example makes a security group named NeutronSecurityGroupRuleTest (can be set with --security-group-name ) and begins posting rules concurrently.

It is reproducible on rdo packstack queens, and also fails on production environment (Ubuntu based queens)

[Bence Romsics (bence-romsics)]

Thank you for your bug report.

As a first step it seems I managed to reproduce the error by running your script in 'concurrent' mode. So I'm setting the status to Confirmed.

 $ sudo journalctl -u devstack@neutron-api --since '10 minutes ago' | egrep 'Retry wrapper got retriable exception' | wc -l
 0
 $python ~/bug-1805132.py concurrent --auth-url http://100.109.0.51/identity/v3 --username demo --project-name demo --domain-id default --rules-amount 50
 # creates 50 secgroup rules
 $ sudo journalctl -u devstack@neutron-api --since '10 minutes ago' | egrep 'Retry wrapper got retriable exception' | wc -l
 169

[Bence Romsics (bence-romsics)]

There are multiple previous bug reports of 'Retry wrapper got retriable exception' on the table 'standardattributes':

https://bugs.launchpad.net/neutron/+bug/1738337
https://bugs.launchpad.net/neutron/+bug/1716321

But so far I could not verify if this is a duplicate of one of them or not.

[Ryan Tidwell (ryan-tidwell)]

I have not been able to confirm that the issues listed above are all duplicates, but these all fail in a very similar way. It does make me wonder whether there is an underlying issue related to the handling updates to standard attributes that affects operations on all resources like networks, subnets, and ports.

[Brian Haley (brian-haley)]

I ran this test multiple times on a recent devstack (Antelope) and could not reproduce it. Since it's 5 years old I'll close since there have been a number of changes in the DB code since then. Please re-open if you see it again.