Too many DBDeadlockError and IP address collision during port creating
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Fix Released
|
High
|
LIU Yulong |
Bug Description
Too many DBDeadlockError and IP address collision during port creating
ENV:
Neutron stable/queens (12.0.1)
CentOS 7 (3.10.0-
This is a edge scenario testing after we meet bug:
https:/
We have 3 neutron-server node, every each one enable 8 API worker.
Try create 1000 port in a single network 2.0.0.0/16.
Exception:
IP address collision:
2018-06-20 18:57:31.121 440352 ERROR oslo_db.api NeutronDbObject
Deadlock:
2018-06-20 18:53:33.367 440348 ERROR neutron.
LOG:
IP address collision 409:
http://
Deadlock 500:
http://
REQ AND RESP:
request:
2018-06-20 18:47:26.993 440352 DEBUG neutron.api.v2.base [req-fe6df326-
HTTP 409 response:
2018-06-20 18:57:31.217 440352 INFO neutron.wsgi [req-fe6df326-
Exception count in each neutron server:
[root@147 neutron]# grep "duplicate IpamAllocation" server.log.bak4|wc -l
3651
[root@148 neutron]# grep "duplicate IpamAllocation" server.log.bak4|wc -l
3831
[root@149 neutron]# grep "duplicate IpamAllocation" server.log.bak4|wc -l
3145
We found two key point of neutron code related to such error:
In [1], the IP range is a little small, and here only return the last one IP, not a random pick, then the IP address collision chance raise.
In [2], there are fixed value max_retries=10 and retry_interval=0.1. It seems not a good practice. And this wrapper just ingore the [database] config
attribute db_retry_interval, db_max_
[1] https:/
[2] https:/
Changed in neutron: | |
importance: | Undecided → High |
Changed in neutron: | |
assignee: | LIU Yulong (dragon889) → hujin (hujin) |
Changed in neutron: | |
assignee: | hujin (hujin) → LIU Yulong (dragon889) |
tags: | added: neutron-proactive-backport-potential |
tags: | added: neutron-easy-proactive-backport-potential |
Hi Liu, are you working on a change for this?