| 2020-08-21 12:34:01 |
Michal Dulko |
description |
In kuryr-kubernetes gate runing NetworkPolicy tests from K8s itself we started to see Neutron returns 500 on SG delete requests. This is found in q-svc logs:
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource [None req-14b3a3ce-74d3-4b84-a458-57acf8e74c0e service kuryr] delete failed: No details.: oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`neutron`.`securitygroupportbindings`, CONSTRAINT `securitygroupportbindings_ibfk_2` FOREIGN KEY (`security_group_id`) REFERENCES `securitygroups` (`id`))')
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: [SQL: DELETE FROM securitygroups WHERE securitygroups.id = %(id)s]
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: [parameters: {'id': 'f62aa0c2-c513-4794-b48a-003a388d0ba0'}]
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: (Background on this error at: http://sqlalche.me/e/13/gkpj)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource Traceback (most recent call last):
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource cursor, statement, parameters, context
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource cursor.execute(statement, parameters)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 163, in execute
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource result = self._query(query)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 321, in _query
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource conn.query(q)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 505, in query
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource self._affected_rows = self._read_query_result(unbuffered=unbuffered)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 724, in _read_query_result
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource result.read()
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 1069, in read
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource first_packet = self.connection._read_packet()
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 676, in _read_packet
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource packet.raise_for_error()
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/protocol.py", line 223, in raise_for_error
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource err.raise_mysql_exception(self._data)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource raise errorclass(errno, errval)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource pymysql.err.IntegrityError: (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`neutron`.`securitygroupportbindings`, CONSTRAINT `securitygroupportbindings_ibfk_2` FOREIGN KEY (`security_group_id`) REFERENCES `securitygroups` (`id`))')
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource
Full log: https://508cf8e3a004102dbdaa-fe9c830c883bfe814f1bba253125965d.ssl.cf1.rackcdn.com/747233/2/check/kuryr-kubernetes-e2e-np-containerized-ovn-provider-ovn/c4fdef2/controller/logs/kubernetes/pod_logs/kube-system-kuryr-controller-5d559448c4-g5xst-controller-prev.txt
It seems like we do not run any POST request for SG rules after that DELETE is called, but honestly I wouldn't be too surprised it may happen as well. Besides that I'm not exactly sure what causes it. |
In kuryr-kubernetes gate runing NetworkPolicy tests from K8s itself we started to see Neutron returns 500 on SG delete requests. This is found in q-svc logs:
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource [None req-14b3a3ce-74d3-4b84-a458-57acf8e74c0e service kuryr] delete failed: No details.: oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`neutron`.`securitygroupportbindings`, CONSTRAINT `securitygroupportbindings_ibfk_2` FOREIGN KEY (`security_group_id`) REFERENCES `securitygroups` (`id`))')
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: [SQL: DELETE FROM securitygroups WHERE securitygroups.id = %(id)s]
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: [parameters: {'id': 'f62aa0c2-c513-4794-b48a-003a388d0ba0'}]
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: (Background on this error at: http://sqlalche.me/e/13/gkpj)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource Traceback (most recent call last):
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource cursor, statement, parameters, context
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource cursor.execute(statement, parameters)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 163, in execute
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource result = self._query(query)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 321, in _query
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource conn.query(q)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 505, in query
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource self._affected_rows = self._read_query_result(unbuffered=unbuffered)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 724, in _read_query_result
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource result.read()
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 1069, in read
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource first_packet = self.connection._read_packet()
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 676, in _read_packet
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource packet.raise_for_error()
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/protocol.py", line 223, in raise_for_error
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource err.raise_mysql_exception(self._data)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource raise errorclass(errno, errval)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource pymysql.err.IntegrityError: (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`neutron`.`securitygroupportbindings`, CONSTRAINT `securitygroupportbindings_ibfk_2` FOREIGN KEY (`security_group_id`) REFERENCES `securitygroups` (`id`))')
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource
Full log: https://508cf8e3a004102dbdaa-fe9c830c883bfe814f1bba253125965d.ssl.cf1.rackcdn.com/747233/2/check/kuryr-kubernetes-e2e-np-containerized-ovn-provider-ovn/c4fdef2/controller/logs/screen-q-svc.txt
It seems like we do not run any POST request for SG rules after that DELETE is called, but honestly I wouldn't be too surprised it may happen as well. Besides that I'm not exactly sure what causes it. |
|
