Keystone throws DBNonExistentTable
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Identity (keystone) |
New
|
Undecided
|
Unassigned |
Bug Description
Environment:
- Keystone runs as a process inside Apache Httpd on RHEL7.7 (OpenStack Stein) with one (default) process
I am using OpenStack Keystone Stein and run into the below error randomly, where Keystone public process(listening to 5000) is running inside Apache httpd. It is seen that in some cases when the httpd service is restarted, Keystone throws the below stacktrace (and is not functional after that). This problem is resolved with a restart of httpd service. This is seen soon after httpd is restarted and does not happen all the time. My environment has MariaDB backend (SELinux is in permissive). This problem is not limited to the assignment table and is seen across all other tables in Keystone. MariaDB service is functional (and has not been restarted) and all the tables are in place.
[Fri Aug 07 08:20:59.936087 2020] [:info] [pid 1420287] mod_wsgi (pid=1420287, process=
[Fri Aug 07 08:20:59.936089 2020] [:info] [pid 1420288] mod_wsgi (pid=1420288, process=
[Fri Aug 07 08:20:59.943431 2020] [ssl:info] [pid 1420290] [client 1.2.3.95:35762] AH01964: Connection to child 1 established (server 1.2.3.50:5000)
[Fri Aug 07 08:21:00.009317 2020] [ssl:info] [pid 1420291] [client 1.2.3.113:60132] AH01964: Connection to child 2 established (server 1.2.3.50:5000)
[Fri Aug 07 08:21:01.243594 2020] [ssl:info] [pid 1420289] [client 1.2.3.50:53996] AH01964: Connection to child 0 established (server 1.2.3.50:5000)
[Fri Aug 07 08:21:01.386329 2020] [ssl:info] [pid 1420293] [client x.x.x.x:38645] AH01964: Connection to child 4 established (server 1.2.3.50:5000)
[Fri Aug 07 08:21:01.824041 2020] [ssl:info] [pid 1420349] [client 1.2.3.101:42974] AH01964: Connection to child 5 established (server 1.2.3.50:5000)
[Fri Aug 07 08:21:02.949166 2020] [ssl:info] [pid 1420378] [client 1.2.3.50:54014] AH01964: Connection to child 9 established (server 1.2.3.50:5000)
[Fri Aug 07 08:21:02.949172 2020] [ssl:info] [pid 1420379] [client 1.2.3.80:46924] AH01964: Connection to child 10 established (server 1.2.3.50:5000)
[Fri Aug 07 08:21:03.286057 2020] [:info] [pid 1420287] mod_wsgi (pid=1420287): Create interpreter '1.2.3.50:5000|'.
[Fri Aug 07 08:21:03.287286 2020] [:info] [pid 1420287] [remote 1.2.3.95:156] mod_wsgi (pid=1420287, process=
[Fri Aug 07 08:21:04.675059 2020] [ssl:info] [pid 1420436] [client 1.2.3.50:54032] AH01964: Connection to child 12 established (server 1.2.3.50:5000)
[Fri Aug 07 08:21:04.705975 2020] [ssl:info] [pid 1420437] [client 1.2.3.107:59554] AH01964: Connection to child 13 established (server 1.2.3.50:5000)
[Fri Aug 07 08:21:06.960940 2020] [ssl:info] [pid 1420438] [client 1.2.3.80:46970] AH01964: Connection to child 14 established (server 1.2.3.50:5000)
[Fri Aug 07 08:21:07.661670 2020] [ssl:info] [pid 1420349] [client 1.2.3.50:54124] AH01964: Connection to child 5 established (server 1.2.3.50:5000)
[Fri Aug 07 08:21:07.683383 2020] [ssl:info] [pid 1420292] [client x.x.x.x:30065] AH01964: Connection to child 3 established (server 1.2.3.50:5000)
[Fri Aug 07 08:21:08.442956 2020] [:error] [pid 1420287] [remote x.x.x.x:144] mod_wsgi (pid=1420287): Exception occurred processing WSGI script '/usr/bin/
[Fri Aug 07 08:21:08.443002 2020] [:error] [pid 1420287] [remote x.x.x.x:144] Traceback (most recent call last):
[Fri Aug 07 08:21:08.443017 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.443509 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.wsgi_
[Fri Aug 07 08:21:08.443525 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.443630 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.app(environ, start_response)
[Fri Aug 07 08:21:08.443644 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.443746 2020] [:error] [pid 1420287] [remote x.x.x.x:144] resp = self.call_func(req, *args, **kw)
[Fri Aug 07 08:21:08.443756 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.443773 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.func(req, *args, **kwargs)
[Fri Aug 07 08:21:08.443781 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.443844 2020] [:error] [pid 1420287] [remote x.x.x.x:144] response = req.get_
[Fri Aug 07 08:21:08.443859 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444194 2020] [:error] [pid 1420287] [remote x.x.x.x:144] application, catch_exc_
[Fri Aug 07 08:21:08.444203 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444220 2020] [:error] [pid 1420287] [remote x.x.x.x:144] app_iter = application(
[Fri Aug 07 08:21:08.444229 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444245 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return resp(environ, start_response)
[Fri Aug 07 08:21:08.444253 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444268 2020] [:error] [pid 1420287] [remote x.x.x.x:144] resp = self.call_func(req, *args, **kw)
[Fri Aug 07 08:21:08.444276 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444292 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.func(req, *args, **kwargs)
[Fri Aug 07 08:21:08.444300 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444315 2020] [:error] [pid 1420287] [remote x.x.x.x:144] response = req.get_
[Fri Aug 07 08:21:08.444323 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444338 2020] [:error] [pid 1420287] [remote x.x.x.x:144] application, catch_exc_
[Fri Aug 07 08:21:08.444346 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444361 2020] [:error] [pid 1420287] [remote x.x.x.x:144] app_iter = application(
[Fri Aug 07 08:21:08.444370 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444385 2020] [:error] [pid 1420287] [remote x.x.x.x:144] resp = self.call_func(req, *args, **kw)
[Fri Aug 07 08:21:08.444393 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444408 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.func(req, *args, **kwargs)
[Fri Aug 07 08:21:08.444416 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444476 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return request.
[Fri Aug 07 08:21:08.444485 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444501 2020] [:error] [pid 1420287] [remote x.x.x.x:144] application, catch_exc_
[Fri Aug 07 08:21:08.444509 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444524 2020] [:error] [pid 1420287] [remote x.x.x.x:144] app_iter = application(
[Fri Aug 07 08:21:08.444533 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444547 2020] [:error] [pid 1420287] [remote x.x.x.x:144] resp = self.call_func(req, *args, **kw)
[Fri Aug 07 08:21:08.444556 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444571 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.func(req, *args, **kwargs)
[Fri Aug 07 08:21:08.444587 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444636 2020] [:error] [pid 1420287] [remote x.x.x.x:144] response = req.get_
[Fri Aug 07 08:21:08.444645 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444660 2020] [:error] [pid 1420287] [remote x.x.x.x:144] application, catch_exc_
[Fri Aug 07 08:21:08.444669 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444684 2020] [:error] [pid 1420287] [remote x.x.x.x:144] app_iter = application(
[Fri Aug 07 08:21:08.444698 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444750 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.app(environ, start_response)
[Fri Aug 07 08:21:08.444759 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444774 2020] [:error] [pid 1420287] [remote x.x.x.x:144] resp = self.call_func(req, *args, **kw)
[Fri Aug 07 08:21:08.444783 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444797 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.func(req, *args, **kwargs)
[Fri Aug 07 08:21:08.444810 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444828 2020] [:error] [pid 1420287] [remote x.x.x.x:144] response = req.get_
[Fri Aug 07 08:21:08.444836 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444851 2020] [:error] [pid 1420287] [remote x.x.x.x:144] application, catch_exc_
[Fri Aug 07 08:21:08.444859 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444874 2020] [:error] [pid 1420287] [remote x.x.x.x:144] app_iter = application(
[Fri Aug 07 08:21:08.444883 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444897 2020] [:error] [pid 1420287] [remote x.x.x.x:144] resp = self.call_func(req, *args, **kw)
[Fri Aug 07 08:21:08.444906 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444920 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.func(req, *args, **kwargs)
[Fri Aug 07 08:21:08.444929 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444944 2020] [:error] [pid 1420287] [remote x.x.x.x:144] response = req.get_
[Fri Aug 07 08:21:08.444952 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444967 2020] [:error] [pid 1420287] [remote x.x.x.x:144] application, catch_exc_
[Fri Aug 07 08:21:08.444975 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.444990 2020] [:error] [pid 1420287] [remote x.x.x.x:144] app_iter = application(
[Fri Aug 07 08:21:08.444998 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445279 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return app(environ, start_response)
[Fri Aug 07 08:21:08.445288 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445304 2020] [:error] [pid 1420287] [remote x.x.x.x:144] response = self.handle_
[Fri Aug 07 08:21:08.445316 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445490 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445500 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445516 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445524 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445539 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445547 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445562 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445570 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445585 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445593 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445608 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445616 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445630 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445639 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445654 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445662 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445676 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445685 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445699 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445708 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445722 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445731 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445745 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445758 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445772 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445780 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445795 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445803 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445818 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445826 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445841 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445849 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445863 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445871 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445886 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445894 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445908 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445917 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445931 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445939 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445954 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445962 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445976 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.445984 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.445999 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446007 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446021 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446030 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446044 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446052 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446067 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446075 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446090 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446098 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446113 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446121 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446135 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446143 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446158 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446166 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446181 2020] [:error] [pid 1420287] [remote x.x.x.x:144] reraise(exc_type, exc_value, tb)
[Fri Aug 07 08:21:08.446190 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446204 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.handle_
[Fri Aug 07 08:21:08.446212 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446227 2020] [:error] [pid 1420287] [remote x.x.x.x:144] response = self.full_
[Fri Aug 07 08:21:08.446236 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446250 2020] [:error] [pid 1420287] [remote x.x.x.x:144] rv = self.handle_
[Fri Aug 07 08:21:08.446259 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446273 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446282 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446296 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446304 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446318 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446327 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446341 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446349 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446363 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446372 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446386 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446395 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446409 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446417 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446432 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446440 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446454 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446463 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446477 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446485 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446500 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446508 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446522 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446531 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446545 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446553 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446568 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446576 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446590 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446599 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446613 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446621 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446636 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446644 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446658 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446667 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446681 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446689 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446704 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446712 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446727 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446735 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446749 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446757 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446772 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446780 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446795 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446803 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446817 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446826 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446840 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446848 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446863 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446871 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446885 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446893 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446908 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446916 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446930 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return original_handler(e)
[Fri Aug 07 08:21:08.446938 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446953 2020] [:error] [pid 1420287] [remote x.x.x.x:144] reraise(exc_type, exc_value, tb)
[Fri Aug 07 08:21:08.446962 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446976 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.handle_
[Fri Aug 07 08:21:08.446984 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.446999 2020] [:error] [pid 1420287] [remote x.x.x.x:144] rv = self.dispatch_
[Fri Aug 07 08:21:08.447007 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.447022 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.view_
[Fri Aug 07 08:21:08.447031 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.447046 2020] [:error] [pid 1420287] [remote x.x.x.x:144] resp = resource(*args, **kwargs)
[Fri Aug 07 08:21:08.447055 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.447119 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self.dispatch_
[Fri Aug 07 08:21:08.447128 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.447144 2020] [:error] [pid 1420287] [remote x.x.x.x:144] resp = meth(*args, **kwargs)
[Fri Aug 07 08:21:08.447152 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.447392 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return f(*args, **kwargs)
[Fri Aug 07 08:21:08.447406 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.447551 2020] [:error] [pid 1420287] [remote x.x.x.x:144] token = authentication.
[Fri Aug 07 08:21:08.447561 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.447652 2020] [:error] [pid 1420287] [remote x.x.x.x:144] app_cred_
[Fri Aug 07 08:21:08.447662 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.447679 2020] [:error] [pid 1420287] [remote x.x.x.x:144] __ret_val = __f(*args, **kwargs)
[Fri Aug 07 08:21:08.447687 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.447706 2020] [:error] [pid 1420287] [remote x.x.x.x:144] token.mint(
[Fri Aug 07 08:21:08.447714 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.448498 2020] [:error] [pid 1420287] [remote x.x.x.x:144] self._validate_
[Fri Aug 07 08:21:08.448508 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.448525 2020] [:error] [pid 1420287] [remote x.x.x.x:144] if self.project_scoped and not self.roles:
[Fri Aug 07 08:21:08.448533 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.448549 2020] [:error] [pid 1420287] [remote x.x.x.x:144] roles = self._get_
[Fri Aug 07 08:21:08.448557 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.448573 2020] [:error] [pid 1420287] [remote x.x.x.x:144] self.user_id, self.project_id
[Fri Aug 07 08:21:08.448581 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.448597 2020] [:error] [pid 1420287] [remote x.x.x.x:144] __ret_val = __f(*args, **kwargs)
[Fri Aug 07 08:21:08.448605 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.449478 2020] [:error] [pid 1420287] [remote x.x.x.x:144] should_cache_fn)
[Fri Aug 07 08:21:08.449488 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.449504 2020] [:error] [pid 1420287] [remote x.x.x.x:144] async_creator) as value:
[Fri Aug 07 08:21:08.449512 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.449967 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self._enter()
[Fri Aug 07 08:21:08.449977 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.449995 2020] [:error] [pid 1420287] [remote x.x.x.x:144] generated = self._enter_
[Fri Aug 07 08:21:08.450004 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.450020 2020] [:error] [pid 1420287] [remote x.x.x.x:144] created = self.creator()
[Fri Aug 07 08:21:08.450029 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.450046 2020] [:error] [pid 1420287] [remote x.x.x.x:144] created_value = creator()
[Fri Aug 07 08:21:08.450055 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.450071 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return fn(*arg, **kw)
[Fri Aug 07 08:21:08.450080 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.450975 2020] [:error] [pid 1420287] [remote x.x.x.x:144] user_id=user_id, project_
[Fri Aug 07 08:21:08.450985 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.451001 2020] [:error] [pid 1420287] [remote x.x.x.x:144] __ret_val = __f(*args, **kwargs)
[Fri Aug 07 08:21:08.451009 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.451025 2020] [:error] [pid 1420287] [remote x.x.x.x:144] strip_domain_roles)
[Fri Aug 07 08:21:08.451033 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.451049 2020] [:error] [pid 1420287] [remote x.x.x.x:144] domain_
[Fri Aug 07 08:21:08.451057 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.451072 2020] [:error] [pid 1420287] [remote x.x.x.x:144] group_ids=
[Fri Aug 07 08:21:08.451081 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib/
[Fri Aug 07 08:21:08.451599 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return [denormalize_
[Fri Aug 07 08:21:08.451609 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/
[Fri Aug 07 08:21:08.451632 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return list(self)
[Fri Aug 07 08:21:08.451641 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/
[Fri Aug 07 08:21:08.451656 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return self._execute_
[Fri Aug 07 08:21:08.451665 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/
[Fri Aug 07 08:21:08.451683 2020] [:error] [pid 1420287] [remote x.x.x.x:144] result = conn.execute(
[Fri Aug 07 08:21:08.451691 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/
[Fri Aug 07 08:21:08.451711 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return meth(self, multiparams, params)
[Fri Aug 07 08:21:08.451720 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/
[Fri Aug 07 08:21:08.451736 2020] [:error] [pid 1420287] [remote x.x.x.x:144] return connection.
[Fri Aug 07 08:21:08.451745 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/
[Fri Aug 07 08:21:08.451762 2020] [:error] [pid 1420287] [remote x.x.x.x:144] distilled_params,
[Fri Aug 07 08:21:08.451771 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/
[Fri Aug 07 08:21:08.451786 2020] [:error] [pid 1420287] [remote x.x.x.x:144] e, statement, parameters, cursor, context
[Fri Aug 07 08:21:08.451795 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/
[Fri Aug 07 08:21:08.451810 2020] [:error] [pid 1420287] [remote x.x.x.x:144] util.raise_
[Fri Aug 07 08:21:08.451818 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/
[Fri Aug 07 08:21:08.451834 2020] [:error] [pid 1420287] [remote x.x.x.x:144] reraise(
[Fri Aug 07 08:21:08.451843 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/
[Fri Aug 07 08:21:08.451858 2020] [:error] [pid 1420287] [remote x.x.x.x:144] cursor, statement, parameters, context
[Fri Aug 07 08:21:08.451866 2020] [:error] [pid 1420287] [remote x.x.x.x:144] File "/usr/lib64/
[Fri Aug 07 08:21:08.451882 2020] [:error] [pid 1420287] [remote x.x.x.x:144] cursor.
[Fri Aug 07 08:21:08.451923 2020] [:error] [pid 1420287] [remote x.x.x.x:144] DBNonExistentTable: (sqlite3.
Some additional piece of information is that this is observed only in cases where httpd service is restarted.. well not in all cases of restart.