Comment 3 for bug 1891019

Upon further analysis, it seems like the keystone /httpd logrotate is what is causing the httpd service to reload. Unlike what was though of earlier, the cause seems to be httpd reload instead of a restart. The log rotate files for keystone / httpd looks like the below:

https://access.redhat.com/solutions/2626601 (some related data)

# cat /etc/logrotate.d/openstack-keystone
/var/log/keystone/*.log {
    size 10M
    rotate 4
    missingok
    compress
postrotate
/bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true >> reloads httpd service after logrotate
endscript
}

/var/log/keystone/*/*.log {
    size 10M
    rotate 4
    missingok
    compress
postrotate
/bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
endscript
}

# cat httpd
/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    delaycompress
    postrotate
        /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
    endscript
}

httpd reload internally kills and starts the wsgidaemon processes running inside it. Keystone processes run as wsgi daemon processes in my environment. In a graceful restart of httpd, keystone processes stop gracefully and start properly. When keystone process starts, it gets a handle to mariadb based connection from /etc/keystone/keystone.conf via oslo.db / sqlalchemy. However, in cases where this problem is reported, a sqllite3 error is reported.

/etc/httpd/conf.d/keystone.conf:
Listen 5000
<VirtualHost *:5000>
   SSLEngine on
   SSLProtocol +TLSv1.2
.....
....
   WSGIDaemonProcess keystone-public user=keystone group=keystone display-name=keystone-public >> wsg daemon running inside httpd for keystone

[:error] [pid 1420287] [remote x.x.x.x:144] DBNonExistentTable: (sqlite3.OperationalError) no such table: assignment [SQL: u'SELECT assignment.type AS assignment_type, assignment.actor_id AS assignment_actor_id, assignment.target_id AS assignment_target_id, assignment.role_id AS assignment_role_id, assignment.inherited AS assignment_inherited \\nFROM assignment \\nWHERE assignment.actor_id IN (?) AND assignment.target_id IN (?) AND assignment.type IN (?) AND assignment.inherited = 0'] [parameters: ('15c2fe91e053af57a997c568c117c908d59c138f996bdc19ae97e9f16df12345', '12345978536e45ab8a279e2b0fa4f947', 'UserProject')] (Background on this error at: http://sqlalche.me/e/e3q8)

My guess is the problem here is that when httpd reload happens, the old keystone process is killed and within a span of 3-5 sec, the new process is spawn and in some very corner cases, the mariadb connection is still with the old process and the new process is unable to acquire it. In such cases, the new process falls back on the default sqllite in-memory keystone.db that does not have any tables. This leads to no such table errors when requests hit keystone process. >> https://github.com/openstack/keystone/blob/stable/stein/keystone/common/sql/core.py#L81

def initialize():
"""Initialize the module."""
db_options.set_defaults(
CONF,
connection="sqlite:///keystone.db")

From >> https://modwsgi.narkive.com/uBZgDFuy/apache-reload-kills-django-process-abruptly

When using daemon mode of mod_wsgi that is how it works unfortunately. This is because Apache only extends gracious shutdown to its own child worker processes and not to managed daemon processes (keystone is a managed daemon process).So the daemon processes only get up to 5 seconds to terminate of their own accord when sent the signal from the Apache parent process, before the Apache parent will send them a SIGKILL. So if you have long running requests they will get terminated if they run longer than 5 seconds.
If this is an issue, the only option you would have is too not use logrotate and use Apache's own mechanism for log file rotation, which doesn't have this issue because it doesn't need a restart to work.
You could also perhaps use mod_wsgi-express to run separately managed instance, using its own log file rotation, and proxy from the main Apache instance.
-Graham