Keystone: Apache restarts frequently (SIGTERM) possibly due to frequent change in the backend
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Keystone LDAP integration |
Fix Released
|
Medium
|
Billy Olsen | ||
keystone (Juju Charms Collection) |
Invalid
|
Undecided
|
Unassigned |
Bug Description
Hi,
We are seeing frequent SIGTERM errors
[Fri Jun 16 09:15:01.508555 2017] [mpm_event:notice] [pid 1426216:tid 140643397523328] AH00491: caught SIGTERM, shutting down
[Fri Jun 16 09:15:02.575654 2017] [mpm_event:notice] [pid 1429831:tid 140469605418880] AH00489: Apache/2.4.10 (Ubuntu) mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
[Fri Jun 16 09:15:02.575726 2017] [core:notice] [pid 1429831:tid 140469605418880] AH00094: Command line: '/usr/sbin/apache2'
[Fri Jun 16 09:17:40.960667 2017] [mpm_event:notice] [pid 1429831:tid 140469605418880] AH00491: caught SIGTERM, shutting down
[Fri Jun 16 09:17:42.028744 2017] [mpm_event:notice] [pid 1431988:tid 139635145045888] AH00489: Apache/2.4.10 (Ubuntu) mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
[Fri Jun 16 09:17:42.028826 2017] [core:notice] [pid 1431988:tid 139635145045888] AH00094: Command line: '/usr/sbin/apache2'
[Fri Jun 16 09:20:04.190324 2017] [mpm_event:notice] [pid 1431988:tid 139635145045888] AH00491: caught SIGTERM, shutting down
[Fri Jun 16 09:20:17.377800 2017] [mpm_event:notice] [pid 1435080:tid 139844850268032] AH00489: Apache/2.4.10 (Ubuntu) mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
[Fri Jun 16 09:20:17.377871 2017] [core:notice] [pid 1435080:tid 139844850268032] AH00094: Command line: '/usr/sbin/apache2'
It appears that the domain-backend is frequently flapping and causing Apache to restart. Charms installed on trusty (4.2.0-57-generic):
trusty/keystone git+https:/
trusty/
Keystone.config (partial)
[identity]
driver = sql
default_domain_id = <masked>
domain_
domain_config_dir = /etc/keystone/
[credential]
driver = sql
[trust]
driver = sql
[os_inherit]
[catalog]
driver = sql
[endpoint_filter]
[token]
driver = sql
provider = keystone.
expiration = 3600
[ldap]
[resource]
admin_project_
admin_project_name = admin
Needs to check why the backends is creating events that are causing needless relation-changed charm events.
summary: |
- Apache on Keystone frequently restart possibly due to frequent change in - the backend + Keystone: Apache restarts frequently (SIGTERM) possibly due to frequent + change in the backend |
Changed in keystone (Juju Charms Collection): | |
status: | New → Invalid |
Changed in charm-keystone-ldap: | |
status: | New → Triaged |
importance: | Undecided → Medium |
assignee: | nobody → Billy Olsen (billy-olsen) |
milestone: | none → 17.08 |
tags: | added: backport-potential |
Changed in charm-keystone-ldap: | |
status: | Fix Committed → Fix Released |
We have thousands of these log messages on each keystone server:
2017-06-09 03:50:40 INFO domain- backend- relation- changed * Restarting web server apache2
While it's possible that the root cause of this bug lies further back, the issue here is that apache2 is restarted across all keystones every time this relation fires. This is causing needless alerts and potential service availability problems.