Long Run: Horizon failed with 500 internal server error
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Mirantis OpenStack |
Incomplete
|
Medium
|
Alexander Nagovitsyn |
Bug Description
Possible workaround:
restart apache2 service on all controller nodes.
Steps To Reproduce:
1. Deploy OpenStack cluster with 3 controllers with vbox scripts
2. Wait 1 week
3. Open Horizon index page: https:/
Expected Result:
User can successfully open Horizon UI
Observed Result:
User can see error: 500: Internal server error.
When we restart apache2 service we will see the following warnings and errors:
root@node-3:~# service apache2 restart
* Restarting web server apache2 AH00316: WARNING: MaxRequestWorkers of 798 is not an integer multiple of
ThreadsPerChild of 25, decreasing to nearest multiple 775,
for a maximum of 31 servers.
(98)Address already in use: AH00072: make_sock: could not bind to address [::]:35357
(98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:35357
no listening sockets available, shutting down
AH00015: Unable to open logs
Action 'start' failed.
The Apache error log may have more information.
Reproduced on
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "8.0"
openstack_
api: "1.0"
build_number: "227"
build_id: "227"
fuel-nailgun_sha: "28025f978a6c5d
python-
fuel-agent_sha: "1672c18c8193e1
fuel-
astute_sha: "b60624ee2c5f1d
fuel-library_sha: "b9f8c28f0e8f7d
fuel-ostf_sha: "fc6648be21f26f
fuel-
fuelmenu_sha: "4064b7723773ab
shotgun_sha: "25a0cc461a9fa4
network-
fuel-upgrade_sha: "1e894e26d4e142
fuelmain_sha: "2eca6adc33f02e
Changed in fuel: | |
milestone: | none → 8.0 |
affects: | fuel → mos |
Changed in mos: | |
milestone: | 8.0 → none |
milestone: | none → 8.0 |
assignee: | nobody → MOS Horizon (mos-horizon) |
importance: | Undecided → Medium |
tags: | added: horizon long-haul-testing |
Changed in mos: | |
milestone: | 8.0 → 9.0 |
Changed in mos: | |
assignee: | MOS Packaging Team (mos-packaging) → Mikhail Ivanov (mivanov) |
Changed in mos: | |
status: | Confirmed → In Progress |
tags: | added: scale |
Changed in mos: | |
assignee: | Mikhail Ivanov (mivanov) → Ivan Udovichenko (iudovichenko) |
Apache logs:
root@node-2:~# tail -n 200 -f /var/log/ apache2/ error.log
[Mon Dec 14 08:12:32.735850 2015] [mpm_event:notice] [pid 4331:tid 140553176475520] AH00489: Apache/2.4.7 (Ubuntu) configured -- resuming normal operations
[Mon Dec 14 08:12:32.735930 2015] [core:notice] [pid 4331:tid 140553176475520] AH00094: Command line: '/usr/sbin/apache2'
[Mon Dec 14 08:12:36.236677 2015] [mpm_event:notice] [pid 4331:tid 140553176475520] AH00491: caught SIGTERM, shutting down
[Mon Dec 14 08:12:36.461300 2015] [mpm_worker:notice] [pid 4674:tid 140649956820864] AH00292: Apache/2.4.7 (Ubuntu) configured -- resuming normal operations
[Mon Dec 14 08:12:36.461329 2015] [core:notice] [pid 4674:tid 140649956820864] AH00094: Command line: '/usr/sbin/apache2'
[Mon Dec 14 08:13:18.850868 2015] [mpm_worker:notice] [pid 4674:tid 140649956820864] AH00297: SIGUSR1 received. Doing graceful restart
[Mon Dec 14 08:13:18.854440 2015] [mpm_worker:warn] [pid 4674:tid 140649956820864] AH00317: MaxRequestWorkers of 798 is not an integer multiple of ThreadsPerChild of 25, decreasing to nearest multiple 775
[Mon Dec 14 08:13:18.854740 2015] [mpm_worker:notice] [pid 4674:tid 140649956820864] AH00292: Apache/2.4.7 (Ubuntu) configured -- resuming normal operations
[Mon Dec 14 08:13:18.854746 2015] [core:notice] [pid 4674:tid 140649956820864] AH00094: Command line: '/usr/sbin/apache2'
[Mon Dec 14 08:17:35.333464 2015] [mpm_worker:notice] [pid 4674:tid 140649956820864] AH00297: SIGUSR1 received. Doing graceful restart
[Mon Dec 14 08:17:35.336561 2015] [mpm_worker:warn] [pid 4674:tid 140649956820864] AH00317: MaxRequestWorkers of 798 is not an integer multiple of ThreadsPerChild of 25, decreasing to nearest multiple 775
[Mon Dec 14 08:17:35.336882 2015] [mpm_worker:notice] [pid 4674:tid 140649956820864] AH00292: Apache/2.4.7 (Ubuntu) configured -- resuming normal operations
[Mon Dec 14 08:17:35.336889 2015] [core:notice] [pid 4674:tid 140649956820864] AH00094: Command line: '/usr/sbin/apache2'
[Mon Dec 14 08:17:53.578509 2015] [mpm_worker:notice] [pid 4674:tid 140649956820864] AH00295: caught SIGTERM, shutting down
[Mon Dec 14 08:17:54.325429 2015] [mpm_worker:notice] [pid 3988:tid 139719608993664] AH00292: Apache/2.4.7 (Ubuntu) mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
[Mon Dec 14 08:17:54.325462 2015] [core:notice] [pid 3988:tid 139719608993664] AH00094: Command line: '/usr/sbin/apache2'
[Mon Dec 14 08:18:28.572007 2015] [mpm_worker:notice] [pid 3988:tid 139719608993664] AH00297: SIGUSR1 received. Doing graceful restart
[Mon Dec 14 08:18:28.581871 2015] [mpm_worker:warn] [pid 3988:tid 139719608993664] AH00317: MaxRequestWorkers of 798 is not an integer multiple of ThreadsPerChild of 25, decreasing to nearest multiple 775
[Mon Dec 14 08:18:28.583127 2015] [mpm_worker:notice] [pid 3988:tid 139719608993664] AH00292: Apache/2.4.7 (Ubuntu) mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
[Mon Dec 14 08:18:28.583155 2015] [core:notice] [pid 3988:tid 139719608993664] AH00094: Command line: '/usr/sbin/apache2'
[Mon Dec 14 08:26:08.311634 2015] [mpm_worker:notice] [pid 3988:tid 139719608993664] AH00297: SIGUSR1 received. Doing graceful resta...