Apache Server Status for 192.168.24.11 (via 192.168.24.11) Server Version: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 Server MPM: prefork Server Built: Nov 14 2016 18:04:44 Current Time: Tuesday, 21-Feb-2017 17:56:19 UTC Restart Time: Tuesday, 21-Feb-2017 16:04:58 UTC Parent Server Config. Generation: 1 Parent Server MPM Generation: 0 Server uptime: 1 hour 51 minutes 20 seconds Server load: 0.29 0.31 0.37 Total accesses: 624 - Total Traffic: 576 kB CPU Usage: u1.32 s.47 cu0 cs0 - .0268% CPU load .0934 requests/sec - 88 B/second - 945 B/request 32 requests currently being processed, 0 idle workers WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW Scoreboard Key: "_" Waiting for Connection, "S" Starting up, "R" Reading Request, "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup, "C" Closing connection, "L" Logging, "G" Gracefully finishing, "I" Idle cleanup of worker, "." Open slot with no current process Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request 0-0 33460 0/22/22 W 0.06 1163 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 1-0 33461 0/17/17 W 0.04 1143 0 0.0 0.01 0.01 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 2-0 33462 0/22/22 W 0.06 1059 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 3-0 33463 0/23/23 W 0.07 1041 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 4-0 33464 0/17/17 W 0.06 957 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 5-0 33465 0/17/17 W 0.05 834 0 0.0 0.01 0.01 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 6-0 33466 0/19/19 W 0.06 853 0 0.0 0.01 0.01 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 7-0 33467 0/19/19 W 0.04 937 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 8-0 33549 0/24/24 W 0.07 1472 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 9-0 33568 0/21/21 W 0.06 1266 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 10-0 33569 0/21/21 W 0.06 1576 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 11-0 33572 0/18/18 W 0.05 411 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 12-0 33573 0/21/21 W 0.06 750 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 13-0 33574 0/22/22 W 0.07 647 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 14-0 33575 0/19/19 W 0.05 1563 0 0.0 0.01 0.01 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 15-0 33634 0/22/22 W 0.05 235 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 16-0 33635 0/19/19 W 0.05 728 0 0.0 0.01 0.01 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 17-0 33636 0/17/17 W 0.05 1369 0 0.0 0.01 0.01 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 18-0 33637 0/21/21 W 0.06 308 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 19-0 33638 0/16/16 W 0.05 1351 0 0.0 0.01 0.01 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 20-0 33639 0/17/17 W 0.06 102 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 21-0 33640 0/17/17 W 0.05 1247 0 0.0 0.01 0.01 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 22-0 33641 0/19/19 W 0.05 544 0 0.0 0.01 0.01 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 23-0 33644 0/21/21 W 0.07 0 0 0.0 0.02 0.02 127.0.0.1 overcloud-controller-0.localdom GET /server-status HTTP/1.1 24-0 33645 0/18/18 W 0.05 339 0 0.0 0.03 0.03 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 25-0 33646 0/18/18 W 0.05 1454 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 26-0 33647 0/17/17 W 0.05 515 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 27-0 33648 0/20/20 W 0.06 441 0 0.0 0.01 0.01 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 28-0 33649 0/17/17 W 0.05 205 0 0.0 0.01 0.01 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 29-0 33650 0/21/21 W 0.06 29 0 0.0 0.04 0.04 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 30-0 33651 0/22/22 W 0.06 132 0 0.0 0.02 0.02 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 31-0 33652 0/20/20 W 0.06 625 0 0.0 0.03 0.03 192.168.24.11 overcloud-controller-0.internal GET /v1/capabilities/ HTTP/1.1 Srv Child Server number - generation PID OS process ID Acc Number of accesses this connection / this child / this slot M Mode of operation CPU CPU usage, number of seconds SS Seconds since beginning of most recent request Req Milliseconds required to process most recent request Conn Kilobytes transferred this connection Child Megabytes transferred this child Slot Total megabytes transferred this slot [root@overcloud-controller-0 ~]# ps aux | grep httpd root 33448 0.0 0.0 157940 5476 ? Ss 16:04 0:01 /usr/sbin/httpd -DFOREGROUND apache 33454 0.0 0.0 326084 6752 ? Sl 16:04 0:00 /usr/sbin/httpd -DFOREGROUND apache 33455 0.0 0.0 391620 6752 ? Sl 16:04 0:00 /usr/sbin/httpd -DFOREGROUND apache 33456 0.0 0.0 326084 6752 ? Sl 16:04 0:00 /usr/sbin/httpd -DFOREGROUND apache 33460 0.0 0.0 162388 7744 ? S 16:04 0:00 /usr/sbin/httpd -DFOREGROUND apache 33461 0.0 0.0 162388 7712 ? S 16:04 0:00 /usr/sbin/httpd -DFOREGROUND apache 33462 0.0 0.0 162388 7748 ? S 16:04 0:00 /usr/sbin/httpd -DFOREGROUND apache 33463 0.0 0.0 162388 7744 ? S 16:04 0:00 /usr/sbin/httpd -DFOREGROUND apache 33464 0.0 0.0 162388 7744 ? S 16:04 0:00 /usr/sbin/httpd -DFOREGROUND apache 33465 0.0 0.0 162388 7728 ? S 16:04 0:00 /usr/sbin/httpd -DFOREGROUND apache 33466 0.0 0.0 162388 7728 ? S 16:04 0:00 /usr/sbin/httpd -DFOREGROUND apache 33467 0.0 0.0 162388 7728 ? S 16:04 0:00 /usr/sbin/httpd -DFOREGROUND apache 33549 0.0 0.0 162388 7748 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33568 0.0 0.0 162388 7712 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33569 0.0 0.0 162388 7696 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33572 0.0 0.0 162388 7728 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33573 0.0 0.0 162388 7748 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33574 0.0 0.0 162388 7748 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33575 0.0 0.0 162388 7752 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33634 0.0 0.0 162388 7748 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33635 0.0 0.0 162388 7748 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33636 0.0 0.0 162388 7716 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33637 0.0 0.0 162388 7752 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33638 0.0 0.0 162380 7608 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33639 0.0 0.0 162388 7716 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33640 0.0 0.0 162388 7732 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33641 0.0 0.0 162388 7748 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33644 0.0 0.0 162388 7748 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33645 0.0 0.0 162388 7752 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33646 0.0 0.0 162388 7748 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33647 0.0 0.0 162388 7732 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33648 0.0 0.0 162388 7732 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33649 0.0 0.0 162388 7704 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33650 0.0 0.0 162388 7732 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33651 0.0 0.0 162388 7748 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND apache 33652 0.0 0.0 162388 7748 ? S 16:05 0:00 /usr/sbin/httpd -DFOREGROUND [root@overcloud-controller-0 ~]# strace -p 33454 Process 33454 attached restart_syscall(<... resuming interrupted call ...> ^CProcess 33454 detached [root@overcloud-controller-0 ~]# strace -p 33455 Process 33455 attached restart_syscall(<... resuming interrupted call ...>^CProcess 33455 detached [root@overcloud-controller-0 ~]# strace -p 33456 Process 33456 attached restart_syscall(<... resuming interrupted call ...>^CProcess 33456 detached [root@overcloud-controller-0 ~]# strace -p 33457 Process 33457 attached restart_syscall(<... resuming interrupted call ...>^CProcess 33457 detached [root@overcloud-controller-0 ~]# strace -p 33652 Process 33652 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110 ^CProcess 33652 detached [root@overcloud-controller-0 ~]# strace -p 33651 Process 33651 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33651 detached [root@overcloud-controller-0 ~]# strace -p 33650 Process 33650 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33650 detached [root@overcloud-controller-0 ~]# strace -p 33649 Process 33649 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33649 detached [root@overcloud-controller-0 ~]# strace -p 33648 Process 33648 attached restart_syscall(<... resuming interrupted call ...>^CProcess 33648 detached [root@overcloud-controller-0 ~]# strace -p 33647 Process 33647 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33647 detached [root@overcloud-controller-0 ~]# strace -p 33646 Process 33646 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33646 detached [root@overcloud-controller-0 ~]# strace -p 33645 Process 33645 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33645 detached [root@overcloud-controller-0 ~]# strace -p 33644 Process 33644 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33644 detached [root@overcloud-controller-0 ~]# strace -p 33643 strace: attach: ptrace(PTRACE_ATTACH, ...): No such process [root@overcloud-controller-0 ~]# strace -p 33641 Process 33641 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33641 detached [root@overcloud-controller-0 ~]# strace -p 33640 Process 33640 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33640 detached [root@overcloud-controller-0 ~]# strace -p 33639 Process 33639 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33639 detached [root@overcloud-controller-0 ~]# strace -p 33638 Process 33638 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33638 detached [root@overcloud-controller-0 ~]# strace -p 33637 Process 33637 attached restart_syscall(<... resuming interrupted call ...>^CProcess 33637 detached [root@overcloud-controller-0 ~]# strace -p 33636 Process 33636 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33636 detached [root@overcloud-controller-0 ~]# strace -p 33635 Process 33635 attached connect(33, {sa_family=AF_LOCAL, sun_path="/var/run/wsgi.33448.0.4.sock"}, 110^CProcess 33635 detached [root@overcloud-controller-0 ~]# lsof /var/run/wsgi.33448.0.4.sock COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME httpd 33448 root 34u unix 0xffff8801a5bd5400 0t0 153190 /var/run/wsgi.33448.0.4.sock httpd 33452 gnocchi 6u unix 0xffff88001fe78800 0t0 897824 /var/run/wsgi.33448.0.4.sock httpd 33452 gnocchi 34u unix 0xffff8801a5bd5400 0t0 153190 /var/run/wsgi.33448.0.4.sock httpd 33453 gnocchi 6u unix 0xffff88001fe7bc00 0t0 894656 /var/run/wsgi.33448.0.4.sock httpd 33453 gnocchi 34u unix 0xffff8801a5bd5400 0t0 153190 /var/run/wsgi.33448.0.4.sock neutron-r 37641 root 5u unix 0xffff8801c765e400 0t0 179563 /tmp/rootwrap-InsfHA/rootwrap.sock These processes are still stuck in apache: /var/log/httpd/gnocchi_wsgi_access.log:192.168.24.11 - - [21/Feb/2017:18:43:24 +0000] "GET /v1/capabilities/ HTTP/1.1" 500 531 "-" "ceilometer-agent-notification keystoneauth1/2.18.0 python-requests/2.11.1 CPython/2.7.5" /var/log/httpd/gnocchi_wsgi_access.log:192.168.24.11 - - [21/Feb/2017:18:27:26 +0000] "GET /v1/capabilities/ HTTP/1.1" 500 531 "-" "ceilometer-agent-notification keystoneauth1/2.18.0 python-requests/2.11.1 CPython/2.7.5" /var/log/gnocchi/app.log 2017-02-21 18:46:18.737 33453 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://192.168.24.6:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",)) 2017-02-21 18:46:32.177 33452 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://192.168.24.6:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",)) 2017-02-21 18:48:04.337 33453 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://192.168.24.6:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",)) 2017-02-21 18:48:15.857 33452 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://192.168.24.6:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",)) 2017-02-21 18:49:48.017 33453 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://192.168.24.6:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",)) 2017-02-21 18:49:58.513 33452 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://192.168.24.6:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",)) /var/log/httpd/gnocchi_wsgi_error.log [Tue Feb 21 18:57:09.162953 2017] [core:error] [pid 33466] [client 192.168.24.11:52820] Script timed out before returning headers: app [Tue Feb 21 18:57:10.683013 2017] [core:error] [pid 33465] [client 192.168.24.11:53516] Script timed out before returning headers: app [Tue Feb 21 18:59:10.174227 2017] [core:error] [pid 33573] [client 192.168.24.11:53468] Script timed out before returning headers: app [Tue Feb 21 18:59:10.695961 2017] [core:error] [pid 33574] [client 192.168.24.11:58688] Script timed out before returning headers: app /var/log/keystone/keystone.log 2017-02-21 18:36:14.754 33458 INFO keystone.common.wsgi [req-cfaabab9-6098-418c-9488-dec81427cbfd - - - - -] POST http://192.168.24.6:5000/v3/auth/tokens 2017-02-21 18:36:14.954 33458 DEBUG keystone.auth.core [req-cfaabab9-6098-418c-9488-dec81427cbfd - - - - -] MFA Rules not processed for user `15c10047252d484a9516c8d097395711`. Rule list: `[]` (Enabled: `True`). check_auth_methods_against_rules /usr/lib/python2.7/site-packages/keystone/auth/core.py:388 2017-02-21 18:37:43.846 33458 DEBUG keystone.middleware.auth [req-f2424fc6-75e4-4def-a0f2-e2db979fc5e2 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth co ntext will be set. fill_context /usr/lib/python2.7/site-packages/keystone/middleware/auth.py:203 [root@overcloud-controller-0 ~]# ps aux | grep gnocchi gnocchi 33452 0.2 1.4 1859664 119220 ? Sl 16:04 0:23 gnocchi_wsgi -DFOREGROUND gnocchi 33453 0.2 1.4 1859664 119220 ? Sl 16:04 0:23 gnocchi_wsgi -DFOREGROUND