OSTF is not available in UI on scale

[Sergey Galkin]

Steps for reproduce:
1. Install 9.0
2. Upgrade to 9.2
3. Deploy big cluster (~400 nodes in my case)
4. Reboot fuel host

Try to open healthcheck tab
The tab shows error: OSTF server is not available.

But ostf server up and running

[root@fuel ostf]# systemctl status ostf
● ostf.service
   Loaded: loaded (/usr/lib/systemd/system/ostf.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/ostf.service.d
           └─fuel.conf
   Active: active (running) since Thu 2017-02-02 10:14:07 UTC; 12min ago
 Main PID: 25244 (ostf-server)
   CGroup: /system.slice/ostf.service
           └─25244 /usr/bin/python /usr/bin/ostf-server

Feb 02 10:19:51 fuel.domain.tld ostf[25244]: 2017-02-02 10:19:51 INFO (connectionpool) Resetting dropped connection: 10.21.0.2
Feb 02 10:19:59 fuel.domain.tld ostf[25244]: 2017-02-02 10:19:59 INFO (connectionpool) Resetting dropped connection: 10.21.0.2
Feb 02 10:20:01 fuel.domain.tld ostf[25244]: 10.21.0.2 - - [2017-02-02 10:20:01] "GET /v1/testsets/2 HTTP/1.0" 200 603 48.410948
Feb 02 10:20:01 fuel.domain.tld ostf[25244]: 10.21.0.2 - - [2017-02-02 10:20:01] "GET /v1/testruns/last/2 HTTP/1.0" 200 143 0.067117
Feb 02 10:20:01 fuel.domain.tld ostf[25244]: 2017-02-02 10:20:01 INFO (connectionpool) Starting new HTTP connection (1): 10.21.0.2
Feb 02 10:20:19 fuel.domain.tld ostf[25244]: 2017-02-02 10:20:19 INFO (connectionpool) Resetting dropped connection: 10.21.0.2
Feb 02 10:20:28 fuel.domain.tld ostf[25244]: 2017-02-02 10:20:28 INFO (connectionpool) Resetting dropped connection: 10.21.0.2
Feb 02 10:20:37 fuel.domain.tld ostf[25244]: 2017-02-02 10:20:37 INFO (connectionpool) Resetting dropped connection: 10.21.0.2
Feb 02 10:20:46 fuel.domain.tld ostf[25244]: 2017-02-02 10:20:46 INFO (connectionpool) Resetting dropped connection: 10.21.0.2
Feb 02 10:20:47 fuel.domain.tld ostf[25244]: 10.21.0.2 - - [2017-02-02 10:20:47] "GET /v1/tests/2 HTTP/1.0" 200 26323 45.989129

[Sergey Galkin (sgalkin)]

in the nginx/error_services.log

2017/02/02 10:35:10 [error] 1571#1571: *1435614 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.3.56.253, server: localhost, request: "GET /ostf/testruns/last/2?_=1486031616200 HTTP/1.1", upstream: "http://10.21.0.2:8777/v1/testruns/last/2", host: "10.3.60.61:8443", referrer: "https://10.3.60.61:8443/"

in the same time
[root@fuel nginx]# curl http://10.21.0.2:8777/v1/testruns/last/2
Authentication required

[Sergey Galkin (sgalkin)]

It seems what http://10.21.0.2:8777/v1/tests/2 request take time more when timeout in UI

[Sergey Galkin (sgalkin)]

[root@fuel nginx]# time curl -H 'X-Auttime h-Token: 2783b8831def42dea8a88767ea0c85c4' http://10.21.0.2:8777/v1/tests/2
real 1m13.034s
user 0m0.005s
sys 0m0.007s

[Sergey Galkin (sgalkin)]

tcpdump for ostf server