Comment 1 for bug 1512852

Revision history for this message
clayg (clay-gerrard) wrote :

So, I shutdown half of my saio object servers and was able to reproduce this:

    Nov 3 22:15:15 saio proxy-server: ERROR with Object server 127.0.0.1:6040/sdb8 re: Trying to GET /v1/AUTH_test/ec-test/test: Connection refused
    Nov 3 22:15:15 saio object-6020: 127.0.0.1 - - [03/Nov/2015:22:15:15 +0000] "GET /sdb2/941/AUTH_test/ec-test/test" 200 82 "GET http://saio:8080/v1/AUTH_test/ec-test/test" "tx6be44b4f68c54c0f881f9-00563931f3" "proxy-server 27232" 0.0024 "-" 27264 1
    Nov 3 22:15:15 saio proxy-server: ERROR with Object server 127.0.0.1:6040/sdb4 re: Trying to GET /v1/AUTH_test/ec-test/test: Connection refused
    Nov 3 22:15:15 saio proxy-server: ERROR with Object server 127.0.0.1:6030/sdb7 re: Trying to GET /v1/AUTH_test/ec-test/test: Connection refused
    Nov 3 22:15:15 saio object-6010: 127.0.0.1 - - [03/Nov/2015:22:15:15 +0000] "GET /sdb1/941/AUTH_test/ec-test/test" 200 82 "GET http://saio:8080/v1/AUTH_test/ec-test/test" "tx6be44b4f68c54c0f881f9-00563931f3" "proxy-server 27232" 0.0008 "-" 27205 1
    Nov 3 22:15:15 saio object-6010: 127.0.0.1 - - [03/Nov/2015:22:15:15 +0000] "GET /sdb5/941/AUTH_test/ec-test/test" 200 82 "GET http://saio:8080/v1/AUTH_test/ec-test/test" "tx6be44b4f68c54c0f881f9-00563931f3" "proxy-server 27232" 0.0011 "-" 27205 1
    Nov 3 22:15:15 saio object-6020: 127.0.0.1 - - [03/Nov/2015:22:15:15 +0000] "GET /sdb6/941/AUTH_test/ec-test/test" 404 - "GET http://saio:8080/v1/AUTH_test/ec-test/test" "tx6be44b4f68c54c0f881f9-00563931f3" "proxy-server 27232" 0.0004 "-" 27264 1
    Nov 3 22:15:15 saio proxy-server: ERROR with Object server 127.0.0.1:6030/sdb3 re: Trying to GET /v1/AUTH_test/ec-test/test: Connection refused
    Nov 3 22:15:15 saio proxy-server: 127.0.0.1 127.0.0.1 03/Nov/2015/22/15/15 GET /v1/AUTH_test/ec-test/test HTTP/1.0 404 - python-swiftclient-2.6.1.dev10 AUTH_tk84450147e... - 70 - tx6be44b4f68c54c0f881f9-00563931f3 - 0.0284 - - 1446588915.392549038 1446588915.420908928 1

Oddly if I disable 3/4 object servers - it get the 503's:

    Nov 3 22:17:21 saio object-6010: 127.0.0.1 - - [03/Nov/2015:22:17:21 +0000] "GET /sdb5/941/AUTH_test/ec-test/test" 200 82 "GET http://saio:8080/v1/AUTH_test/ec-test/test" "txc2605e2641694ae69f1d2-0056393271" "proxy-server 27232" 0.0009 "-" 27205 1
    Nov 3 22:17:21 saio object-6010: 127.0.0.1 - - [03/Nov/2015:22:17:21 +0000] "GET /sdb1/941/AUTH_test/ec-test/test" 200 82 "GET http://saio:8080/v1/AUTH_test/ec-test/test" "txc2605e2641694ae69f1d2-0056393271" "proxy-server 27232" 0.0006 "-" 27205 1
    Nov 3 22:17:21 saio proxy-server: ERROR with Object server 127.0.0.1:6040/sdb4 re: Trying to GET /v1/AUTH_test/ec-test/test: Connection refused
    Nov 3 22:17:21 saio proxy-server: ERROR with Object server 127.0.0.1:6040/sdb8 re: Trying to GET /v1/AUTH_test/ec-test/test: Connection refused
    Nov 3 22:17:21 saio proxy-server: ERROR with Object server 127.0.0.1:6030/sdb7 re: Trying to GET /v1/AUTH_test/ec-test/test: Connection refused
    Nov 3 22:17:21 saio proxy-server: ERROR with Object server 127.0.0.1:6020/sdb2 re: Trying to GET /v1/AUTH_test/ec-test/test: Connection refused
    Nov 3 22:17:21 saio proxy-server: ERROR with Object server 127.0.0.1:6030/sdb3 re: Trying to GET /v1/AUTH_test/ec-test/test: Connection refused
    Nov 3 22:17:21 saio proxy-server: ERROR with Object server 127.0.0.1:6020/sdb6 re: Trying to GET /v1/AUTH_test/ec-test/test: Connection refused
    Nov 3 22:17:21 saio proxy-server: Object returning 503 for [] (txn: txc2605e2641694ae69f1d2-0056393271) (client_ip: 127.0.0.1)
    Nov 3 22:17:21 saio proxy-server: 127.0.0.1 127.0.0.1 03/Nov/2015/22/17/21 GET /v1/AUTH_test/ec-test/test HTTP/1.0 503 - python-swiftclient-2.6.1.dev10 AUTH_tk84450147e... - 118 - txc2605e2641694ae69f1d2-0056393271 - 0.0166 - - 1446589041.169792891 1446589041.186393023 1