swift proxy throws timeout errors

Bug #1700150 reported by Pradeep Kilambi on 2017-06-23
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Unassigned

Bug Description

[root@overcloud-controller-0 ~]# systemctl status openstack-swift-proxy
● openstack-swift-proxy.service - OpenStack Object Storage (swift) - Proxy Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-swift-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2017-06-23 16:05:01 UTC; 2h 22min ago
 Main PID: 29562 (swift-proxy-ser)
   CGroup: /system.slice/openstack-swift-proxy.service
           ├─29562 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
           └─29899 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf

Jun 23 18:26:59 overcloud-controller-0 proxy-server[29899]: - - 23/Jun/2017/18/26/59 HEAD /v1/AUTH_8943125e619d...3 -
Jun 23 18:26:59 overcloud-controller-0 proxy-server[29899]: Authorization failed for token
Jun 23 18:26:59 overcloud-controller-0 proxy-server[29899]: Identity response: {"error": {"message": "Failed to..."}}
Jun 23 18:26:59 overcloud-controller-0 proxy-server[29899]: STDERR: ERROR:oslo.messaging._drivers.impl_rabbit:[...one
Jun 23 18:27:31 overcloud-controller-0 proxy-server[29899]: STDERR: ERROR:root:Timeout talking to memcached: 17...73)
Jun 23 18:27:31 overcloud-controller-0 proxy-server[29899]: Authorization failed for token
Jun 23 18:27:31 overcloud-controller-0 proxy-server[29899]: Invalid user token
Jun 23 18:27:31 overcloud-controller-0 proxy-server[29899]: 172.16.1.8 172.16.1.8 23/Jun/2017/18/27/31 GET /v1/...6 -
Jun 23 18:27:31 overcloud-controller-0 proxy-server[29899]: - - 23/Jun/2017/18/27/31 HEAD /v1/AUTH_9af79014806e...3 -
Jun 23 18:27:31 overcloud-controller-0 proxy-server[29899]: STDERR: ERROR:oslo.messaging._drivers.impl_rabbit:[...one
Hint: Some lines were ellipsized, use -l to show in full.

swift.log:

Jun 23 18:27:29 localhost container-server: no_change:138 ts_repl:0 diff:0 rsync:0 diff_capped:0 hashmatch:0 empty:0
Jun 23 18:27:31 localhost proxy-server: STDERR: ERROR:root:Timeout talking to memcached: 172.16.2.7:11211 (txn: tx1f3
1276cf17a4d39916d7-00594d5c73)
Jun 23 18:27:31 localhost proxy-server: Authorization failed for token
Jun 23 18:27:31 localhost proxy-server: Invalid user token
Jun 23 18:27:31 localhost proxy-server: 172.16.1.8 172.16.1.8 23/Jun/2017/18/27/31 GET /v1/AUTH_8943125e619d485c87965
261f5704d4e/incoming128-77%3Fformat%3Djson%26delimiter%3D/ HTTP/1.0 401 - python-swiftclient-3.3.0 gAAAAABZTT1aSb5C..
. - 131 - tx1f31276cf17a4d39916d7-00594d5c73 - 288.4203 - - 1498242163.534487963 1498242451.954771996 -
Jun 23 18:27:31 localhost container-server: 172.16.3.8 - - [23/Jun/2017:18:27:31 +0000] "HEAD /d1/253/AUTH_8943125e61
9d485c87965261f5704d4e/incoming128-0" 204 - "HEAD http://10.0.0.5:13808/v1/AUTH_8943125e619d485c87965261f5704d4e/inco
ming128-0" "tx50e1afaacdd84d65ae732-00594d5d13" "proxy-server 29899" 0.0012 "-" 30317 0
Jun 23 18:27:31 localhost proxy-server: - - 23/Jun/2017/18/27/31 HEAD /v1/AUTH_9af79014806e4b42900ef416ee86c394 HTTP/1.0 204 - Swift - - - - tx5406b393eb614722995fa-00594d5d53 - 32.0519 RL - 1498242419.904859066 1498242451.956803083 -
Jun 23 18:27:31 localhost proxy-server: STDERR: ERROR:oslo.messaging._drivers.impl_rabbit:[e9c92434-c682-43c7-8a88-b0e5328b91e0] AMQP server on overcloud-controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None

Pradeep Kilambi (pkilambi) wrote :

full swift log:

http://chunk.io/f/588ab4b3d42c4944ba99a923a437b7e8

The above error is on overcloud controller node, using basic deploy with 1 controller and 1 compute.

Changed in tripleo:
status: New → Triaged
importance: Undecided → High
milestone: none → pike-3
Changed in tripleo:
milestone: pike-3 → pike-rc1
Ben Nemec (bnemec) wrote :

Log entries like

Jun 23 18:31:48 localhost container-server: 172.16.3.8 - - [23/Jun/2017:18:31:48 +0000] "HEAD /d1/253/AUTH_8943125e619d485c87965261f5704d4e/incoming128-0" 204 - "HEAD http://10.0.0.5:13808/v1/AUTH_8943125e619d485c87965261f5704d4e/incoming128-0" "tx5cce2b4d87ca442697823-00594d5e14" "proxy-server 29899" 0.0020 "-" 30317 0

make me think this may be an SSL misconfiguration. There shouldn't be an http request going to 13808, which is the SSL port.

That being said, my guess would be that the rabbit failure is happening because Swift is trying to send an error notification about the bad request. Whether that's failing because of the SSL misconfiguration or because there's legitimately a problem I can't really say. I think we need more details about how this environment was deployed and what was going on to cause these bogus calls to Swift. I don't see similar errors in CI so it's not a general problem with Swift.

Changed in tripleo:
status: Triaged → Incomplete
Changed in tripleo:
milestone: pike-rc1 → pike-rc2
Changed in tripleo:
milestone: pike-rc2 → queens-1
Changed in tripleo:
milestone: queens-1 → queens-2
Changed in tripleo:
milestone: queens-2 → queens-3
Changed in tripleo:
milestone: queens-3 → queens-rc1
Changed in tripleo:
milestone: queens-rc1 → rocky-1
Changed in tripleo:
milestone: rocky-1 → rocky-2
Changed in tripleo:
milestone: rocky-2 → rocky-3
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Changed in tripleo:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers