Certain OS API services don't work after a successful deployment
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Fuel for OpenStack |
Invalid
|
Undecided
|
Unassigned | ||
Bug Description
{"ostf_sha": "d2a894d228c1f3
Environment: Ubuntu/HA/VLAN 3x controller/cinder lvm + 1 compute
After puppet runs successfully on all the nodes Astute times out uploading an image to Glance. Log: http://
When trying to glance image-list, most of the time the client just hangs up until haproxy returns HTTP 504, and sometimes it completes successfully:
root@node-2:~# glance image-list
Request returned failure status.
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
(HTTP N/A)
root@node-2:~# glance image-list
Request returned failure status.
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
(HTTP N/A)
root@node-2:~# glance image-list
+------
| ID | Name | Disk Format | Container Format | Size | Status |
+------
| 94f648eb-
| cfd780f4-
+------
Running glance client with --debug reveals that it hangs upon making a request to Glance API:
root@node-2:~# glance --debug image-list
curl -i -X GET -H 'X-Auth-Token: 4338291964f0462
HTTP/1.0 504 Gateway Time-out
connection: close
content-type: text/html
cache-control: no-cache
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
Request returned failure status.
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
(HTTP N/A)
When trying to curl to individual api servers, it sometimes works and sometimes doesn't: http://
Only successful requests appear in glance's log file.
tcpdump shows that client and server perform a tcp handshake, then client passes a http request to which server replies with an ACK.
The issue can be fixed by restarting affected service. Affected services seem to include Glance, Nova and Neutron.
