Certain OS API services don't work after a successful deployment

Bug #1358658 reported by Dmitry Nikishov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Undecided
Unassigned

Bug Description

{"ostf_sha": "d2a894d228c1f3c22595a77f04b1e00d09d8e463", "auth_required": true, "api": "1.0", "nailgun_sha": "36d27ff737b361f92093986d061bbfc1670bee45", "production": "docker", "fuelmain_sha": "a19c69d1b80abb6dfa5bc8caafb11b32551c2cf8", "astute_sha": "efe3cb3668b9079e68fb1534fd4649ac45a344e1", "feature_groups": ["experimental"], "release": "5.1", "fuellib_sha": "7c3c2175952780bcbeed00f524af3cfbe8c4d290"}

Environment: Ubuntu/HA/VLAN 3x controller/cinder lvm + 1 compute

After puppet runs successfully on all the nodes Astute times out uploading an image to Glance. Log: http://pastebin.com/gEfRMiUk

When trying to glance image-list, most of the time the client just hangs up until haproxy returns HTTP 504, and sometimes it completes successfully:

root@node-2:~# glance image-list
Request returned failure status.
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
 (HTTP N/A)
root@node-2:~# glance image-list
Request returned failure status.
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
 (HTTP N/A)
root@node-2:~# glance image-list
+--------------------------------------+--------+-------------+------------------+----------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+--------+-------------+------------------+----------+--------+
| 94f648eb-80ca-47ba-aa03-a7267610a133 | TestVM | qcow2 | bare | 13167616 | active |
| cfd780f4-faef-467c-93f3-693d7dab8858 | TestVM | qcow2 | bare | 13167616 | active |
+--------------------------------------+--------+-------------+------------------+----------+--------+

Running glance client with --debug reveals that it hangs upon making a request to Glance API:
root@node-2:~# glance --debug image-list
curl -i -X GET -H 'X-Auth-Token: 4338291964f04628b02ab292ce88075b' -H 'Content-Type: application/json' -H 'User-Agent: python-glanceclient' http://192.168.123.2:9292/v1/images/detail?sort_key=name&sort_dir=asc&limit=20

HTTP/1.0 504 Gateway Time-out
connection: close
content-type: text/html
cache-control: no-cache

<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

Request returned failure status.
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
 (HTTP N/A)

When trying to curl to individual api servers, it sometimes works and sometimes doesn't: http://pastebin.com/rk2WtQj2
Only successful requests appear in glance's log file.
tcpdump shows that client and server perform a tcp handshake, then client passes a http request to which server replies with an ACK.
The issue can be fixed by restarting affected service. Affected services seem to include Glance, Nova and Neutron.

description: updated
Changed in fuel:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.