Swift: No space left on device
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Invalid
|
Undecided
|
Unassigned | ||
Glance |
Invalid
|
Undecided
|
Unassigned | ||
OpenStack Object Storage (swift) |
Invalid
|
Undecided
|
Unassigned | ||
devstack |
Fix Released
|
Undecided
|
Joe Gordon |
Bug Description
Intermittent failures with tempest.
2013-09-15 12:19:50,370 Glance request id req-e344e633-
2013-09-15 12:19:52,372 Request: HEAD http://
2013-09-15 12:19:52,400 Response Status: 200
2013-09-15 12:19:52,400 Glance request id req-c6b0838c-
2013-09-15 12:19:54,403 Request: HEAD http://
2013-09-15 12:19:54,428 Response Status: 200
2013-09-15 12:19:54,428 Glance request id req-46580425-
2013-09-15 12:19:56,430 Request: HEAD http://
2013-09-15 12:19:56,458 Response Status: 200
2013-09-15 12:19:56,458 Glance request id req-de198fae-
2013-09-15 12:19:58,460 Request: HEAD http://
2013-09-15 12:19:58,485 Response Status: 200
2013-09-15 12:19:58,485 Glance request id req-5797d453-
2013-09-15 12:19:59,490 Request: DELETE http://
2013-09-15 12:19:59,560 Response Status: 200
2013-09-15 12:19:59,560 Glance request id req-7f160544-
}}}
Traceback (most recent call last):
File "tempest/
self.
File "tempest/
raise exceptions.
TimeoutException: Request timed out
Details: Time Limit Exceeded! (400s)while waiting for active, but we got killed.
Full logs here: http://
Changed in openstack-ci: | |
milestone: | none → icehouse |
no longer affects: | openstack-ci |
summary: |
- tempest.api.volume.test_volumes_actions.VolumesActionsTestXML flakey - failure + Swift: No space left on device |
Changed in devstack: | |
status: | New → Fix Committed |
assignee: | nobody → Joe Gordon (jogo) |
Changed in glance: | |
status: | New → Invalid |
Changed in swift: | |
status: | New → Invalid |
Changed in devstack: | |
status: | Fix Committed → Fix Released |
glance uses swift backend and you can see the followings in the glance log:
http:// logs.openstack. org/23/ 42523/4/ check/gate- tempest- devstack- vm-full/ 8352d30/ logs/screen- g-api.txt. gz#_2013- 09-15_11_ 59_14_678
http:// logs.openstack. org/23/ 42523/4/ check/gate- tempest- devstack- vm-full/ 8352d30/ logs/screen- g-api.txt. gz#_2013- 09-15_12_ 13_42_773
At same time the swift proxy sent an 500:
http:// logs.openstack. org/23/ 42523/4/ check/gate- tempest- devstack- vm-full/ 8352d30/ logs/syslog. txt.gz precise- hpcloud- az2-285663 proxy-server ERROR 500 Traceback (most recent call last):#012 File "/opt/stack/ new/swift/ swift/obj/ server. py", line 631, in __call__#012 res = method(req)#012 File "/opt/stack/ new/swift/ swift/common/ utils.py" , line 1896, in wrapped#012 return func(*a, **kw)#012 File "/opt/stack/ new/swift/ swift/common/ utils.py" , line 686, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/opt/stack/ new/swift/ swift/obj/ server. py", line 378, in PUT#012 writer. write(chunk) #012 File "/opt/stack/ new/swift/ swift/obj/ diskfile. py", line 301, in write#012 self.threadpool .run_in_ thread( _write_ entire_ chunk, chunk)#012 File "/opt/stack/ new/swift/ swift/common/ utils.py" , line 2151, in run_in_thread#012 return func(*args, **kwargs)#012 File "/opt/stack/ new/swift/ swift/obj/ diskfile. py", line 297, in _write_ entire_ chunk#012 written = os.write(self.fd, chunk)#012OSError: [Errno 28] No space left on device#012 From Object Server re: /v1/AUTH_ 66c403bb38b94b7 4851290e6a83f16 dd/glance/ 5e4d189e- 8de7-428a- 923f-1c1090745b f9 127.0.0.1:6013/sdb1 (txn: tx8c6af44ce8af4 02ba2aa8- 005235a45f)
Sep 15 12:13:41 devstack-
Swift storage size should increased.