object-server and object-replicator not robust to zero-byte hashes.pkl files
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Fix Released
|
Undecided
|
Darrell Bishop |
Bug Description
This is with swift 1.7.6-dev at commit 13937ad69689389
Dec 11 14:21:39 object-server ERROR __call__ error with REPLICATE /d25/206982 :
Traceback (most recent call last):
File "/usr/lib/
res = method(req)
File "/usr/lib/
return func(*a, **kw)
File "/usr/lib/
resp = func(ctrl, *args, **kwargs)
File "/usr/lib/
_junk, hashes = tpool_reraise(
File "/usr/lib/
raise resp
RuntimeError: maximum recursion depth exceeded
root@swift-
total 1812
drwxr-xr-x 2 swift swift 35 2012-12-04 17:23 ./
drwxr-xr-x 49316 swift swift 1241088 2012-12-04 20:00 ../
-rw------- 1 swift swift 0 2012-11-01 19:31 hashes.pkl
-rwxr-xr-x 1 swift swift 0 2012-09-13 00:00 .lock*
176 hashed = 0
177 hashes_file = join(partition_dir, HASH_FILE)
178 modified = False
179 hashes = {}
180 mtime = -1
181 try:
182 with open(hashes_file, 'rb') as fp:
183 hashes = pickle.load(fp)
184 mtime = os.path.
185 except Exception:
186 do_listdir = True
187 if do_listdir:
188 for suff in os.listdir(
189 if len(suff) == 3:
190 hashes.
191 modified = True
192 hashes.
193 for suffix, hash_ in hashes.items():
194 if not hash_:
195 suffix_dir = join(partition_dir, suffix)
196 try:
197 hashes[suffix] = hash_suffix(
198 hashed += 1
199 except PathNotDir:
200 del hashes[suffix]
201 except OSError:
202 logging.
203 modified = True
204 if modified:
205 with lock_path(
206 if not os.path.
207 os.path.
208 write_pickle(
209 hashes, hashes_file, partition_dir, PICKLE_PROTOCOL)
210 return hashed, hashes
211 return get_hashes(
212 reclaim_age)
213 else:
214 return hashed, hashes
With a zero-byte "hashes_file", line 183 raises an exception; mtime remains -1, and do_listdir is set to True.
"modified" will be set to True.
Line 206 will get evaluated, but the conditional will be False (the file *does* exist and its mtime will not be == -1).
Then, line 211 recurses causing infinite recursion and raising the above RuntimeError.
You could solve this two ways:
1) assign to mtime before trying to load the pickle data (move line 184 above line 182).
2) have the conditional on line 206 include a test for os.path.getsize() == 0.
Changed in swift: | |
milestone: | none → 1.7.6 |
status: | Fix Committed → Fix Released |
The code in question is formatted better here: http:// paste.openstack .org/raw/ 6iSCAkxpM0Uerp0 Jg8J5/
Also, the file in question is swift/obj/ replicator. py.