Duplicity Hangs intermittently when using multiprocess uploads to S3
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Duplicity |
Fix Released
|
Medium
|
Unassigned |
Bug Description
Python Version 2.7.5
Duplicity Latest Dev (from my s3-glacier branch) Though I believe this issue exists in 0.6.22
OS: Arch Linux (64-bit) fully updated
Randomly during full backups duplicity hangs. My usual incremental backup has under 100 volumes, I've seen the hangups starting to occur anywhere from volume 300 all the way to volume 8000.
Options I give:
/bin/duplicity --full-
I added additional logging and saw that the multiprocess pool is spawned and the main process is waiting for the pool to finish, although the pool processes never start. Looks like they all hang while waiting to access the internal Queue of the pool. I found other people reporting similar behavior with Python: http://
When I issue a `kill -2` to the processes that spawned, I get the following stack trace:
Process PoolWorker-1209:
Process PoolWorker-1208:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/
File "/usr/lib/
Process PoolWorker-1210:
Traceback (most recent call last):
Process PoolWorker-1207:
File "/usr/lib/
Traceback (most recent call last):
File "/usr/lib/
self.run()
File "/usr/lib/
self.run()
self.run()
self.run()
self.
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
self.
File "/usr/lib/
self.
self.
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
task = get()
File "/usr/lib/
task = get()
File "/usr/lib/
task = get()
File "/usr/lib/
task = get()
File "/usr/lib/
racquire()
racquire()
KeyboardInterrupt
return recv()
KeyboardInterrupt
KeyboardInterrupt
racquire()
KeyboardInterrupt
Related branches
- edso: Needs Information
-
Diff: 998 lines (+321/-416)8 files modifiedREADME (+1/-0)
bin/duplicity (+11/-0)
bin/duplicity.1 (+38/-0)
duplicity/backends/_boto_multi.py (+75/-308)
duplicity/backends/_boto_single.py (+171/-104)
duplicity/backends/botobackend.py (+11/-4)
duplicity/commandline.py (+8/-0)
duplicity/globals.py (+6/-0)
Changed in duplicity: | |
milestone: | none → 0.6.24 |
status: | New → Fix Committed |
importance: | Undecided → Medium |
Changed in duplicity: | |
status: | Fix Committed → Fix Released |