Use Python 3.3 os.posix_fadvise()
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
FileStore |
Fix Released
|
High
|
Jason Gerard DeRose |
Bug Description
Python 3.3 is going to be awesome. Among the new awesomeness is os.posix_advise():
http://
This is great for filestore as mostly what it does is sequential passes through entire files (importing, copying from one store to another, verifying, etc). This should allow us to get similar (or even higher) read IO utilization without using such deep a queue, which will reduce memory usage during imports (always nice).
My hunch is we'll still get the best results with 2 threads, but pehaps the work should be reorganized a bit so that the producer thread reads and hashes, while the consumer thread writes and saves metadata to CouchDB.
Currently the producer just reads, while the consumer hashes, writes, and saves metadata. This was needed so that the producer would *always* be reading during an import so we could get the read IO utilization up very near the theoretical maximum (which we have).
Python 3.3 is due out August 2012, so we should be able to land this in the Novacut 12.08 release.
Related branches
- James Raymond: Approve
-
Diff: 208 lines (+68/-24)2 files modifiedbenchmark.py (+25/-22)
filestore.py (+43/-2)
Changed in filestore: | |
milestone: | 12.06 → 12.07 |
Changed in filestore: | |
milestone: | 12.07 → 12.08 |
Changed in filestore: | |
milestone: | 12.08 → 12.09 |
Changed in filestore: | |
milestone: | 12.09 → 12.10 |
Changed in filestore: | |
milestone: | 12.10 → 12.11 |
Changed in filestore: | |
status: | In Progress → Fix Committed |
Changed in filestore: | |
status: | Fix Committed → Fix Released |
I re-targeted this for 12.06 as Python 3.3 beta1 comes out in June, so that would a good time to implement this and start serious testing with Python 3.3.
http:// www.python. org/dev/ peps/pep- 0398/#id2
The fallback for Python 3.2 will be to not use fadvise, same as we do currently.