use larger chunks to avoid write amplification
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Snapcraft |
Invalid
|
Medium
|
Sergio Schvezov |
Bug Description
https:/
This pull included some code that operated on kilobyte-sized chunks of data:
+ with open(destination, 'wb') as destination_file:
+ for buf in request_
+ destination_
Because Advanced Format disks use 4k sectors and SSDs typically use 8K sectors (I've heard 2K-32K before), this may introduce write amplifications. Hopefully the OS can coalesce these small writes but we could avoid the issue entirely by using larger write(2) sizes. Most disks appear happiest writing 128K blocks, but even 8K would be an improvement for many environments.
Thanks
Changed in snapcraft: | |
status: | New → Confirmed |
importance: | Undecided → Medium |
Changed in snapcraft: | |
assignee: | nobody → Sergio Schvezov (sergiusens) |
milestone: | none → 2.28 |
status: | Confirmed → Triaged |
Changed in snapcraft: | |
milestone: | 2.28 → none |
I've looked into this a bit more. Reading the python docs for open, where we do _not_ supply the `buffering` parameter:
buffering is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable in text mode), and an integer > 1 to indicate the size in bytes of a fixed-size chunk buffer. When no buffering argument is given, the default buffering policy works as follows:
- Binary files are buffered in fixed-size chunks; the size of the buffer is chosen using a heuristic trying to determine the underlying device’s “block size” and falling back on io.DEFAULT_ BUFFER_ SIZE. On many systems, the buffer will typically be 4096 or 8192 bytes long.
- “Interactive” text files (files for which isatty() returns True) use line buffering. Other text files use the policy described above for binary files.
Thanks to Python's default of buffering with a heuristic to determine the underlying device's block size, I think this bug is invalid. Please let me know if you disagree.