Comment 9 for bug 1909861

Revision history for this message
maksis (maksis) wrote :

I ran some additional tests with different read buffer sizes on different platforms. Cached hashing speed on macOS is CPU-bound and the same most likely applies to Windows results as well (both disks are able to read 1000 MB/s+). 1024 KB buffer size looks quite good based on the tests.

macOS 11 (MacBook Pro, SSD)

256 KB:

Cached: 2 files (20.76 GiB) have been hashed in 59 seconds (357.97 MiB/s)
F_NOCACHE: 2 files (20.76 GiB) have been hashed in 1 minute 54 seconds (186.08 MiB/s)

1024 KB:

Cached: 2 files (20.76 GiB) have been hashed in 1 minute 0 second (354.01 MiB/s)
F_NOCACHE: 2 files (20.76 GiB) have been hashed in 1 minute 17 seconds (275.21 MiB/s)

4096 KB:

Cached: 2 files (20.76 GiB) have been hashed in 59 seconds (355.90 MiB/s)
F_NOCACHE: 2 files (20.76 GiB) have been hashed in 1 minute 6 seconds (320.85 MiB/s)

Ubuntu 20.04 (Celeron N3150, internal HDD)

256 KB:

Cached: 671 files (21.70 GiB) in 9 directories have been hashed in 3 minutes 58 seconds (93.08 MiB/s)
O_DIRECT: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 10 seconds (88.62 MiB/s)

1024 KB:

POSIX_FADV_DONTNEED: 671 files (21.70 GiB) in 9 directories have been hashed in 3 minutes 55 seconds (94.55 MiB/s)
O_DIRECT: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 1 second (91.95 MiB/s)

4096 KB:

POSIX_FADV_DONTNEED: Hashing finished: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 58 seconds (74.42 MiB/s)
O_DIRECT: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 55 seconds (75.24 MiB/s)

Windows 10 (AMD 2700X, SSD)

OVERLAPPED, 256 KB: 191 files (21,36 GiB) have been hashed in 39 seconds (551,41 MiB/s)
OVERLAPPED, 1024 KB: 191 files (21,36 GiB) have been hashed in 37 seconds (577,62 MiB/s)
OVERLAPPED, 4096 KB: 191 files (21,36 GiB) have been hashed in 37 seconds (581,33 MiB/s)