Add quick id function to speed up import dedup

Bug #671949 reported by Jason Gerard DeRose
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Dmedia
Fix Released
High
Akshat Jain

Bug Description

We need a quick dedup to run when importing files that may have already been imported. This is especially important when importing large files (say 4GB MOV) over slow USB2. A good algorithm would probably be something like:

  hash(utf-8 repr of mtime + 1024k + last 1024k)

Corner cases are when total file is less than 1MiB and when less than 2MiB.

This function should go in dmedialib/filestore.py and should have a signature like:

  quick_id(filename=None, fp=None)

When you can pass either a filename or an open file handle.

Related branches

Changed in dmedia:
milestone: none → 0.1
importance: Undecided → High
status: New → Triaged
Akshat Jain (akshatj)
Changed in dmedia:
assignee: nobody → Akshat Jain (ssj6akshat1234)
status: Triaged → In Progress
Akshat Jain (akshatj)
Changed in dmedia:
status: In Progress → Fix Committed
Changed in dmedia:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.