Add statvfs() function to be used by dmedia importer

Bug #863822 reported by Jason Gerard DeRose
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
FileStore
Fix Released
High
Jason Gerard DeRose

Bug Description

FileStore.statvfs() returns a StatVFS namedtuple with:

 * size - fs size in bytes
 * avail - free space in bytes available to non-privileged users
 * used - space in bytes used by existing files

We want similar information to be included in the import document, but this time the statvfs() info for the card being imported. The reason for this is a strengthens the audit trail and allows us to do better sanity checks... in particular before deciding it's okay to format a card at the end of an import.

The StatVFS.used attribute will give us the approximate total bytes we expect to be imported, which will allow us to do a good sanity check on whether iter_files() actually found all the files.

Needed changes:

 * Add Batch.statfvs attribute to Batch namedtuple (this will be 4th argument)

 * Changed scandir() to get statfvs info for basedir before calling iter_files()... this should be done the same way FileStore.statvfs() dose it

 * Add a sanity check to scandir()

As far as the sanity check, it may depend a bit on the exact file system and its state, but in theory this should hold true:

batch = scandir(basedir)
st = os.statvfs(basedir)
used = (st.f_blocks * st.f_frsize) - (st.f_bfree * st.f_frsize)
assert (used - batch.count * st.f_frsize) <= batch.size <= used

Related branches

Revision history for this message
Jason Gerard DeRose (jderose) wrote :

This is probably a bit clearer way to state the above assertion

batch = scandir(basedir)
st = os.statvfs(basedir)
high = (st.f_blocks - st.f_bfree) * st.f_frsize
low = (st.f_blocks - st.f_bfree - batch.count) * st.f_frsize
assert low <= batch.size <= high

Revision history for this message
Jason Gerard DeRose (jderose) wrote :

Ah, actually, the sanity check should not be inside statfvs()... higher level code must decide when to do this because the assertion only holds true when you run scandir() on a mount point (in other words, when you import all files on a partition, like an entire card).

Changed in filestore:
status: Triaged → In Progress
assignee: nobody → Jason Gerard DeRose (jderose)
Revision history for this message
Jason Gerard DeRose (jderose) wrote :

After thinking about it more, I decided that having scandir() include the statvfs() info was the best idea as it overloads scandir() a bit to much, plus there are places where scandir() would be used but the statvfs info isn't needed (migrations come to mind).

So instead, I'm just adding a filestore.statvfs() function, which will be used by both FileStore.statvfs() and dmedia importer.

summary: - scandir() should include os.statvfs() info for card being imported
+ Add statvfs() function to be used by dmedia importer
Changed in filestore:
status: In Progress → Fix Committed
Changed in filestore:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.