Critical: fix non-convergent scenario
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Dmedia |
In Progress
|
Critical
|
Jason Gerard DeRose |
Bug Description
I'm still sorting out the details, but I've found a non-convergent scenario, a case where Dmedia wont get all the user files up to 3 copies, even though there is sufficient storage available.
This is what led to the non-convergent scenario:
1) I started with a Dmedia library containing 8 drives but insufficient storage (in my case, there were still several hundred files with only 2 copies due to lack of storage); I believe this scenario can only occur if you have at least 4 drives, but that needs more investigation still
2) I added a new drive, to bring the total storage in the library up to an amount that will allow for 3 copies of all the user files
3) I manually purged a few smaller drives that I needed to re-purpose, although I'm not sure this was necessary to create the non convergent scenario... AFAIK, the important thing is that all the previous drives in the library were full before I added the new drive
I'm now in a situation where Dmedia is stuck with hundreds of files that have only 2 copies, even though there is nearly 3 TB of free space still on my new drive. The key is that there is already a copy of all these fragile files on the new drive, yet the new drive is the only one with any available space. So what needs to happen is some non-fragile files need to be brought up to 4 copies (by creating a copy on the new drive) so that space can be reclaimed on some of the full drives that don't already contain a copy of the fragile files.
I'm working on coming up with a good model of the scenario so we can build unit tests for it, etc.
Ideally we can fix this with a small tweak to the existing copy increasing behavior rather than adding an entirely new special case behavior, but I'm not sure on that yet.
Although this is a very serious bug, from what I understand so far, this bug couldn't itself cause data loss. But it does mean Dmedia would be stuck in a state where there is less statistical breathing room than we want, where Dmedia isn't as resilient as it should be when it comes to metadata being out of sync between devices, to catastrophic hardware failure, and to human error.
Changed in dmedia: | |
milestone: | 13.11 → 13.12 |
Changed in dmedia: | |
milestone: | 13.12 → 14.01 |
Changed in dmedia: | |
milestone: | 14.01 → 14.02 |
Changed in dmedia: | |
milestone: | 14.02 → 14.03 |
Changed in dmedia: | |
milestone: | 14.03 → 14.04 |
Changed in dmedia: | |
milestone: | 14.04 → 14.05 |
So after having a day to think about this, I do feel a bit silly for not catching this sooner, but that's how it goes.
Although surprising at first glance, this scenario is more strait forward than I first thought. However, it does seem like fixing this really needs a 5th behavior. Dmedia has had 4 automatic background behaviors for a long time:
1) Verification - Dmedia makes sure the metadata matches reality, and that the files have perfect file integrity; this includes MetaStore.scan(), MetaStore.relink(), MetaStore. verify_ by_downgraded( ), MetaStore. verify_ by_mtime( ), and MetaStore. verify_ by_verified( )
2) Downgrading - Dmedia automatically lowers its confidence in the aspects of reality it hasn't been able to verify for as certain amount of time; this includes MetaStore. purge_or_ downgrade_ by_store_ atime() , MetaStore. downgrade_ by_mtime( ), and MetaStore. downgrade_ by_verified( )
3) Copy increasing - when there are user files with less than 3 copies of durability, Dmedia will create new copies on any FileStore (drives) such that at least MIN_FREE_SPACE will still remain after creating the new copy, by either copying the file from one locally connected drive to one or more other locally connected drives, or by downloading the copy from a peer on the local network; this is performed by the vigilance worker, driven by MetaStore. iter_actionable _fragile( )
4) Copy decreasing - when there is a locally connected drive with less than RECLAIM_BYTES free space available, and when there are copies on that drive such that after deleting that copy, the file will still have a durability of 3, those copies are automatically deleted (reclaimed) on that drive, starting with the least recently used file (base on doc.atime)
I'd describe the 5th behavior that I think is probably the best fix to this problem something like this:
5) Shuffling - when a drive in the library has less than RECLAIM_BYTES available and contains files with a durability of 3, and when there is a locally connected drive upon which at least MIN_FREE_SPACE would remain after creating a 4th copy of a file, Dmedia will create new copies of these files by copying them from locally connected drives or downloading them from peers on the local network; this behavior creates the needed scenario under which behavior (4) can reclaim space on the first drive
David Jordan brought up the very good point that we need to consider how file "pinning" effects this. Currently, files are never reclaimed when they're pinned. David also suggested that pinning be a convergence behavior... that pinning doesn't necessarily reflect the current state Dmedia is in, it reflects the state Dmedia should be moving toward, and if needed for data safety reasons, Dmedia might temporarily ignore the user's pinning requests.
Still more thought/ experimentation /testing needed on this, but I think we're making progress.