File data sometimes silently erased when the tangler fails!!!

[Gregory Crosswhite]

It seems that Leo doesn't always store file data inside the outline when the tangler fails to write the file data --- e.g., when the file pathname does not exist, when undefined sections are present, etc. Instead, sometimes Leo ***marks the file nodes as no longer being saved but silently discards the data so that it won't appear the next time the outline is opened!***

To experience this for yourself, try opening either of the attached files which contain examples of cases where the tangler can fail, and change file to @file. Save and quit Leo, and then open the file again. If you still see the children and content in bug.py, make a trivial change to it (to make bug.py dirty), save, quit, and open it again. Repeat this process a few times and you will eventually see that bug.py no longer has any content! The bug is nondeterministic, so sometimes its contents are lost on the first save (i.e., just after changing file to @file), sometimes it takes many iterations of this process for the contents to be lost.

And this isn't just a strange corner case bug that is unlikely to trip up users in practice, but one that has actually caused me to lose data and have to recreate a file from scratch *twice* so far, which is what led me to investigate what was going on.

[Edward K. Ream (edreamleo)]

On Tue, Jun 14, 2011 at 7:53 PM, Gregory Crosswhite
<email address hidden> wrote:

> It seems that Leo doesn't always store file data inside the outline when
> the tangler fails to write the file data

Many thanks for this report. I'll fix it today, with one or more unit tests.

Edward

[Edward K. Ream (edreamleo)]

On Tue, Jun 14, 2011 at 7:53 PM, Gregory Crosswhite
<email address hidden> wrote:

> To experience this for yourself, try opening either of the attached
> files which contain examples of cases where the tangler can fail, and
> change file to @file.  Save and quit Leo, and then open the file again.
> If you still see the children and content in bug.py, make a trivial
> change to it (to make bug.py dirty), save, quit, and open it again.
> Repeat this process a few times and you will eventually see that bug.py
> no longer has any content!  The bug is nondeterministic, so sometimes
> its contents are lost on the first save (i.e., just after changing file
> to @file), sometimes it takes many iterations of this process for the
> contents to be lost.

I do not believe for a moment that this bug is "nondeterministic".
Rather, it depends on a certain sequence of actions.

At the code level, the bug will happen if and only if forceWrite is
computed to be False in fc.putVnode. I just saw the bug in action: it
involved me saving an "improper" outline several times. I knew I
would lose data when I saw the trace of forceWrite go from True to
False.

I suspect, but do not know for sure, that forceWrite goes to False
because some part of the write logic has cleared root's orphan bit.

Edward

[Edward K. Ream (edreamleo)]

On Wed, Jun 15, 2011 at 11:37 AM, Edward K. Ream <email address hidden> wrote:

> I do not believe for a moment that this bug is "nondeterministic".
> Rather, it depends on a certain sequence of actions.

Here is how to reproduce the bug:

1. Start path-noexist-bug.leo with --no-cache.

2. Change the text of the root node.

3. Save the outline twice.

The first time, a trace in fc.putVnode will report forceWrite = True
and isOrphan = True.

The second time, the trace will report both are False. This is the
proximate cause of the data loss.

So it looks like the *first* save clears the orphan bit improperly,
creating a time bomb.

Edward

[Edward K. Ream (edreamleo)]

On Wed, Jun 15, 2011 at 12:04 PM, Edward K. Ream <email address hidden> wrote:

> 3. Save the outline twice.

BTW, after saving the outline twice, I can "rescue" the outline by
changing the root node *again*, then saving the outline *once*. This
causes forceWrite to become True again, thereby rewriting the outline
with the orphan bit set in the root node.

EKR

[Edward K. Ream (edreamleo)]

On Wed, Jun 15, 2011 at 12:04 PM, Edward K. Ream <email address hidden> wrote:

> So it looks like the *first* save clears the orphan bit improperly,
> creating a time bomb.

We can now see the dog that isn't barking: no write error is issued
for the second write, and I suppose you could say that as a result
v.setOrphan isn't called. Alas, v.clearOrphan *is* called, which is
why we are having problems.

Perhaps disabling the call to a.error was supposed to be more calm,
and it does seem pedantic to keep issuing the same error message over
and over. I think the solution is simply to keep setting the orphan
bit :-)

Edward

[Edward K. Ream (edreamleo)]

On Wed, Jun 15, 2011 at 12:19 PM, Edward K. Ream <email address hidden> wrote:

> We can now see the dog that isn't barking: no write error is issued
> for the second write, and I suppose you could say that as a result
> v.setOrphan isn't called.  Alas, v.clearOrphan *is* called, which is
> why we are having problems.

The fix is now on the trunk at rev 4379. All unit tests pass, include
a fairly weak new test.

Happily, the fix itself is quite strong: at.clearAllOrphanBits now
retains the orphan bits of all @<file> nodes. This is the one and
only place where the orphan bit got cleared improperly, so we should
be good to go.

Thanks again, Gregory, for reporting this bug. It was indeed very serious.

Edward