Upgrading to 1.5.0 dropped a lot of pages

[Lorenzo Miniero]

My Fedora 28 laptop recently had an upgrade of lifeograph to 1.5.0, which triggered a notice the DB would be updated, and I proceeded with it. After a few days of usage, I noticed that some info I knew was there couldn't be found anymore, and I verified today that a lot of pages seem to be missing. The pages that were dropped seem to be random, so no idea what triggered it... there's years of useful info from work, there, so I'd really like to keep it.

Is there any way to recover the old pages, in case? I think I found an older version of the diary (if I use that I'm asked to upgrade again, with the same result), any debug information I can collect somehow to help iron out the cause of the issue and fix it? I know that I can try to install an older version and see if the data is still there and stick to that, but that sounds like a poor solution.

Thanks,
Lorenzo

[Ahmet Öztürk (dmxe)]

Thank you for the report.

First of all, at every DB upgrade, Lifeograph saves a backup copy of the last version with a name that equals the original name plus version number. So, your pages (entries?) should be safe unless you deleted that backup file.

Second of all, this kind of problems are very important as preservation of user data is of topmost priority. So, I am really interested in diagnosing and fixing this problem.

The dropped pages should not be random. Especially if you lose the same pages when you upgrade again. There should be a pattern, a common property among the pages that are lost.

Probably the file contains sensitive data and you cannot share it. So, can you do the following:
1) Create a copy the pre-1.5 version of the diary file.
2) Open the copy on Lifeograph 1.4 and delete all the entries except for those that get deleted.
3) Edit the contents of the remaining entries to remove any personal or sensitive data but make sure that entries still have some content, they should not be completely empty.
4) Create a new backup copy of the new file.
5) Open the latest copy in v1.5 to make sure that the entries are still getting deleted on upgrade.
6) Share the not upgraded copy with me to investigate.

[Lorenzo Miniero (lminiero)]

Thanks for the quick answer!

Yes, the backup file is there, as it's the one I used to confirm the issue by trying to upgrade again. I saved it to a separate folder for those additional tests, in order to be sure it wouldn't be lost, so good to know the data is still there.

With respect to your suggestion, it won't be easy. I've been keeping a diary with lifeograph pretty much every work day since 2011, so there's *A LOT* of pages: it would be very hard, if not impossible, to remove all entries except those that are deleted. I know many pages have been deleted because I noticed some missing patterns (e.g., events that I knew had 8 entries in a row with the same title, but now only have 4), but identifying them in ~2500 diary pages would be a titanic effort.

What I can do is try create a new small diary that only contains 4-5 of the pages that I know are not in the upgraded diary, and a few that are, and try to upgrade that one. In that case, it should end up with only the pages that remain in the original diary as well. Would that work and give you enough to work on?

If there's any investigation I can do myself, e.g., logging that happens somewhere during the upgrade process that I can look at to give a hand, please let me know.

[Ahmet Öztürk (dmxe)]

As long as you can send me a diary with some entries that disappear on upgrade that may help.

My email address is <email address hidden>; you can send the diary as an attachment.

There is no logging during upgrade but you may check the command line output to see if there is any suspicious message there.

[Lorenzo Miniero (lminiero)]

No suspicious command line output, unfortunately...

I just sent you a portion of the diary that fails to upgrade: it's 9 days of diary, where five pages are successfully imported (1/11, 3/11, 6/11, 7/11, 8/11), while four are lost in the process (2/11, 4/11, 5/11, 9/11). I verified these are exactly the same pages that are lost when upgrading the full diary (among others I didn't copy here, of course). Not sure what in them confuses the upgrade process: in some cases there are addresses, partial or complete; or maybe some of the characters I use are now used by lifeograph to identify new features?

Looking forward to your feedback!

[Ahmet Öztürk (dmxe)]

I have opened your diary, and to my great relief seen that no entries are deleted in it. What you experience is not a bug but a feature :)
Let me explain what happens:

In Lifeograph 1.5, todo status of an entry is automatically derived from its content. e.g. Entries that contain todo items in open status are assigned open status, when some items are open, some are done; entry gains "in progress" status, so on and so forth...

So, the enries you report missing only contain "done" items consequently making the entries themselves "done". And default filter setting in Lifeograph hides done entries. That's it.

If you click the diary icon, you will see that the diary still contains 9 entries in total and if you click filter and open done entries, you will be able to see those entries again. You can make this default filtering setting if that suits your use-case better.

[Lorenzo Miniero (lminiero)]

I may be looking in the wrong place but I can't find anything you described.

If I click the application diary icon in the upper left, the only options available are "Preferences", "About Lifeograph" and "Quit": the "Preferences" has very few settings I can tweak (Privacy, Date Format, Appearance), and nothing that mentions filters. The red monochrome icon more on the right just has Import/Export/ChangePassword buttons, so again, no filter; I can see that icon changes when a page is selected, but I can only change the icon for that page, there's no filter it allows me to apply.

The startup page doesn't seem to help either: there's a Spell Checking option that I disabled, a Startup Action selector that doesn't contain anything relevant apparently, and a Current Category in Chapters that is set to Default.

Am I missing something obvious? That said, such a huge change, especially if marked as a feature, actually seems breaking to me without some documentation after the upgrade that explains it properly.

[Lorenzo Miniero (lminiero)]

Edit: found the icon... it was not the diary icon, but the funnel one next to the search box. I chose to show everything and now I see them.

Thanks for the support! I still think this should be explained after upgrading, though, as I'm sure I won't be the only one to be confused, and some will just think the application is broken.

[Ahmet Öztürk (dmxe)]

This was specified in the change log and program manual was updated accordingly (see section 1.15). It did not occur to me that this might confuse users. I am going to add a warning to the web site.