Batch update- remove duplicate links, losing data

[Lex Oulu]

Performed a test run to remove duplicate links, by using Batch Update.
Noticed that in certain unknown conditions, the program will remove not only the duplicate link, but also the original link from the `wt_individuals` table `i_gedcom` field.

This means all occurrences of (for example)1 OBJE @Mxxx@ are removed. The program should of retained one occurrence and remove the duplicates. In this example, the photo will no longer be shown on the individual page /tabs.

I will try to create test cases.

[Lex Oulu (lexoulu)]

In attached file a simple test case, 2 persons = 1 family.

When running the batch update - remove duplicates, after a few cycles of running it
you will see the line

1 OBJE @M8@

will be totally removed = lost data.

[kiwi (kiwi3685-deactivatedaccount)]

Its a pretty extreme case, but yes, I agree it does happen with this file.

Not sure how hard it would be to find the cause, but balancing time spent fixing it, with the value of the plug-in, perhaps we should consider removing it instead?

As I recall it was originally only created to solve a short-term bug in PGV that created the duplicates?

[fisharebest (fisharebest)]

IIRC, the code tries to find duplicates at all levels, and I think the level 2 OBJE followed immediately by level 1 OBJE may be the cause of the problem.

I would first attempt to simplify the function (restrict to level 1 OBJE) records.

[fisharebest (fisharebest)]

The batch update module has another (unrelated) problem. The code to generate the differences (the minimal set of additions and deletions) can take a very long time to run on records with many differences.

I've tried running the batch update on the supplied GEDCOM, but it has taken over an hour and is failing to show the first change.

Because of this, I am unable to run the update once, let alone the "few cycles" that are needed to reproduce the error.

I presume you ran it with "auto-accept", and hence didn't need to see the intermediate results.

Unfortunately, I really need to see the proposed changes at each step...

[Lex Oulu (lexoulu)]

Yes, I think I ran it with auto-accept. It is very sloooooo...Zzzzzzz to run. I tried to debug it, but was just too frustrated to wait for such a long time.
I'd agree with kiwi, that removing the program should be considered, as it could lose data.