Auto-add with duplicate detection messes with FileTypePlugins

[Florian Bach]

I'm reporting a bug with FileTypePlugins which is probably similar to the fixed bugs 1945889 and the "2nd part" of bug 1992244.

When a book is imported through the auto-add folder and the "Check for duplicates when automatically adding files" option in the Calibre settings is enabled, it does not correctly update the book file type.

So, using my ACSM-to-EPUB plugin as an example again, same as in the last two bugs:

1. I add an ACSM file to the auto-add folder which already exists in my Calibre library.
2. Calibre shows a dialog "do you really want to import this?"
3. I click "Yes"
4. The book gets imported

That's how it should happen. What happens instead is the following:

1. I add an ACSM file to the auto-add folder.
2. Calibre runs all file-type plugins in the background, ignoring and silently discarding any print statements in the plugins that would make debugging easier ...
3. My file type plugin turns that ACSM file into an EPUB file.
4. Calibre notices that a book with that name and author already exists, and asks me if I want to import it anyways.
5. When I click yes, Calibre runs my file type plugin again. As I've learned from previous report(s) that's not something that can be fixed, but this is not the issue.
The issue is that it runs the file type plugin with the file name "dup_cache.acsm" as input, despite the fact that that temporary file "dup_cache" is no longer an ACSM file since it's already been turned into an EPUB by the plugin in step 3.
6. My plugin realizes "Hey, someone gave me an EPUB file that's named ACSM, nothing to do" and returns the path without any changes.
7. Calibre adds the book to the library with the correct metadata - but with the book format "ACSM". The file name in the library is also "Title - Author.acsm" instead of "Title - Author.epub", despite the file itself being a valid EPUB file. Also, any file type plugins specifically for "EPUB" fail to run since Calibre still thinks this is an ACSM file.

So what I believe to be the bug, but that might be wrong, is the following:

- The first run of the FileTypePlugins doesn't log anything, is stdout/stderr not correctly redirected for that duplicate check when adding books from an auto-add folder?
- Despite the fact that in step 3 the file has been turned into an EPUB, it still has the name ".acsm" in step 5 which causes ACSM-file-type plugins to run again, and EPUB-file-type-plugins to be skipped. It should *either* re-run the plugins with the original file (not the modified one returned by plugins in step 3), or it needs to use the correct file name extension as returned by plugins in step 3, not the file type extension the file originally had.

I hope that explaination was detailed enough, it seems like a fairly complex bug. Let me know if I need to explain this again differently or provide an example file.

My vague guess - but I don't know that much about Calibre so forgive me if that suggestion is crap - is that the bugfix you added to the web server here - https://github.com/kovidgoyal/calibre/commit/a32750b4be814fa7b3991941eb7110023f91b522 - would need to be added to the auto-add duplicate detection code as well.

I'm testing this in Calibre 6.10 on Linux, but I have no reason to believe the situation would be different on other OSes.

[Florian Bach (leseratte10)]

Small correction: EPUB-only file type plugins *do* run at step 3 of my list, but also without their print statements (which made me think they didn't run).

To my eyes that makes this even more likely to be very similar to the bug fixed in a32750b4be814fa7b3991941eb7110023f91b522.

[Kovid Goyal (kovid)]

Fixed in branch master. The fix will be in the next release. calibre is usually released every alternate Friday.

 status fixreleased

[Florian Bach (leseratte10)]

Thanks a lot for the quick fix.

When I now import an ACSM file through the auto-add function it does work correctly and it runs all the plugins in the correct order.

Though, no matter if I have the "Check for duplicates" enabled or disabled, the stdout and stderr of the ACSM-to-EPUB-plugin is not included in the debug output (calibre-debug -g). Only the stdout and stderr of additional EPUB file type plugins running after the 1st plugin are printed. Could this be added as well to make debugging plugin errors easier?

[Kovid Goyal (kovid)]

Those are run in a separate worker process for reading metadata.
Forwarding the output from there is a bit too much work for this use
case for me, but patches are welcome.