Bulk Polishing fails with 'Access Denied'

Bug #1873949 reported by Calum McConnell on 2020-04-20
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
calibre
Undecided
Unassigned

Bug Description

Whenever I attempt to polish several books with the setting 'add soft hyphens' enabled, a random selection of the books will not polish, firing off this error message:

Polish book 3 of 6 (Cold Springs)
Python function terminated unexpectedly
  [Error 5] Access is denied (Error Code: 1)
 Traceback (most recent call last):
  File "site.py", line 114, in main
  File "site.py", line 88, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 209, in main
  File "site-packages\calibre\ebooks\oeb\polish\main.py", line 291, in gui_polish
  File "site-packages\calibre\ebooks\oeb\polish\main.py", line 269, in polish
  File "site-packages\calibre\ebooks\oeb\polish\main.py", line 258, in polish_one
  File "site-packages\calibre\ebooks\oeb\polish\hyphenation.py", line 16, in add_soft_hyphens
  File "site-packages\calibre\utils\hyphenation\hyphenate.py", line 116, in add_soft_hyphens_to_html
  File "site-packages\calibre\utils\hyphenation\hyphenate.py", line 102, in add_to_tag
  File "site-packages\backports\functools_lru_cache.py", line 152, in wrapper
  File "site-packages\calibre\utils\hyphenation\hyphenate.py", line 27, in dictionary_for_locale
  File "site-packages\calibre\utils\hyphenation\dictionaries.py", line 106, in path_to_dictionary
  File "site-packages\calibre\utils\hyphenation\dictionaries.py", line 71, in extract_dicts
WindowsError: [Error 5] Access is denied

When reproducing the above error, I also incidentally created the error below.

Polish book 0 of 6 (The Demigod Diaries)
Python function terminated unexpectedly
  [Error 183] Cannot create a file when that file already exists (Error Code: 1)
 Traceback (most recent call last):
  File "site.py", line 114, in main
  File "site.py", line 88, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 209, in main
  File "site-packages\calibre\ebooks\oeb\polish\main.py", line 291, in gui_polish
  File "site-packages\calibre\ebooks\oeb\polish\main.py", line 269, in polish
  File "site-packages\calibre\ebooks\oeb\polish\main.py", line 258, in polish_one
  File "site-packages\calibre\ebooks\oeb\polish\hyphenation.py", line 16, in add_soft_hyphens
  File "site-packages\calibre\utils\hyphenation\hyphenate.py", line 116, in add_soft_hyphens_to_html
  File "site-packages\calibre\utils\hyphenation\hyphenate.py", line 102, in add_to_tag
  File "site-packages\backports\functools_lru_cache.py", line 152, in wrapper
  File "site-packages\calibre\utils\hyphenation\hyphenate.py", line 27, in dictionary_for_locale
  File "site-packages\calibre\utils\hyphenation\dictionaries.py", line 106, in path_to_dictionary
  File "site-packages\calibre\utils\hyphenation\dictionaries.py", line 75, in extract_dicts
WindowsError: [Error 183] Cannot create a file when that file already exists

I suspect that there is some sort of data race going on as Calibre tried to polish multiple books in parallel.

See https://manual.calibre-ebook.com/faq.html#i-am-getting-a-permission-denied-error

In this case it looks like an antivirus or something else preventing
calibre from working with files in its cache directory.

 status invalid

Changed in calibre:
status: New → Invalid
Calum McConnell (themageking) wrote :

I disagree with that assesment (though I don't know that you'll see this)
The error persisted when I disabled most of Window's antivirus settings (all the real-time-protection stuff).
In fact, it seemed to get worse: only the last book polished sucseeded. The in question book varried, which to mean implies that it was based soley on the ordering Calibre picked for polishing. However, it getting worse seemed weird.

I suspected some form of data race between the 8 calibre subprocesses (my computer has 4 physical cores and 8 logical: this being an AMD, they named it as a "Octo-Core processor"). To diagnose it, I set the number of calibre worker processes to 1. It completed without a single error (though at the speed of a snail)

Changed in calibre:
status: Invalid → New
Calum McConnell (themageking) wrote :

I figured out how to manually set this to be looked at again: I figured I ought to explain a bit more, and I am sorry for disrupting your process: it feels pretty invasive.

The error only occurs when doing bulk ebook processing, and windows showed me no errors or warnings about possible virus attacks. Windows defender is the only antivirus-like program I permit to run on my computer: and I look at Task Manager often enough to say for certain that another one didn't get installed. I don't even have ransomware protection set up!

Further, I have been running calibre for a while, and ran some really massive ebook-polishing jobs with it. Jobs that involved several hundred books that were in the tens of megabytes. I only recently updated to the 4.0 branch, and that was the first time I saw this error (also, the first time I used this specific option in a bulk processing: but that is a bit beside the point).

It's because of those things that I am almost completely certain that this bug is some sort of thread race: the line encountering the error is attempting to rename a set file (based on method names: I don't really speak advanced python), which strongly implies to me that, as 8 threads attempt to rename one file, windows says "one at a time!" and denies the rest access.

Fixed in branch master. The fix will be in the next release. calibre is usually released every alternate Friday.

 status fixreleased

Changed in calibre:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers