ebook-convert hangs on rendering (possible packaging issue)

Bug #1857377 reported by Zachary Waldowski on 2019-12-23
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
calibre
Undecided
Unassigned

Bug Description

Version: ebook-convert (calibre 4.6.0)
Operating System: macOS 10.15.2 (19C57); appears to occur on other macOS
Input file: attached

I'm trying to convert an unzipped EPUB to PDF via `ebook-convert`. It gets as far as rendering the HTML contents, and then hangs:

[snip]
Creating PDF Output...
67% Running PDF Output plugin
Converting input as a text based book...
68% Parsed all content for markup transformation
70% Completed markup transformation
[1223/152751.093745:ERROR:icu_util.cc(177)] Invalid file descriptor to ICU data received.
[1223/152751.093776:ERROR:icu_util.cc(177)] Invalid file descriptor to ICU data received.
[1223/152751.093815:ERROR:icu_util.cc(177)] Invalid file descriptor to ICU data received.
[1223/152751.093860:ERROR:icu_util.cc(177)] Invalid file descriptor to ICU data received.
[1223/152751.433137:ERROR:icu_util.cc(177)] Invalid file descriptor to ICU data received.
[1223/152751.435393:ERROR:icu_util.cc(177)] Invalid file descriptor to ICU data received.
[1223/152751.437799:ERROR:icu_util.cc(177)] Invalid file descriptor to ICU data received.
[1223/152751.438585:ERROR:icu_util.cc(177)] Invalid file descriptor to ICU data received.

QtWebEngineProcess never appears to run. ebook-convert appears to run indefinitely in the Qt run loop. I didn't get any farther with debugging it than that.

The ICU data bit doesn't appear to be a red herring, there's prior art here with f.ex., https://github.com/pyinstaller/pyinstaller/pull/3233. I see icudtl.dat in the application bundle, but it appears the command-line tools don't know how to get to it.

Let me know if there's anything else I can help with debugging! Thanks.

Zachary Waldowski (zwaldowski) wrote :

Which ebook-convert are you using? The one you should run is
/Applications/calibre.app/Contents/MacOS/ebook-convert

Conversions to PDf work fine for me with that.

 status invalid

Changed in calibre:
status: New → Invalid
Zachary Waldowski (zwaldowski) wrote :

Hi, sorry about the confusion. ebook-convert was symlinked into my PATH. Is there a reason for that fairly common practice on Unix systems not to work? I'm running into a number of issues with third-party code that scripts ebook-convert using, i.e., `subprocess.run` or equivalent. It's certainly a regression from 3.x, which those scripts functioned using.

Zachary Waldowski (zwaldowski) wrote :

More to the point, I was relying on the symlinking that happened automatically here: https://github.com/Homebrew/homebrew-cask/blob/master/Casks/calibre.rb#L17-L36

And suffice to say, if the tools must be run from an absolute path, it should have a better error message?

Are you confirming that using the actual location was successful?
Because it seems like an unlikely problem.

Kovid's suggestion was based on the fact that there are multiple
locations in the archive, IIRC, which contain things that look like a
collection of all the command-line entry points, but only one of those
directories is correct. (Others may be e.g. second-level wrappers.)

It's actually baked into the linux installer to symlink these to the
PATH, by the way. Common Unix practice FTW. ;)

Sorry, I have limited patience to deal with all the special snowflake
problems Apple keep causing. PDF conversion requires Qt WebEngine and
that is sandboxed on macOS using Apple's bundle based sandboxing, I
assume. And presumably that sandbox does not work when launched via
symlink because it uses the path of the executable to establish its
sandbox. Fixing this in calibre would require having the command line
executables re-exec themselves with an absolute path, which is a waste
of time and should be completely un-necessary. Apple should fix their
software to work with symlinks.

Use an absolute path in your scripts, it's as simple as replacing calls
to ebook-convert with os.path.realpath(shutil.which('ebook-convert'))

Kovid Goyal (kovid) wrote :

Oh and if you cannot fix the scripts, just create a script launcher for ebook-convert in PATH that calls the real ebook-convert using an absolute PATH.

Fixed in branch master. The fix will be in the next release. calibre is usually released every alternate Friday.

 status fixreleased

Changed in calibre:
status: Invalid → Fix Released
Zachary Waldowski (zwaldowski) wrote :

Hey, thanks! Us snowflakes really appreciate it.

Since you mentioned sandboxing, it does appear there is a deeper problem at work:

```
~ cd /Applications/calibre.app/Contents/Frameworks
Frameworks spctl -a -vvvv QtWebEngineCore.framework/
QtWebEngineCore.framework/: rejected (invalid destination for symbolic link in bundle)
origin=Developer ID Application: Kovid Goyal (NTY7FVCEKP)
Frameworks ls -la QtWebEngineCore.framework/Versions/Current/Resources/
total 8
drwxr-xr-x@ 9 zw admin 288 Dec 12 15:32 .
drwxr-xr-x@ 6 zw admin 192 Dec 12 15:34 ..
-rw-r--r--@ 1 zw admin 631 Sep 5 11:47 Info.plist
lrwxr-xr-x 1 zw admin 89 Dec 12 15:44 icudtl.dat -> ../../../../../ebook-viewer.app/Contents/ebook-edit.app/Contents/SharedSupport/icudtl.dat
lrwxr-xr-x 1 zw admin 113 Dec 12 15:44 qtwebengine_devtools_resources.pak -> ../../../../../ebook-viewer.app/Contents/ebook-edit.app/Contents/SharedSupport/qtwebengine_devtools_resources.pak
lrwxr-xr-x 1 zw admin 98 Dec 12 15:44 qtwebengine_locales -> ../../../../../ebook-viewer.app/Contents/ebook-edit.app/Contents/SharedSupport/qtwebengine_locales
lrwxr-xr-x 1 zw admin 104 Dec 12 15:44 qtwebengine_resources.pak -> ../../../../../ebook-viewer.app/Contents/ebook-edit.app/Contents/SharedSupport/qtwebengine_resources.pak
lrwxr-xr-x 1 zw admin 109 Dec 12 15:44 qtwebengine_resources_100p.pak -> ../../../../../ebook-viewer.app/Contents/ebook-edit.app/Contents/SharedSupport/qtwebengine_resources_100p.pak
lrwxr-xr-x 1 zw admin 109 Dec 12 15:44 qtwebengine_resources_200p.pak -> ../../../../../ebook-viewer.app/Contents/ebook-edit.app/Contents/SharedSupport/qtwebengine_resources_200p.pak
```

That's definitely considered outside of the sandbox. It's not really any special applesauce; chroot or containers on Linux could conceivably run into similar problems. I don't know what the rationale is for ebook-edit.app having its own copies; since it also relaunches the main binary, it seems like one copy can live inside `QtWebEngineCore`.

Anyway, if you're happy with the relaunching way, that's wonderful! Thanks you so much, again, for all your hard work, and have a happy holidays.

No there is no deeper issue, run spctl on the top level bundle and it
will work fine. And its made that way precisely because of the
combination of sandboxing and the way bundles work in apple land. This
problem absolutely does not exist with chroot.

ebook-viewer and ebook-edit need to be sub-bundles with their own
Info.plist so that they can be used independently and also so that they
show correct icons in the dock. Bundles + codesigning require their own
individual executables. That is why there are executables named, for
example: ebook-viewer-placeholder-for-codesigning

However the bundles cannot duplicate webengine since that would blow up
their size. Which means there can be only one real copy of WebEngine, in
the outermost bundle, with symlinks in inner bundles. And the Webengine
sandbox restricts itself to the bundle the current executable
is running from, determined by argv[0] of all the stupid things.
Which means it needs to be inside the ebook-edit bundle
that is inside the ebook-viewer bundle that is inside the calibre
bundle. So that it can be used from calibre, ebook-viewer and
ebook-edit.

This architecture is totally ridiculous. And no I am not happy about it.
Bundles, code signing and the absolute joke that is notarization, are
all clearly inferior technologies that force insane contortions on
application developers, for zero real benefit. All this crap is badly
designed, horribly documented and changes from macOS release to macOS
release. I wasted two weeks of my life because of it. Just look at the
code in the calibre source tree for creating these bundles, signing and
notarizing them, piles and piles of un-dcumented black magic, ugh.

And that is not even to mention the ridiculous bugs in macOS, like
https://bugs.launchpad.net/calibre/+bug/1855515
https://discussions.apple.com/thread/250717809

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments