Distribution file for 6.12 missing calibre_msgpack files

[Guido Falsi]

Hi,

I'm the maintainer of the calibre FreeBSD port.

I was trying to update the port to version 6.12 and I was getting this error:

```
Traceback (most recent call last):
  File "/wrkdirs/usr/ports/deskutils/calibre/work/calibre-src-6.12.0/setup.py", line 119, in <module>
    sys.exit(main())
  File "/wrkdirs/usr/ports/deskutils/calibre/work/calibre-src-6.12.0/setup.py", line 104, in main
    command.run_all(opts)
  File "/wrkdirs/usr/ports/deskutils/calibre/work/calibre-src-6.12.0/setup/__init__.py", line 239, in run_all
    self.run_cmd(self, opts)
  File "/wrkdirs/usr/ports/deskutils/calibre/work/calibre-src-6.12.0/setup/__init__.py", line 233, in run_cmd
    cmd.run(opts)
  File "/wrkdirs/usr/ports/deskutils/calibre/work/calibre-src-6.12.0/setup/install.py", line 153, in run
    self.run_postinstall()
  File "/wrkdirs/usr/ports/deskutils/calibre/work/calibre-src-6.12.0/setup/install.py", line 180, in run_postinstall
    PostInstall(self.opts, info=self.info, warn=self.warn,
  File "/wrkdirs/usr/ports/deskutils/calibre/work/calibre-src-6.12.0/src/calibre/linux.py", line 758, in __init__
    scripts = msgpack_loads(P('scripts.calibre_msgpack', data=True))
  File "/wrkdirs/usr/ports/deskutils/calibre/work/calibre-src-6.12.0/src/calibre/utils/resources.py", line 89, in get_path
    with open(fpath, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/wrkdirs/usr/ports/deskutils/calibre/work/calibre-src-6.12.0/resources/scripts.calibre_msgpack'
*** Error code 1
```

Investigating it I noticed that the missing file (resources/scripts.calibre_msgpack) used to be included in the distribution and is not included anymore, like all other calibre_msgpack files.

Is this intentional? Or maybe the files were missed in the new distribution?

If it is intentional what would be the correct way to produce them? I tried to use `setup.py resources` but it fails with an icu related error.

Thanks in advance.

[Kovid Goyal (kovid)]

Those are generated files, they will be built when building calibre from
source. Instructions for building from source are at
https://calibre-ebook.com/download_linux

If you are facing some problems with those instructions post the actual
error essage and I might be able to help.

[Guido Falsi (madpilot78)]

Thanks a lot for your feedback.

The FreeBSD port is running `setup.py build` and `setup.py install` separately since the FreeBSD ports system requires the two steps to be separated.

Unluckily the files are not being generated by the build phase.

I'm now playing with the build system to try to adapt it to the FreeBSD ports system to be able to generate all the needed files.

Thanks for your help for now, I hope I can figure things out myself from now on, but I'll write if I need more help.

[Eli Schwartz (eschwartz)]

For the record, you need to run the "gui" subcommand after the "build" one.

See https://github.com/kovidgoyal/calibre/blob/master/INSTALL.rst#build

(The install command depends on build && gui and will recheck them to make sure they were run. To guarantee the build stage is successfully separated from the install stage, you must run both. I don't know whether accidentally running some stuff as the install stage could potentially mess up the FreeBSD packaging environment.)

[Guido Falsi (madpilot78)]

Thanks for the suggestion, I'll try that.

The build system is not going to be messed up, since everything is run in a jail on the package builder, but the build system blocks downloads during the build and install stages.

I noticed calibre tries to download a few things, so I'll need to work around that, providing it the required files.

Before 6.12 many of these files were bundled with the distribution so we were skipping this part. Anyway if this is how things are done now, I'll adapt the port.

[Jhonny Oliveira (jhonny-oliveira)]

Hi!
I have made a couple of tweaks:
- downloaded the dictionaries into the package
- patched hypenization.py not to cleanup every time and allow this to work in advance:
  setup.py hyphenation --hyphenation-url="file://$(CURDIR)/debian/dictionaries/master.tar.gz"
- appended the following steps after build
  setup.py iso639
  setup.py iso3166
  setup.py translations
  setup.py gui
  setup.py resources

I'm able to build Calibre, but this test fails with the following error:

setup.py test --test-name=dom_load

test_dom_load (calibre.scraper.simple.find_tests.<locals>.TestSimpleWebEngineScraper) ... ERRORTraceback (most recent call last):
  File "/<<PKGBUILDDIR>>/setup/run-calibre-worker.py", line 14, in <module>
    sys.exit(main())
  File "/<<PKGBUILDDIR>>/src/calibre/utils/ipc/worker.py", line 196, in main
    exec(sys.argv[-1])
  File "<string>", line 1, in <module>
  File "/<<PKGBUILDDIR>>/src/calibre/scraper/simple.py", line 28, in worker_main
    s = SimpleScraper(source)
  File "/<<PKGBUILDDIR>>/src/calibre/scraper/simple_backend.py", line 56, in __init__
    profile = create_profile(source)
  File "/<<PKGBUILDDIR>>/src/calibre/scraper/simple_backend.py", line 34, in create_profile
    ans.setHttpUserAgent(random_common_chrome_user_agent())
  File "/<<PKGBUILDDIR>>/src/calibre/utils/random_ua.py", line 44, in random_common_chrome_user_agent
    return choose_randomly_by_popularity(tuple(common_chrome_user_agents()))
  File "/<<PKGBUILDDIR>>/src/calibre/utils/random_ua.py", line 30, in common_chrome_user_agents
    for x in user_agent_data()['common_user_agents']:
  File "/<<PKGBUILDDIR>>/src/calibre/utils/random_ua.py", line 14, in user_agent_data
    P('user-agent-data.json', data=True, allow_user_override=False))
  File "/<<PKGBUILDDIR>>/src/calibre/utils/resources.py", line 89, in get_path
    with open(fpath, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/<<PKGBUILDDIR>>/resources/user-agent-data.json'
 [0.5 s]

Any idea how to overcome this? Did I miss something?

Thank you!

[Eli Schwartz (eschwartz)]

You shouldn't need to patch it to accept a file url, the ReVendor framework I contributed to calibre supports automatically handling command line options for specifying an unpacked source directory -- I needed that so I could do my own distro packaging for calibre.

But also, you shouldn't need to constantly rerun that, why are you doing so?

Is this for the official Debian packaging of calibre? It shouldn't be needed...

Why are you recompiling lots of resources that already ship in the dist tarballs? Are you building from a calibre source checkout? Why not run the bootstrap command as discussed in the link I provided?

[Jhonny Oliveira (jhonny-oliveira)]

If you compare both mathjax.py and hyphenation.py you will understand why the patch is needed. As opposed to mathjax, hyphenization always cleans itself in the beggining.

I'm packaging Calibre for xtradeb.net (unofficial). Usually, my package is kept very closely to the official one (Debian), but gets updated more frequently. Unless, I'm struggling a bit, like now. :-)

There are a lot of files that are no longer being shipped by default in the release tarball. Therefore, I decided to run all install subcommands to get them in place. I realized now, that the gui does not seem to be required, as it is automatically executed during setup.py install.

I started from here; https://launchpad.net/~xtradeb/+archive/ubuntu/apps/+sourcefiles/calibre/6.11.0+dfsg-1~xtradeb1/calibre_6.11.0+dfsg-1~xtradeb1.debian.tar.xz

These are the changes I made to the previous recipe:

$cat debian/patches/0025-Do-not-download-hyphenation-if_already-present.patch
Index: calibre-6.12.0+dfsg/setup/hyphenation.py
===================================================================
--- calibre-6.12.0+dfsg.orig/setup/hyphenation.py 2023-02-10 15:31:50.728763069 +0100
+++ calibre-6.12.0+dfsg/setup/hyphenation.py 2023-02-10 16:16:19.105858295 +0100
@@ -87,7 +87,16 @@
     DOWNLOAD_URL = 'https://github.com/LibreOffice/dictionaries/archive/%s.tar.gz' % VERSION
     CAN_USE_SYSTEM_VERSION = False

+ def already_present(self):
+ manifest = self.j(self.vendored_dir, 'locales.json')
+ if os.path.exists(manifest):
+ return True
+ return False
+
     def run(self, opts):
+ if self.already_present():
+ self.info('hyphenation dictionaries already present in the resources directory, not downloading')
+ return
         self.clean()
         os.makedirs(self.vendored_dir)
         with self.temp_dir() as dl_src, self.temp_dir() as output_dir:

$ diff {../calibre-6.11.0+dfsg/,}debian/rules
40a41
> $(SETUP) hyphenation --hyphenation-url="file://$(CURDIR)/debian/dictionaries/master.tar.gz"
48a50,54
> [ -d build ] && $(SETUP) iso639
> [ -d build ] && $(SETUP) iso3166
> [ -d build ] && $(SETUP) translations
> # [ -d build ] && $(SETUP) gui
> [ -d build ] && $(SETUP) resources
60c66
< env LANGUAGE= LC_ALL=C LC_CTYPE=C LC_MESSAGES=C LANG=C TZ=UTC $(CI_ENV) $(SETUP) test --test-name=dom_load $(TEST_FLAGS)
---
> # env LANGUAGE= LC_ALL=C LC_CTYPE=C LC_MESSAGES=C LANG=C TZ=UTC $(CI_ENV) $(SETUP) test --test-name=dom_load $(TEST_FLAGS)
89a96,98
> #
> # # Complete patch: 0008-Don-t-change-book-file-unless-user-s-consent-Closes-.patch
> sed 's/save_annotations_in_ebook"] = true/save_annotations_in_ebook"] = false/' debian/tmp/usr/share/calibre/viewer.js
96a106
> $(SETUP) hyphenation --clean
97a108,112
> # $(SETUP) iso639 --clean
> # $(SETUP) iso3166 --clean
> # $(SETUP) translations --clean
> # $(SETUP) gui --clean
> # $(SETUP) resources --clean

I still don't know how to get around the missing: resources/user-agent-data.json. I can confirm this is required by calibre when one tries to open a book. So, it needs further investigation.

[Guido Falsi (madpilot78)]

Hi again,

Thanks for the feedback. Ive been able to have the FreeBSD port updated and it's building it's parts successfully now. I still have to do some testing.

Unluckily there is an issue I also need to solve to respect FreeBSD ports rules (which are required by the build cluster). The build happens in a FreeBSD jail with a clean filesystem, and the build is expected to leave it clean. Some cache and temporary files are allowed, but nothing else

After the build a new empty directory appears in "/usr/local/libexec/qt6/locales".

While it looks innocuous I need to understand why it gets there at a minimum.

It appears to happen during the "resources" phase.

I suspect it gets created while compiling some translation, but I am having an hard time identifying the exact phase of the build.

Can you help shed some light?

[Guido Falsi (madpilot78)]

I have isolated the issue.

The empty directory gets created when running the line QApplication.instance().processEvents(QEventLoop.ProcessEventsFlag.ExcludeUserInputEvents) in src/calibre/utils/rapydscript.py

This happens when compiling things with the included rapydscript compiler. Maybe using an external compiler can work around the issue, but at present there is no rapydscript compiler included in the ports tree.

For the time being I'm just cleaning after myself, as not to trigger warnings and failures due to failed checks. I can improve things later.

[Kovid Goyal (kovid)]

On Sun, Feb 12, 2023 at 01:42:51PM -0000, Guido Falsi wrote:
> After the build a new empty directory appears in
> "/usr/local/libexec/qt6/locales".

IIRC this is created at runtime by Qt WebEngine, you can simply delete
it after you are done building in your build script, it's not actually
needed by calibre. Qt WebEngine is used in the build process to compile
RapydScript code and of course by the tests.

[Jhonny Oliveira (jhonny-oliveira)]

I finally cracked it, but I changed the approach. I ran the bootstrap, collected the missing files and added them as a binary into the package recipe. Before the build, I unpacked them and off we go. It doesn't look very neat, but for the time being, it is good enough. I'm curious to see what the official release maintainers will do.

This is my complete recipe: https://launchpad.net/~xtradeb/+archive/ubuntu/apps/+sourcefiles/calibre/6.12.0+dfsg-1~xtradeb3/calibre_6.12.0+dfsg-1~xtradeb3.debian.tar.xz

[Guido Falsi (madpilot78)]

Yes, I ended up cleaning up the empty directory.

The rule about not leaving files outside of the workdir is enforced because there is no warranty any directory outside of to be writable. But it should not be a problem in this case.

I have finally updated the FreeBSD port, this is the relevant commit:

https://cgit.freebsd.org/ports/commit/?id=e65d3de319a7f795e28cfc4f9dcef6e1a5ec396b

So, from my point of view, this issue can be closed.

I raised it to understand if the exclusion of the files I asked about was intentional, which it turned out to be the case, so I adapted the port.

Thanks to you all for the support and feedback.