Calibre OPDS server cannot show more than 30 authors at a time

[mcbastos]

I have a large library. As I was downloading some books to my phone (I use Aldiko to connect to the OPDS server), I couldn't find some authors when using the "by authors" listing.

Further investigation revealed that when there are more than 30 authors under the same letter (for instance, I have 36 authors with the initial "B"), only the first 30 are listed.

At first I was unsure if this was a Calibre or Aldiko bug, so I double-checked by opening my web browser on 127.0.0.1:8080/opds on the PC.

Sure enough, the bug also showed under browser access. Furthermore, I noticed that the "calibre Library :: By Authors" page had wrong information:

In the regular Calibre UI, the library reports 36 authors under the "B" letter... but the webpage reports only 31, and when the link is opened it only actually displays 30, as mentioned above.

Similarly, other letters with large numbers of authors had problems:
- Letter C: 36 authors, 35 reported, 30 actually listed
- Letter M: 36 authors, 34 reported, 30 actually listed
- Letter S: 46 authors, 45 reported, 30 actually listed

[Kovid Goyal (kovid)]

The OPDS feed is paginated. Look for the pagination links <link
rel="next">, for example for an author category with thirty five items,
in my library:

  <link href="/opds/categorygroup/617574686f7273/53?library_id=library&amp;offset=30" type="application/atom+xml;type=feed;profile=opds-catalog" title="Next" rel="next"/>

Presumably Aldiko does not support pagination. You can work around it in
the calibre server by increasing the max_opds_items setting in
Preferences->Sharing over the net->Advanced

But really, the correct fix is for Aldiko to support pagination, which
is part of the OPDS spec.

 status invalid

[mcbastos (mcbastos)]

Well, that answers *half* of the question. I'll talk to the Aldiko people about it.

HOWEVER...

There is STILL the matter of the main "by authors" list informing the *incorrect* number of author entries under each letter subpage, as mentioned in the original report.

[Kovid Goyal (kovid)]

I cannot replicate that. The number of authors in each sub-group matches
the number shown in the calibre GUI (although, I have to say that there
is no a-priori reason they should match, as the calibre GUI has various
settings that can affect how grouping is done, that dont apply to the
OPDS feed). What must match is the number shown in the opds feed and the
number of actual authors accessible via the feed, including pagination.
And those do match for you as well as far as I can tell from what you
posted.

[mcbastos (mcbastos)]

Uh, no, they don't match. And the issue appears to be unrelated to the paging thing, because I raised the max_opds_items value so they can all fit in a single page. I'm checking via Firefox pointed to http://127.0.0.1:8080/opds/ , not Aldiko. (Firefox is the one browser that displays the OPDS server in a convenient way -- IE has no clickable links, Chrome shows raw XML and Edge refuses to load the page. But hand-copying the URL from Firefox yields the same results, so it doesn't seem to be a browser issue)

Anyway... I have 36 authors with surnames beginning in "B". The Calibre GUI shows me 36. If I load the "B" author listing in OPDS, I get 36 names (hand-counted), as expected. But the "By Authors" OPDS page says that there are only 31 items under "B", which is plainly wrong.

The actual XML block for the "B" group is as follows:
<entry>
    <title>B</title>
    <id>calibre:category-group:authors:B</id>
    <updated>2017-07-10T02:44:24+00:00</updated>
    <content type="text">31 items</content>
    <link href="/opds/categorygroup/617574686f7273/42?library_id=E-books_library" type="application/atom+xml;type=feed;profile=opds-catalog"/>
  </entry>

As I said, this does not seem to be directly related to the OPDS paging, since it was unaffected by raising the max_opds_items value. But, coincidentally, it appears to affect only author groups with more than 30 names. ALL such groups in my library are affected. It's not a big annoyance, and I wouldn't have even noticed it if I wasn't trying to figure out the page limit thing, but it's there.

[Kovid Goyal (kovid)]

As I said, it works for me. For example, for S, the entry in my OPDS feed is

<entry>
    <title>S</title>
    <id>calibre:category-group:authors:S</id>
    <updated>2017-07-07T03:35:45+00:00</updated>
    <content type="text">36 items</content>
    <link href="/opds/categorygroup/617574686f7273/53?library_id=library" type="application/atom+xml;type=feed;profile=opds-catalog"/>
  </entry>

See the 36 which is the correct number.

Are you running an up-to-date version of calibre, i.e. version 3.3?

If you are, attach the metadata.db file from your calibre library folder and I will see if I can replicate with that.

[mcbastos (mcbastos)]

Yes, latest version, 3.3 64bit for Windows.

I'm attaching the database (zipped, since it's a rather large file that compresses well).

If this is a database issue, I hope there's some way to repair it. I have already ran the maintenance on it and got a clean bill of health.

[Kovid Goyal (kovid)]

Fixed in branch master. The fix will be in the next release. calibre is usually released every alternate Friday.

 status fixreleased

[mcbastos (mcbastos)]

Glad to find out I wasn't bothering you for a nonexistent bug after all. Thank you for all your work.