[Kindle, periodical] View sections and articles doesn't work the way intended

[noobalhoi]

If I press the "View Sections & Articles" button in the New York Times it will jump to the first article at the beginning and not to the TOC and the used article.

 It's working with the bought NYT so it's either a problem of the grabbing script or it's a Calibre problem/bug (means it doesn't save your actual page so it can refer to the TOC)

[John Schember (user-none)]

It is an issue with changes to the periodical format Amazon introduced in the 3.1 firmware update to the Kindle. See http://www.mobileread.com/forums/showthread.php?t=123231 for more information.

[John Schember (user-none)]

I've add to the title to make it clear what this issue is in regard to.

[Ian E. Gorman (iegorman)]

<fact>
I have one periodical subscription from Amazon (Globe and Mail) and several periodicals via calibre.

I have noticed two differences between Amazon and calibre periodical.

1 --

When reading the Amazon periodical on Kindle 3, if I select an article from Sections & Articles, read one or more articles in sequence, and then hit the "Back" button, I return to the list at the point (section and article) where I left from the list.

If I upload a periodical from calibre to Kindle, and do the same things, the "Back" button takes me to the first section instead of returning me to the point from which I left "Sections & Articles".

2 --

With the Amazon periodical, if I select a section, move the cursor to the "articles" side and then down to select "View Articles List", I will get to the list of articles, and be at the first article of the selected section, with three articles showing on the page, and will be able to select any article.

If I do the same thing with an uploaded calibre periodical, I will get the title of the article at the beginning of the selected section, but will not be able to read the article because I have "Article 1 of 0".
</fact>

<opinion>
This tells me that there is some trick of formatting in the Amazon periodical that is not in the calibre periodical. I don't know what the difference is because the Amazon periodical is DRMed and I have not tried to "crack" the DRM.
</opinion>

[John Schember (user-none)]

Calibre created periodical.

[John Schember (user-none)]

Kindlegen created periodical. Same content as the calibre created one.

[John Schember (user-none)]

Source content both MOBI's were created using.

Note:

The OPF contains the section

<x-metadata>
    <output encoding="utf-8" content-type="application/x-mobipocket-subscription-magazine"></output>
    </output>
</x-metadata>

Which is used by kindlegen to use the periodical structure. This will need to be removed to as it is kindlegen specific and will cause calibre's conversion to fail.

Also, calibre's mobi output.py needs to be modified. The check_for_periodical function needs to have self.opts.mobi_periodical = True in order to use the periodical structure. Otherwise the conversion will always output as a book.

[Eddie (tylau0)]

We may consider adapting the code at http://www.mobileread.com/forums/showthread.php?p=1666880#post1666880 back to Calibre. That solution uses an external kindlegen to fix the problem.

[John Schember (user-none)]

Quoting Kovid from that thread, "No need, I started working on MOBI indexing a few days ago. Hopefully I will be able to figure out the problem. I've written code that decompiles the MOBI, including all indexing information which should allow me to see what the differences between kindlegen generated periodicals and calibre ones are. You can run it with

calibre-debug --inspect-mobi filename.mobi

You will need to be running from latest calibre source for this to work."

[Kovid Goyal (kovid)]

Well, I've spent two weeks on this, and I can't get it to work. So I'm giving up as I have many other things I need to work on in calibre. If anybody wants to pick up where I've left of, go to preferences->tweaks and add a plugin tweak

new_mobi_writer = True

This will cause calibre to use my new code. The new code generates all the indexing data, including the trailing byte sequences in the same way as kindlegen 1.2. It also wraps sections and articles in <divs> and adds an empty div tag at the end of every article. The resulting MOBI has a working "View articles list", but reading articles and going back to the sections list does not preserve your place.

I suspect that for that to work, we need some special markup in the HTML itself, but I'm out of patience with this.

You can decompile any MOBI produced with my new code or with kindlegen 1.2 by running

calibre-debug --inspect-mobi filename.mobi

This generates a nice textual representation of all the data in the MOBI file including indexing records and trailing byte sequences.

The new code is in calibre.ebooks.mobi.writer2 it is nicely modularized and much easier to understand/modify than the original.

If you plan to work on this and have questions, feel free to ask me by posting to this ticket.

[Kovid Goyal (kovid)]

Fix for this is in 0.8.16, go to Preferences->Tweaks->Plugin tweaks and add

new_mobi_writer = True

to test the fix.