[Enhancement] Full-text search for doc/docx/zip (containing htm/html)

[sacharja]

Hi,

would be awesome if indexing could include the formats doc/docx/zip (containing htm/html). Don't know Python, but found the following examples (seems it is somehow technical possible):

DOCX

from spire.doc import *
from spire.doc.common import *

# Create a Document object
doc = Document()

# Load a Word file
doc.LoadFromFile("C:\\Users\\Administrator\\Desktop\\input.docx")

# Get text from the entire document
text = doc.GetText()

# Print result
print(text)

DOC (2003 format)

maybe possible via GPL cross-plattform / no longer developed Antiword:
https://en.wikipedia.org/wiki/Antiword

ZIP (containing htm / html)

from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "http://news.bbc.co.uk/2/hi/health/2284783.stm"
html = urlopen(url).read()
soup = BeautifulSoup(html, features="html.parser")

# kill all script and style elements
for script in soup(["script", "style"]):
    script.extract() # rip it out

# get text
text = soup.get_text()

# break into lines and remove leading and trailing space on each
lines = (line.strip() for line in text.splitlines())
# break multi-headlines into a line each
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
# drop blank lines
text = '\n'.join(chunk for chunk in chunks if chunk)

print(text)

Greetz

[Kovid Goyal (kovid)]

full text indexing works for any format for which calibre has an input
format plugin. That is a format calibre can convert *from*.
DOCX is one of them already. DOC is not supported. And zip with HTML is also supported.

[sacharja (c3521333)]

right, docx is working, but for sure html is not working or indexed.

Example attached:
1. drag & drop into Calibre (html is added via zip)
2. restart, so that it's indexed
3. full text search for "ipsum" --> no results

[Kovid Goyal (kovid)]

Then convert the ZIP file to EPUB and you will be fine.

[Kovid Goyal (kovid)]

Fixed in branch master. The fix will be in the next release. calibre is usually released every alternate Friday.

[sacharja (c3521333)]

Unfortunately this is not fixed in Calibre 8.0.1, htm/html is still not indexed

[Kovid Goyal (kovid)]

Works for me. You will ne to *readd* any books you want indexed or re-run the indexing.

[sacharja (c3521333)]

Strange, did both on Windows:

- installed a fresh Calibre portable
- added via drag & drop: https://bugs.launchpad.net/calibre/+bug/2100891/+attachment/5862625/+files/lorem.htm
- fulltext search with completed indexing for "lorem" or "ipsum"

--> no result

[Kovid Goyal (kovid)]

Fixed in branch master. The fix will be in the next release. calibre is usually released every alternate Friday.