search: search hangs for books with large OCR files

[mangtronix]

Search hangs for this book (e.g. search for "street"). Possibly happens in other large books.

http://www.archive.org/stream/bostonmassachuse1916112samp#page/n9/mode/2up

[mangtronix (mang)]

The existing search implementation exhausts memory. Probably hold off on this until the search rewrite.

2010/02/23 01:37:29 [error] 2207#0: *1103601 FastCGI sent in stderr: "PHP Fatal error: Allowed memory size of 209715200 bytes exhausted (tried to allocate 82161 bytes) in /var/www/BookReader/flipbook_search_br.php on line 151" while reading response header from upstream, client: 71.202.38.39, server: ia311210.us.archive.org, request: "GET /BookReader/flipbook_search_br.php?url=/2/items/bostonmassachuse1916112samp/bostonmassachuse1916112samp_djvu.xml&term=ferries&format=XML&callback=br.BRSearchCallback HTTP/1.1", upstream: "fastcgi://unix:/tmp/.fastcgi.www-data/socket:", host: "ia311210.us.archive.org", referrer: "http://www.archive.org/stream/bostonmassachuse1916112samp"

[mangtronix (mang)]

Also reported for this book: http://www.archive.org/stream/publicrecord01reel01#page/n253/mode/2up/search/house

The OCR data for this book is >300MB.

[mangtronix (mang)]

I see this line in the code. A temporary fix may be to increase the memory limit.

ini_set("memory_limit","200M"); // XML can be big, esp. brittanica (100MB)

[mangtronix (mang)]

Yikes even setting the limit to 600M is not enough.

2010/03/24 21:31:30 [error] 4569#0: *379504 FastCGI sent in stderr: "PHP Fatal error: Allowed memory size of 629145600 bytes exhausted (tried to allocate 532840 bytes) in /home/mang/bookreader/BookReaderIA/datanode/flipbook_search_br.php on line 151" while reading response header from upstream, client: 71.202.38.39, server: ia361309.us.archive.org, request: "GET /~mang/BookReader/flipbook_search_br.php?url=/23/items/publicrecord01reel01/publicrecord01reel01_djvu.xml&term=house&format=XML&callback=br.BRSearchCallback HTTP/1.1", upstream: "fastcgi://unix:/tmp/.fastcgi.www-data/socket:", host: "ia361309.us.archive.org:80", referrer: "http://www-mang.archive.org/stream/publicrecord01reel01"