Memory error converting large book

[Lynn Maddry]

I tried to convert a very large book 230 MB from mobi to epub. I am running Windows 7 64bit with 8 GB of memory.
The book was purchase from Mobilibrary. I can up load it if you want, but is is 230,330,916 bytes.
I understand this might just be to big to convert.
The error details
calibre, version 0.8.8
ERROR: Conversion Error: <b>Failed</b>: Convert book 1 of 1 (My Big Fat Library)

Convert book 1 of 1 (My Big Fat Library)
Resolved conversion options
calibre version: 0.8.8
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0.0,
 'book_producer': None,
 'change_justification': u'original',
 'chapter': u"//*[((name()='h1' or name()='h2') and re:test(., 'chapter|book|section|part\\s+', 'i')) or @class = 'chapter']",
 'chapter_mark': u'pagebreak',
 'comments': None,
 'cover': u'c:\\users\\gmaddry\\appdata\\local\\temp\\calibre_0.8.8_tmp_hryqud\\calibre_0.8.8_w4zgyw.jpeg',
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_split_on_page_breaks': False,
 'enable_heuristics': False,
 'epub_flatten': False,
 'extra_css': None,
 'extract_to': None,
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x05C55CB0>,
 'insert_blank_line': False,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0.0,
 'linearize_tables': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.OutputProfile object at 0x05C55E90>,
 'page_breaks_before': u"//*[name()='h1' or name()='h2']",
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': u'c:\\users\\gmaddry\\appdata\\local\\temp\\calibre_0.8.8_tmp_hryqud\\calibre_0.8.8_pv3u_m.opf',
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': u'',
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': None,
 'sr1_search': None,
 'sr2_replace': None,
 'sr2_search': None,
 'sr3_replace': None,
 'sr3_search': None,
 'tags': None,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: MOBI Input running
on c:\users\gmaddry\appdata\local\temp\calibre_0.8.8_tmp_hryqud\calibre_0.8.8_n6zzib.mobi
Extracting text...
Extracting text...
Python function terminated unexpectedly
   (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 132, in main
  File "site.py", line 109, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 181, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 31, in gui_convert_override
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 922, in run
  File "site-packages\calibre\customize\conversion.py", line 204, in __call__
  File "site-packages\calibre\ebooks\mobi\input.py", line 27, in convert
  File "site-packages\calibre\ebooks\mobi\reader.py", line 312, in extract_content
  File "site-packages\calibre\ebooks\mobi\reader.py", line 866, in extract_text
MemoryError

[Kovid Goyal (kovid)]

calibre is not designed to convert such books. It needs to keep things in
memory to be able to perform all the magic that happens during conversion.

 status wontfix

[Lynn Maddry (lgmaddry)]

OK. I expected that. Thanks for checking and letting me know.

> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf Of
> Kovid Goyal
> Sent: Saturday, July 02, 2011 14:23
> To: <email address hidden>
> Subject: [Bug 804829] Re: calibre bug 804829
>
> calibre is not designed to convert such books. It needs to keep things
> in
> memory to be able to perform all the magic that happens during
> conversion.
>
> status wontfix
>
> ** Changed in: calibre
> Status: New => Won't Fix
>
> --
> You received this bug notification because you are subscribed to the
> bug
> report.
> https://bugs.launchpad.net/bugs/804829
>
> Title:
> Memory error converting large book
>
> Status in calibre: e-book management:
> Won't Fix
>
> Bug description:
> I tried to convert a very large book 230 MB from mobi to epub. I am
> running Windows 7 64bit with 8 GB of memory.
> The book was purchase from Mobilibrary. I can up load it if you want,
> but is is 230,330,916 bytes.
> I understand this might just be to big to convert.
> The error details
> calibre, version 0.8.8
> ERROR: Conversion Error: <b>Failed</b>: Convert book 1 of 1 (My Big
> Fat Library)
>
> Convert book 1 of 1 (My Big Fat Library)
> Resolved conversion options
> calibre version: 0.8.8
> {'asciiize': False,
> 'author_sort': None,
> 'authors': None,
> 'base_font_size': 0.0,
> 'book_producer': None,
> 'change_justification': u'original',
> 'chapter': u"//*[((name()='h1' or name()='h2') and re:test(.,
> 'chapter|book|section|part\\s+', 'i')) or @class = 'chapter']",
> 'chapter_mark': u'pagebreak',
> 'comments': None,
> 'cover':
> u'c:\\users\\gmaddry\\appdata\\local\\temp\\calibre_0.8.8_tmp_hryqud\\c
> alibre_0.8.8_w4zgyw.jpeg',
> 'debug_pipeline': None,
> 'dehyphenate': True,
> 'delete_blank_paragraphs': True,
> 'disable_font_rescaling': False,
> 'dont_split_on_page_breaks': False,
> 'enable_heuristics': False,
> 'epub_flatten': False,
> 'extra_css': None,
> 'extract_to': None,
> 'fix_indents': True,
> 'flow_size': 260,
> 'font_size_mapping': None,
> 'format_scene_breaks': True,
> 'html_unwrap_factor': 0.4,
> 'input_encoding': None,
> 'input_profile': <calibre.customize.profiles.InputProfile object at
> 0x05C55CB0>,
> 'insert_blank_line': False,
> 'insert_metadata': False,
> 'isbn': None,
> 'italicize_common_cases': True,
> 'keep_ligatures': False,
> 'language': None,
> 'level1_toc': None,
> 'level2_toc': None,
> 'level3_toc': None,
> 'line_height': 0.0,
> 'linearize_tables': False,
> 'margin_bottom': 5.0,
> 'margin_left': 5.0,
> 'margin_right': 5.0,
> 'margin_top': 5.0,
> 'markup_chapter_headings': True,
> 'max_toc_links': 50,
> 'minimum_line_height': 120.0,
> 'no_chapters_in_toc': False,
> 'no_default_epub_cover': False,
> 'no_inline_navbars': False,
> 'no_svg_cover': False,
> 'output_profile': <calibre.customize.profiles.OutputProfile object
> at 0x05C55E90>,
> 'page_breaks_before': u"//*[name()='h1' or name()='h2']",
> 'prefer_metadata_cover': False,
> 'preserve_cover_aspec...

[Kimura (ike2079)]

 I am having the same problem with a 30meg mobi file. Trying to convert to Epub. I have 4gigs of ram in a Windows 7 64bit machine. Any solutions?

[Robert Simmons (bsimmons)]

I am having the same problem! I have a Dell with 2GB RAM and Calibre fails in converting a 5.5MB MOBI. Kovid, please reconsider fixing this!

Here is the log:

calibre, version 0.9.21 (win32, isfrozen: True)
Conversion Error: Failed: Convert book 1 of 1 (HarperCollins German-English College Dictionary)

Convert book 1 of 1 (HarperCollins German-English College Dictionary)
Resolved conversion options
calibre version: 0.9.21
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0.0,
 'book_producer': None,
 'change_justification': u'original',
 'chapter': u"//*[((name()='h1' or name()='h2') and re:test(., '\\s*((chapter|book|section|part)\\s+)|((prolog|prologue|epilogue)(\\s+|$))', 'i')) or @class = 'chapter']",
 'chapter_mark': u'pagebreak',
 'comments': None,
 'cover': u'C:\\DOCUME~1\\Bob\\LOCALS~1\\Temp\\calibre_0.9.21_tmp_x_aixd\\nawjcr.jpeg',
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'duplicate_links_in_toc': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': u'',
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x034473F0>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0.0,
 'linearize_tables': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_toc_at_start': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': False,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindleOutput object at 0x03447730>,
 'page_breaks_before': u"//*[name()='h1' or name()='h2']",
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': u'C:\\DOCUME~1\\Bob\\LOCALS~1\\Temp\\calibre_0.9.21_tmp_x_aixd\\ukaqgi.opf',
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': u'',
 'search_replace': '[]',
 'series': None,
 'series_index': None,
 'share_not_sync': False,
 'smarten_punctuation': False,
 'sr1_replace': None,
 'sr1_search': None,
 'sr2_replace': None,
 'sr2_search': None,
 'sr3_replace': None,
 'sr3_search': None,
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: MOBI Input running
on C:\DOCUME~1\Bob\LOCALS~1\Temp\calibre_0.9.21_tmp_x_aixd\elmgo...

[Martin Manscher (manscher)]

Same problem with two 15MB ebooks I purchased recently. 8GB RAM. Attaching ebook.

calibre, version 2.73.0 (win32, embedded-python: True)
Conversion Error: Failed: Convert book 1 of 1 (Value Proposition Design)

Convert book 1 of 1 (Value Proposition Design)
Resolved conversion options
calibre version: 2.73.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0.0,
 'book_producer': None,
 'change_justification': u'original',
 'chapter': u"//*[((name()='h1' or name()='h2') and re:test(., '\\s*((chapter|book|section|part)\\s+)|((prolog|prologue|epilogue)(\\s+|$))', 'i')) or @class = 'chapter']",
 'chapter_mark': u'pagebreak',
 'comments': None,
 'cover': u'C:\\Users\\mah\\AppData\\Local\\Temp\\calibre_zdaq1t\\wkvevw.jpeg',
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': u'',
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x038BB290>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0.0,
 'linearize_tables': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_file_type': u'old',
 'mobi_ignore_margins': False,
 'mobi_keep_original_images': False,
 'mobi_toc_at_start': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': True,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindlePaperWhiteOutput object at 0x038BB650>,
 'page_breaks_before': u"//*[name()='h1' or name()='h2']",
 'personal_doc': u'[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': u'C:\\Users\\mah\\AppData\\Local\\Temp\\calibre_zdaq1t\\kavxvh.opf',
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': u'',
 'search_replace': '[]',
 'series': None,
 'series_index': None,
 'share_not_sync': False,
 'smarten_punctuation': False,
 'sr1_replace': None,
 'sr1_search': None,
 'sr2_replace': None,
 'sr2_search': None,
 'sr3_replace': None,
 'sr3_search': None,
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': '[]',
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: EPUB ...

[Eli Schwartz (eschwartz)]

calibre cannot fix the problem of your computer not having enough RAM to complete the conversion.

Make sure you are using 64-bit calibre on a 64-bit OS. Otherwise calibre cannot even access all the RAM your computer has (32-bit processes are capped at 2 GB).

Try adding more RAM to your computer (or just use a computer with more RAM).

If neither of those help, then you are simply out of luck, the book is too big/complex.
It is that simple.

[Stuart Flicker (flicker)]

Eli,

I hope that the week has started well for you.

This problem was fixed ages ago when the 64 bit version of Calibre was
released.

Thank-you for following up with me.

Stuart

On 11/28/2016 11:57 AM, Eli Schwartz wrote:
> calibre cannot fix the problem of your computer not having enough RAM to
> complete the conversion.
>
> Make sure you are using 64-bit calibre on a 64-bit OS. Otherwise calibre
> cannot even access all the RAM your computer has (32-bit processes are
> capped at 2 GB).
>
> Try adding more RAM to your computer (or just use a computer with more
> RAM).
>
> If neither of those help, then you are simply out of luck, the book is too big/complex.
> It is that simple.
>

[Martin Manscher (manscher)]

Hi Eli,

I've installed the 64-bit version (thanks for pointing that out), and the
conversion has now been running for 8 hours. Current memory consumption 3,5
GB for a 15 MB epub file. I'll revert here once it finishes.

Martin

On 28 November 2016 at 23:57, Stuart Flicker <email address hidden> wrote:

> Eli,
>
> I hope that the week has started well for you.
>
> This problem was fixed ages ago when the 64 bit version of Calibre was
> released.
>
> Thank-you for following up with me.
>
> Stuart
>
> On 11/28/2016 11:57 AM, Eli Schwartz wrote:
> > calibre cannot fix the problem of your computer not having enough RAM to
> > complete the conversion.
> >
> > Make sure you are using 64-bit calibre on a 64-bit OS. Otherwise calibre
> > cannot even access all the RAM your computer has (32-bit processes are
> > capped at 2 GB).
> >
> > Try adding more RAM to your computer (or just use a computer with more
> > RAM).
> >
> > If neither of those help, then you are simply out of luck, the book is
> too big/complex.
> > It is that simple.
> >
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/804829
>
> Title:
> Memory error converting large book
>
> Status in calibre:
> Won't Fix
>
> Bug description:
> I tried to convert a very large book 230 MB from mobi to epub. I am
> running Windows 7 64bit with 8 GB of memory.
> The book was purchase from Mobilibrary. I can up load it if you want,
> but is is 230,330,916 bytes.
> I understand this might just be to big to convert.
> The error details
> calibre, version 0.8.8
> ERROR: Conversion Error: <b>Failed</b>: Convert book 1 of 1 (My Big Fat
> Library)
>
> Convert book 1 of 1 (My Big Fat Library)
> Resolved conversion options
> calibre version: 0.8.8
> {'asciiize': False,
> 'author_sort': None,
> 'authors': None,
> 'base_font_size': 0.0,
> 'book_producer': None,
> 'change_justification': u'original',
> 'chapter': u"//*[((name()='h1' or name()='h2') and re:test(.,
> 'chapter|book|section|part\\s+', 'i')) or @class = 'chapter']",
> 'chapter_mark': u'pagebreak',
> 'comments': None,
> 'cover': u'c:\\users\\gmaddry\\appdata\\local\\temp\\calibre_0.8.8_
> tmp_hryqud\\calibre_0.8.8_w4zgyw.jpeg',
> 'debug_pipeline': None,
> 'dehyphenate': True,
> 'delete_blank_paragraphs': True,
> 'disable_font_rescaling': False,
> 'dont_split_on_page_breaks': False,
> 'enable_heuristics': False,
> 'epub_flatten': False,
> 'extra_css': None,
> 'extract_to': None,
> 'fix_indents': True,
> 'flow_size': 260,
> 'font_size_mapping': None,
> 'format_scene_breaks': True,
> 'html_unwrap_factor': 0.4,
> 'input_encoding': None,
> 'input_profile': <calibre.customize.profiles.InputProfile object at
> 0x05C55CB0>,
> 'insert_blank_line': False,
> 'insert_metadata': False,
> 'isbn': None,
> 'italicize_common_cases': True,
> 'keep_ligatures': False,
> 'language': None,
> 'level1_toc': None,
> 'level2_toc': None,
> 'level3_toc': None,
> 'line_height': 0.0,
> 'linearize_tables': False,
> 'margin_bottom': 5.0,
> 'margin_left': 5.0,
> 'margi...

[Martin Manscher (manscher)]

After 3500 minutes of runtime, and a memory usage of up to 5 GB (for a 15 MB input file) I decided to stop the conversion.

[fgnievinski (fgnievinski)]

My computer has 16GB of RAM but Calibre has is using only 2.8GB. The job has been stuck at 47% for 5 hours while "Removing fake margins..." This bug is definitely NOT caused by lack of RAM.

[fgnievinski (fgnievinski)]

while using version 4.0, 64-bit for Windows.