Süddeutsche

Bug #1771833 reported by Harald Riedel on 2018-05-17
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
calibre
Undecided
darkom

Bug Description

# Calibre Version:
Calibre 3.23.0 - OS X Version.
(Testet with 3.10 too, same result).

# Hint:

Maybe "Süddeutsche Zeitung" changed something

---8<--- Snip Snip Snip ---8<---

# Call and Log:

Süddeutsche Zeitung
Erstellt von Darko Miletic
News from Germany. Access to paid content.

---8<--- Snip Snip Snip ---8<---

After pressing the Button "Jetzt downloaden" - Waiting -pressing the Button "Speichern" -> nothing happend.
The log says:

---8<--- Snip Snip Snip ---8<---

calibre, version 3.23.0 (darwin, embedded-python: True)
Konvertierungsfehler: Fehlgeschlagen: Nachrichten abrufen von Süddeutsche Zeitung

Nachrichten abrufen von Süddeutsche Zeitung
Conversion options changed from defaults:
  output_profile: u'kindle_oasis'
  verbose: 2
Resolved conversion options
calibre version: 3.23.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x11874d390>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_file_type': 'old',
 'mobi_ignore_margins': False,
 'mobi_keep_original_images': False,
 'mobi_toc_at_start': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': True,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindleOasisOutput object at 0x11874db10>,
 'page_breaks_before': None,
 'personal_doc': '[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'share_not_sync': False,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
Python function terminated unexpectedly: HTTP Error 404: Not Found
InputFormatPlugin: Recipe Input running
Using custom recipe
Traceback (most recent call last):
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 154, in main
    return run_entry_point()
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 114, in run_entry_point
    return getattr(pmod, func)()
  File "site-packages/calibre/utils/ipc/worker.py", line 195, in main
  File "site-packages/calibre/gui2/convert/gui_conversion.py", line 26, in gui_convert
  File "site-packages/calibre/ebooks/conversion/plumber.py", line 1088, in run
  File "site-packages/calibre/customize/conversion.py", line 245, in __call__
  File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 119, in convert
  File "site-packages/calibre/web/feeds/news.py", line 1018, in download
  File "site-packages/calibre/web/feeds/news.py", line 1185, in build_index
  File "<string>", line 97, in parse_index
  File "site-packages/calibre/web/feeds/news.py", line 675, in index_to_soup
  File "site-packages/mechanize/_mechanize.py", line 239, in open_novisit
  File "site-packages/mechanize/_mechanize.py", line 310, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 404: Not Found

Changing the component for this bug.

 assignee darko-miletic
 status triaged

Changed in calibre:
assignee: nobody → darkom (darko-miletic)
status: New → Triaged
darkom (darko-miletic) wrote :

To test this I would need valid access credentials. If you want this fixed contact me directly on my personal email <email address hidden>

Axel (nospamforaxel-1) wrote :
Download full text (6.8 KiB)

Apparently this is not limited to paid content. On my setup here, Ubuntu 19.10 with Calibre 4.2, I cannot download news. Not from Süddeutsche nor from Heise.

Is there a solution for that?

Here is the log of Calibre

---
calibre, version 4.2.0 (linux2, embedded-python: True)
Konvertierungsfehler: Fehlgeschlagen: Nachrichten abrufen von Heise Open

Nachrichten abrufen von Heise Open
Conversion options changed from defaults:
  output_profile: u'generic_eink'
  verbose: 2
Resolved conversion options
calibre version: 4.2.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': u'original',
 'chapter': None,
 'chapter_mark': u'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'dont_split_on_page_breaks': True,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'epub_flatten': False,
 'epub_inline_toc': False,
 'epub_toc_at_end': False,
 'epub_version': u'2',
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x7fdf6c4911d0>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.GenericEink object at 0x7fdf6c4915d0>,
 'page_breaks_before': None,
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': u'',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': u'',
 'sr1_search': u'',
 'sr2_replace': u'',
 'sr2_search': u'',
 'sr3_replace': u'',
 'sr3_search': u'',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:hei...

Read more...

Axel (nospamforaxel-1) wrote :

Hi all,

Is there any clue how to solve this? I recognized that this bug does not exists on my Desktop-Computer runnung Ubuntu 18.04. However it happens on my Laptop runnung 19.10.

Kovid Goyal (kovid) wrote :

Heise has nothing to do with this, the log you posted for it is incomplete. As for the original recipe the darko asked for login credentials for it, but no one responded.

Download full text (5.4 KiB)

What about heise? It is about "Süddeutsche Zeitung«
I hat given my credential at the time I posted the bug.

Am 11.11.2019 um 10:45 schrieb Kovid Goyal <email address hidden>:

Heise has nothing to do with this, the log you posted for it is
incomplete. As for the original recipe the darko asked for login
credentials for it, but no one responded.

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1771833

Title:
 Süddeutsche

Status in calibre:
 Triaged

Bug description:
 # Calibre Version:
 Calibre 3.23.0 - OS X Version.
 (Testet with 3.10 too, same result).

 # Hint:

 Maybe "Süddeutsche Zeitung" changed something

 ---8<--- Snip Snip Snip ---8<---

 # Call and Log:

 Süddeutsche Zeitung
 Erstellt von Darko Miletic
 News from Germany. Access to paid content.

 ---8<--- Snip Snip Snip ---8<---

 After pressing the Button "Jetzt downloaden" - Waiting -pressing the Button "Speichern" -> nothing happend.
 The log says:

 ---8<--- Snip Snip Snip ---8<---

 calibre, version 3.23.0 (darwin, embedded-python: True)
 Konvertierungsfehler: Fehlgeschlagen: Nachrichten abrufen von Süddeutsche Zeitung

 Nachrichten abrufen von Süddeutsche Zeitung
 Conversion options changed from defaults:
   output_profile: u'kindle_oasis'
   verbose: 2
 Resolved conversion options
 calibre version: 3.23.0
 {'asciiize': False,
  'author_sort': None,
  'authors': None,
  'base_font_size': 0,
  'book_producer': None,
  'change_justification': 'original',
  'chapter': None,
  'chapter_mark': 'pagebreak',
  'comments': None,
  'cover': None,
  'debug_pipeline': None,
  'dehyphenate': True,
  'delete_blank_paragraphs': True,
  'disable_font_rescaling': False,
  'dont_compress': False,
  'dont_download_recipe': False,
  'duplicate_links_in_toc': False,
  'embed_all_fonts': False,
  'embed_font_family': None,
  'enable_heuristics': False,
  'expand_css': False,
  'extra_css': None,
  'extract_to': None,
  'filter_css': None,
  'fix_indents': True,
  'font_size_mapping': None,
  'format_scene_breaks': True,
  'html_unwrap_factor': 0.4,
  'input_encoding': None,
  'input_profile': <calibre.customize.profiles.InputProfile object at 0x11874d390>,
  'insert_blank_line': False,
  'insert_blank_line_size': 0.5,
  'insert_metadata': False,
  'isbn': None,
  'italicize_common_cases': True,
  'keep_ligatures': False,
  'language': None,
  'level1_toc': None,
  'level2_toc': None,
  'level3_toc': None,
  'line_height': 0,
  'linearize_tables': False,
  'lrf': False,
  'margin_bottom': 5.0,
  'margin_left': 5.0,
  'margin_right': 5.0,
  'margin_top': 5.0,
  'markup_chapter_headings': True,
  'max_toc_links': 50,
  'minimum_line_height': 120.0,
  'mobi_file_type': 'old',
  'mobi_ignore_margins': False,
  'mobi_keep_original_images': False,
  'mobi_toc_at_start': False,
  'no_chapters_in_toc': False,
  'no_inline_navbars': True,
  'no_inline_toc': False,
  'output_profile': <calibre.customize.profiles.KindleOasisOutput object at 0x11874db10>,
  'page_breaks_before': None,
  'personal_doc': '[PDOC]',
  'prefer_author_sort': False,
  'prefer_metadata_cover': False,
  'pretty_print': False,
  'pubdate': None,
  'publisher...

Read more...

Kovid Goyal (kovid) wrote :

I just tried heise it works fine.

Harald Riedel (lisux) wrote :
Download full text (5.2 KiB)

With what credential do you test?

Am 11.11.2019 um 13:01 schrieb Kovid Goyal <email address hidden>:

I just tried heise it works fine.

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1771833

Title:
 Süddeutsche

Status in calibre:
 Triaged

Bug description:
 # Calibre Version:
 Calibre 3.23.0 - OS X Version.
 (Testet with 3.10 too, same result).

 # Hint:

 Maybe "Süddeutsche Zeitung" changed something

 ---8<--- Snip Snip Snip ---8<---

 # Call and Log:

 Süddeutsche Zeitung
 Erstellt von Darko Miletic
 News from Germany. Access to paid content.

 ---8<--- Snip Snip Snip ---8<---

 After pressing the Button "Jetzt downloaden" - Waiting -pressing the Button "Speichern" -> nothing happend.
 The log says:

 ---8<--- Snip Snip Snip ---8<---

 calibre, version 3.23.0 (darwin, embedded-python: True)
 Konvertierungsfehler: Fehlgeschlagen: Nachrichten abrufen von Süddeutsche Zeitung

 Nachrichten abrufen von Süddeutsche Zeitung
 Conversion options changed from defaults:
   output_profile: u'kindle_oasis'
   verbose: 2
 Resolved conversion options
 calibre version: 3.23.0
 {'asciiize': False,
  'author_sort': None,
  'authors': None,
  'base_font_size': 0,
  'book_producer': None,
  'change_justification': 'original',
  'chapter': None,
  'chapter_mark': 'pagebreak',
  'comments': None,
  'cover': None,
  'debug_pipeline': None,
  'dehyphenate': True,
  'delete_blank_paragraphs': True,
  'disable_font_rescaling': False,
  'dont_compress': False,
  'dont_download_recipe': False,
  'duplicate_links_in_toc': False,
  'embed_all_fonts': False,
  'embed_font_family': None,
  'enable_heuristics': False,
  'expand_css': False,
  'extra_css': None,
  'extract_to': None,
  'filter_css': None,
  'fix_indents': True,
  'font_size_mapping': None,
  'format_scene_breaks': True,
  'html_unwrap_factor': 0.4,
  'input_encoding': None,
  'input_profile': <calibre.customize.profiles.InputProfile object at 0x11874d390>,
  'insert_blank_line': False,
  'insert_blank_line_size': 0.5,
  'insert_metadata': False,
  'isbn': None,
  'italicize_common_cases': True,
  'keep_ligatures': False,
  'language': None,
  'level1_toc': None,
  'level2_toc': None,
  'level3_toc': None,
  'line_height': 0,
  'linearize_tables': False,
  'lrf': False,
  'margin_bottom': 5.0,
  'margin_left': 5.0,
  'margin_right': 5.0,
  'margin_top': 5.0,
  'markup_chapter_headings': True,
  'max_toc_links': 50,
  'minimum_line_height': 120.0,
  'mobi_file_type': 'old',
  'mobi_ignore_margins': False,
  'mobi_keep_original_images': False,
  'mobi_toc_at_start': False,
  'no_chapters_in_toc': False,
  'no_inline_navbars': True,
  'no_inline_toc': False,
  'output_profile': <calibre.customize.profiles.KindleOasisOutput object at 0x11874db10>,
  'page_breaks_before': None,
  'personal_doc': '[PDOC]',
  'prefer_author_sort': False,
  'prefer_metadata_cover': False,
  'pretty_print': False,
  'pubdate': None,
  'publisher': None,
  'rating': None,
  'read_metadata_from_opf': None,
  'remove_fake_margins': True,
  'remove_first_image': False,
  'remove_paragraph_spacing': False,
  'remove_paragraph_spacing_indent_size': 1.5,
  're...

Read more...

heise does not need credentials

Download full text (5.2 KiB)

With what credential do you test "Süddeutsche Zeitung«?

Am 11.11.2019 um 13:57 schrieb Kovid Goyal <email address hidden>:

heise does not need credentials

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1771833

Title:
 Süddeutsche

Status in calibre:
 Triaged

Bug description:
 # Calibre Version:
 Calibre 3.23.0 - OS X Version.
 (Testet with 3.10 too, same result).

 # Hint:

 Maybe "Süddeutsche Zeitung" changed something

 ---8<--- Snip Snip Snip ---8<---

 # Call and Log:

 Süddeutsche Zeitung
 Erstellt von Darko Miletic
 News from Germany. Access to paid content.

 ---8<--- Snip Snip Snip ---8<---

 After pressing the Button "Jetzt downloaden" - Waiting -pressing the Button "Speichern" -> nothing happend.
 The log says:

 ---8<--- Snip Snip Snip ---8<---

 calibre, version 3.23.0 (darwin, embedded-python: True)
 Konvertierungsfehler: Fehlgeschlagen: Nachrichten abrufen von Süddeutsche Zeitung

 Nachrichten abrufen von Süddeutsche Zeitung
 Conversion options changed from defaults:
   output_profile: u'kindle_oasis'
   verbose: 2
 Resolved conversion options
 calibre version: 3.23.0
 {'asciiize': False,
  'author_sort': None,
  'authors': None,
  'base_font_size': 0,
  'book_producer': None,
  'change_justification': 'original',
  'chapter': None,
  'chapter_mark': 'pagebreak',
  'comments': None,
  'cover': None,
  'debug_pipeline': None,
  'dehyphenate': True,
  'delete_blank_paragraphs': True,
  'disable_font_rescaling': False,
  'dont_compress': False,
  'dont_download_recipe': False,
  'duplicate_links_in_toc': False,
  'embed_all_fonts': False,
  'embed_font_family': None,
  'enable_heuristics': False,
  'expand_css': False,
  'extra_css': None,
  'extract_to': None,
  'filter_css': None,
  'fix_indents': True,
  'font_size_mapping': None,
  'format_scene_breaks': True,
  'html_unwrap_factor': 0.4,
  'input_encoding': None,
  'input_profile': <calibre.customize.profiles.InputProfile object at 0x11874d390>,
  'insert_blank_line': False,
  'insert_blank_line_size': 0.5,
  'insert_metadata': False,
  'isbn': None,
  'italicize_common_cases': True,
  'keep_ligatures': False,
  'language': None,
  'level1_toc': None,
  'level2_toc': None,
  'level3_toc': None,
  'line_height': 0,
  'linearize_tables': False,
  'lrf': False,
  'margin_bottom': 5.0,
  'margin_left': 5.0,
  'margin_right': 5.0,
  'margin_top': 5.0,
  'markup_chapter_headings': True,
  'max_toc_links': 50,
  'minimum_line_height': 120.0,
  'mobi_file_type': 'old',
  'mobi_ignore_margins': False,
  'mobi_keep_original_images': False,
  'mobi_toc_at_start': False,
  'no_chapters_in_toc': False,
  'no_inline_navbars': True,
  'no_inline_toc': False,
  'output_profile': <calibre.customize.profiles.KindleOasisOutput object at 0x11874db10>,
  'page_breaks_before': None,
  'personal_doc': '[PDOC]',
  'prefer_author_sort': False,
  'prefer_metadata_cover': False,
  'pretty_print': False,
  'pubdate': None,
  'publisher': None,
  'rating': None,
  'read_metadata_from_opf': None,
  'remove_fake_margins': True,
  'remove_first_image': False,
  'remove_paragraph_spacing': False,
  'remove_paragraph_spacing_ind...

Read more...

Axel (nospamforaxel-1) wrote :

I think, at least in my case, this bug is not limited to the "Süddeutsche Zeitung", therefore I mentioned "heise online" as an example to demonstrate that it also does not depend on a login. I tried also "Zeit online", same thing.

Correct me if I'm wrong, but this would be the most suspicious part in the logs:

---
Traceback (most recent call last):
  File "site-packages/calibre/web/feeds/news.py", line 1331, in download_cover
  File "site-packages/calibre/web/feeds/news.py", line 1304, in _download_cover
  File "site-packages/mechanize/_mechanize.py", line 253, in open
  File "site-packages/mechanize/_mechanize.py", line 309, in _mech_open
httperror_seek_wrapper: HTTP Error 404: Not Found
---

And yes, no problem at all on my desktop computer with Ubuntu 18.04. My desktop (works) and my laptop (does not work) share even the same calibre library that is synchronized over Nextcloud. I don't know whether this affects the bug, but probably not.

that's the cover download failing it wont affect overall download.

Axel (nospamforaxel-1) wrote :

Perhaps I should mention that in my laptop, where this bug is showing up, calibre is installed as flatpak. On my desktop, where everything runs smoothly, it is a "normal" installation with the command from the calibre website.

Is there anything I can contribute to the debugging?

Kovid Goyal (kovid) wrote :

Dont use flatpaks, they are totally unsupported.

Axel (nospamforaxel-1) wrote :

I confirm that in my case when using the installation from the calibre-website the issue is not there. Downloading of news-sources works as expected.

This suggests that the issue is related to flatpak.

I thought flatpak is okay to use, since it is offered in flathub.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers