Unable to download content from a magazine site get CERTIFICATE_VERIFY_FAILED

[darkom]

I just installed latest version of Calibre

When I try to download harper's magazine from cli this is what I get:

darko:harp> ebook-convert harpers_full.recipe .mobi --username=someuser --password=somepass
Conversion options changed from defaults:
  test: None
1% Converting input to HTML...
InputFormatPlugin: Recipe Input running
Using custom recipe
Traceback (most recent call last):
  File "site.py", line 66, in main
  File "site-packages/calibre/ebooks/conversion/cli.py", line 393, in main
  File "site-packages/calibre/ebooks/conversion/plumber.py", line 1088, in run
  File "site-packages/calibre/customize/conversion.py", line 245, in __call__
  File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 118, in convert
  File "site-packages/calibre/web/feeds/news.py", line 904, in __init__
  File "<string>", line 55, in get_browser
  File "site-packages/mechanize/_mechanize.py", line 203, in open
  File "site-packages/mechanize/_mechanize.py", line 230, in _mech_open
  File "site-packages/mechanize/_opener.py", line 204, in open
  File "site-packages/mechanize/_urllib2_fork.py", line 457, in http_response
  File "site-packages/mechanize/_opener.py", line 221, in error
  File "site-packages/mechanize/_urllib2_fork.py", line 332, in _call_chain
  File "site-packages/mechanize/_urllib2_fork.py", line 571, in http_error_302
  File "site-packages/mechanize/_mechanize.py", line 203, in open
  File "site-packages/mechanize/_mechanize.py", line 230, in _mech_open
  File "site-packages/mechanize/_opener.py", line 193, in open
  File "site-packages/mechanize/_urllib2_fork.py", line 344, in _open
  File "site-packages/mechanize/_urllib2_fork.py", line 332, in _call_chain
  File "site-packages/calibre/utils/browser.py", line 28, in https_open
  File "site-packages/mechanize/_urllib2_fork.py", line 1118, in do_open
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

OS: Ubuntu 14.04 64bit
Python: 2.7.6
Calibre: 2.85.1

[Kovid Goyal (kovid)]

That means your OS is missing some needed root certificates. You can use
the root certificates bundled with calibre instead, by setting the
environment variable:

SSL_CERT_FILE=/opt/calibre/resources/mozilla-ca-certs.pem

 status invalid

[Kovid Goyal (kovid)]

Of course it could also be that the site's ceritifcate has expired --
can you visit it in a browser?

[darkom (darko-miletic)]

I tried the same thing on another machine with Windows 7 and it works. I will check the certificate suggestion and get back here with results.

[David West (gdw238)]

I use a Windows 7 system which is current on Windows Updates. I have also had my Calibre downloads from Harpers Magazine (both regular and print articles)fail because of CERTIFICATE_VERIFY_FAILED. These failures began with an attempted download on May 16, 2017. I am able to login to the Harpers website. Has there been any further development on this bug?

[Kovid Goyal (kovid)]

Firefox does not use the system certificates. Either visit the site in Internet Explorer which will cause windows to install the needed certificate, or set the SSL_CERT_FILE environment variable to cause calibre to use its own certificates instead of the system ones.

[David West (gdw238)]

Thank you. Did not know about the Firefox/Internet Explorer/certificate situation. Harpers downloads beautifully. Again, many thanks for the lesson.

[darkom (darko-miletic)]

For people using Ubuntu 14.04 or more recent a missing intermediate certificate from DigiCert needs to be manually installed.

wget -P /tmp/ -nv https://www.digicert.com/CACerts/DigiCertSHA2SecureServerCA.crt
openssl x509 -in /tmp/DigiCertSHA2SecureServerCA.crt -inform DER \
             -out /tmp/DigiCertSHA2SecureServerCA.pem -outform PEM
sudo cp -uv /tmp/DigiCertSHA2SecureServerCA.pem /etc/ssl/certs
sudo c_rehash

That fixes the issue.