SSL error for Wallhaven (not solved by dependency installation)

[Peter Levi]

When trying to add a new Wallhaven source for a simple keyword or full URL:

ERROR: 2016-05-02 11:23:13,499: validate() 'Error while validating wallhaven search'
Traceback (most recent call last):
  File "/home/peter/dev/variety/variety/WallhavenDownloader.py", line 70, in validate
    s, count = WallhavenDownloader(None, location).search()
  File "/home/peter/dev/variety/variety/WallhavenDownloader.py", line 56, in search
    soup = Util.html_soup(url)
  File "/home/peter/dev/variety/variety/Util.py", line 440, in html_soup
    return bs4.BeautifulSoup(Util.urlopen(url, data).read())
  File "/home/peter/dev/variety/variety/Util.py", line 428, in urlopen
    return urllib2.urlopen(request, data=urllib.urlencode(data) if data else None, timeout=20)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 442, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 629, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 1] _ssl.c:510: error:14094438:SSL routines:SSL3_READ_BYTES:tlsv1 alert internal error>

When downloading from Wallhaven:

INFO: 2016-05-02 11:26:38,425: fill_queue() 'Filling wallhaven queue: tilt shift'
INFO: 2016-05-02 11:26:38,425: search() 'Performing wallhaven search: url=http://alpha.wallhaven.cc/search?q=tilt%20shift&categories=111&purity=100&sorting=favorites&order=desc'
ERROR: 2016-05-02 11:26:38,730: download_thread() 'Could not download wallpaper:'
Traceback (most recent call last):
  File "/home/peter/dev/variety/variety/VarietyWindow.py", line 974, in download_thread
    self.download_one_from(downloader)
  File "/home/peter/dev/variety/variety/VarietyWindow.py", line 1006, in download_one_from
    file = downloader.download_one()
  File "/home/peter/dev/variety/variety/WallhavenDownloader.py", line 96, in download_one
    self.fill_queue()
  File "/home/peter/dev/variety/variety/WallhavenDownloader.py", line 132, in fill_queue
    s, count = self.search()
  File "/home/peter/dev/variety/variety/WallhavenDownloader.py", line 56, in search
    soup = Util.html_soup(url)
  File "/home/peter/dev/variety/variety/Util.py", line 440, in html_soup
    return bs4.BeautifulSoup(Util.urlopen(url, data).read())
  File "/home/peter/dev/variety/variety/Util.py", line 428, in urlopen
    return urllib2.urlopen(request, data=urllib.urlencode(data) if data else None, timeout=20)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 442, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 629, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 1] _ssl.c:510: error:14094438:SSL routines:SSL3_READ_BYTES:tlsv1 alert internal error>

[Peter Levi (peterlevi)]

This is most probably due to a bug in openssl, triggered by some recent update to Wallhaven's SSL setup: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1475228

[Gregory Knott (knott-gregory)]

This is also an issue for me running on Mint 17.

[Gregory Knott (knott-gregory)]

Peter,

I'm unable to decipher the lingo to understand if this is a server side or client side issue.

If server side, do I approach wallhaven support?

If client side, is there a work around like reverting to a previous version of java?

Please comment.

Greg.

[Peter Levi (peterlevi)]

It's a combination. openssl seems to not work well with Wallhaven's current SSL configuration. Most probably no sort of reverting will fix this, Java is not relevant here. I haven't yet investigated the problem deeper to give more feedback.

[kilotaras (kilotaras)]

This can be solved by moving to requests library.

In [5]: url = 'https://alpha.wallhaven.cc/search?q=tilt%20shift&categories=111&purity=100&sorting=favorites&order=desc'

In [6]: urllib2.urlopen(url)
URLError: <urlopen error [Errno 1] _ssl.c:510: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error>

In [7]: requests.get(url)
Out[7]: <Response [200]>

[Peter Levi (peterlevi)]

Oops, requests fixed the problem on my machine too, but on a clean Ubuntu 14.04 installation, it does not:

>>> import requests
>>> requests.get('https://alpha.wallhaven.cc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:510: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error

It seems additional SNI-related python packages are needed, but they are not installable via apt-get, so Variety cannot include them as dependencies in debian/control:
https://stackoverflow.com/questions/18578439/using-requests-with-tls-doesnt-give-sni-support/18579484#18579484

Hm, can we depend on pip and have these installed via pip...

[Peter Levi (peterlevi)]

Fixed: Variety will suggest to install the required dependencies if it hits an SSL error.

[Gregory Knott (knott-gregory)]

How do I get an updated executable with this fix?

[Peter Levi (peterlevi)]

I will be making an official release with this fix soon, or you can use the deb from the "daily" ppa https://launchpad.net/~variety/+archive/ubuntu/daily

[morriscode (morriscode)]

This is still happening to me in 0.6.3 on Ubuntu 14.04 LTS... How do I fix this?

SSLError: [Errno 1] _ssl.c:510: error:14094438:SSL routines:SSL3_READ_BYTES:tlsv1 alert internal error
ERROR: 2017-02-11 18:21:47,781: download_thread() 'Could not download wallpaper:'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/variety/VarietyWindow.py", line 1038, in download_thread
    self.download_one_from(downloader)
  File "/usr/lib/python2.7/dist-packages/variety/VarietyWindow.py", line 1070, in download_one_from
    file = downloader.download_one()
  File "/usr/lib/python2.7/dist-packages/variety/WallhavenDownloader.py", line 97, in download_one
    self.fill_queue()
  File "/usr/lib/python2.7/dist-packages/variety/WallhavenDownloader.py", line 139, in fill_queue
    s, count = self.search()
  File "/usr/lib/python2.7/dist-packages/variety/WallhavenDownloader.py", line 57, in search
    soup = Util.html_soup(url)
  File "/usr/lib/python2.7/dist-packages/variety/Util.py", line 542, in html_soup
    return bs4.BeautifulSoup(Util.fetch(url, data))
  File "/usr/lib/python2.7/dist-packages/variety/Util.py", line 530, in fetch
    return Util.request(url, data).text
  File "/usr/lib/python2.7/dist-packages/variety/Util.py", line 510, in request
    verify=False)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)

[morriscode (morriscode)]

Installing sudo pip install ndg-httpsclient fixed it. Sorry

[Matias Rossi (sitiomovil)]

In debian:
run sudo pip3 install ndg-httpsclient

[Gregory (gmarthews-1)]

I got this bug after upgrading to 17.04, and carefully went through all the dependencies to make sure I had them. Still got the bug after several restarts, so decided to do a simple reinstall, still the bug. So the nuclear option, purge and reinstall, and yet again this bug persists. It is not too difficult to simply ignore it, but you'd think a reinstall would fix it.

[Robert (antinsa)]

I have this bub also on 17.04 and have tried pip install ndg-httpsclient and its not fixed.Really would like to fix this old bug. I have the same bug on multiple 16.10 machinese. Using the daily build ppa

[James Lu (jlu5)]

Can you reproduce this with the official Ubuntu package? I can't reproduce the bug on my 17.04 install (0.6.3-2), and I haven't ever installed the extra dependencies. pip isn't even installed to make that happen, for that matter.

[Robert (antinsa)]

yes, this is happening on about 20 ubuntu machinese I administrate...

[vanessa (vanessa0x2a)]

Same error still occurring on openSUSE tumbleweed

[Kiwi Rider (kiwirider)]

Hi there,

Just in relation to this (and let me know if you wish me to post it
directly) - I tried the test mentioned on the related issue you reported
back, and also got the "yes" result :

  python3 -c 'import requests; r =
requests.get("https://www.ssllabs.com/ssltest/viewMyClient.html");
print(r.text)' | grep SNI -A1
                         <td class="tableLabel" width="250">Server Name
Indication (SNI)</td>
/            <td class="tableCell">//*Yes*//</td>

/Thanks for your time and your work on his project.

On 17/04/18 11:11, James Lu wrote:
> ** Changed in: variety
> Milestone: None => 0.6.9
>
> ** Changed in: variety
> Status: Confirmed => In Progress
>
> ** Changed in: variety
> Assignee: (unassigned) => James Lu (tacocat)
>