Meta source Douban Books can't work any more

[xuyang]

Starting a few days before, meta source Douban Books can't work any more。

Error message:
---------------------------------------------------------------------------------------
Running identify query with parameters:
{u'authors': [u'\u5f20\u9a8f'], u'identifiers': {u'mobi-asin': u'B07YFPC1P3', u'douban': u'34455756', u'isbn': u'9787121366727'}, u'title': u'\u8fb9\u7f18\u8ba1\u7b97\u65b9\u6cd5\u4e0e\u5de5\u7a0b\u5b9e\u8df5', u'timeout': 30}
Using plugins: Douban Books (2, 1, 1)
The log from individual plugins is below

****************************** Douban Books (2, 1, 1) ******************************
Found 0 results
Downloading from Douban Books took 0.44000005722
Failed to make identify query: u'https://api.douban.com/book/subject/isbn/9787121366727?apikey=0bd1672394eb1ebf2374356abec15c3d'
Traceback (most recent call last):
  File "site-packages\calibre\ebooks\metadata\sources\douban.py", line 334, in identify
  File "site-packages\mechanize\_mechanize.py", line 238, in open_novisit
  File "site-packages\mechanize\_mechanize.py", line 309, in _mech_open
httperror_seek_wrapper: HTTP Error 403: Forbidden

********************************************************************************
The identify phase took 0.60 seconds
The longest time (0.440000) was taken by: Douban Books
Merging results from different sources
We have 0 merged results, merging took: 0.00 seconds
------------------------------------------------------------------------------------------

I guess it may be because apikey was blocked? Is there a way to change apikey for Douban Books API?

[Kovid Goyal (kovid)]

As far as I know the API key is hardcoded in the plugin source code, so someone will need to get a working key and replace it.

[Li Fanxi (lifanxi)]

Douban.com has announced to close the public API for a long time. However, it keeps working until last week.

I'm contacting Douban to see if there's any way to workaround this problem. I'll update once I get response from Douban.

[idealclover (idealclover)]

If Douban cannot provide the public API, it would be possible to get data by web.
Reffered to https://github.com/AlanDecode/Typecho-Plugin-DoubanBoard/blob/master/Action.php .

[xuyang (xuygelba)]

I found a valid key from internet and wrote a test code like this:
--------------------------
.....
headers = {
 # 测试必须用老版本的UA标识，否则会失败
 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)' +
      ' Chrome/70.0.3538.110 Safari/537.36',
 'Referer': 'https://www.douban.com'
}

# 从网上找的别人的apikey
url = douban_base_url_ofc + 'isbn/' + isbn + '?apikey=my_key_xxxx'

log.debug("url:{}".format(url))
item = DoubanItem()
...
--------------------------

the code worked, then i downloaded source code of calibre and replaced key in calibre\ebooks\metadata\douban.py, generated douban.pyo and replaced it into pylib.zip of current version in my computer, but it didn't work. I didn't read calibre source code carefully and completely, maybe should i also change request headers in calibre douban plugin?

Anather solution is i send this key to you, but i don't know if there will be a legal problem to use this key in calibre.

[xuyang (xuygelba)]

The key what i found, is in this web site: https://blog.csdn.net/qq_24810411/article/details/89176404, it's up to you.

[Kovid Goyal (kovid)]

You can run calibre directly from source by following the instructions: https://manual.calibre-ebook.com/develop.html
no need to mess around with .pyo replacement.

[Kovid Goyal (kovid)]

And I think a more sustainable solution is to use web scraping for this, since API keys are trivial to block. Or if you really want to use API keys, maybe make this a non-builtin plugin.

[xuyang (xuygelba)]

I ran calibre directly from source, found that, old key can not be replaced directly with the key which i found.

Because, calibre are using api '...api.douban.com/book/subject/isbn/...', which is an old XML api, it can not be used anymore. What i had done a test, using api '...api.douban.com/v2/book/isbn/...', which is JSON api. there was still much to be done.

[Kevin Liao (pwliao)]

I used douban api v2 to write a new plugin.
https://github.com/pwliao/calibre-douban/releases
After installing the plugin, you need to click "Customize Plugin" and input apikey by yourself.
You can find apikey on the internet.

[xcffl (xcffl)]

I made a patch based on the previous plugin.

[Kovid Goyal (kovid)]

Fixed in branch master. The fix will be in the next release. calibre is usually released every alternate Friday.

 status fixreleased

[xuyang (xuygelba)]

Great jobs! tkx.