The New Republic recipe no longer works

[darkom]

I tried using the included recipe for The New republic magazine and notice it is obsolete and no longer working. When trying to update the recipe I stumbled upon this error that is full blocker to fix this issue.

I have this method for login

    def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
        self.log("trying to open site")
        br.open('https://newrepublic.com/magazine')
        if self.username is not None and self.password is not None:
           data = urllib.urlencode({'email': self.username, 'password': self.password})
           br.open('https://newrepublic.com/api/subscriptions/cds', data)
        return br

And it fails on first br.open with this message:

InputFormatPlugin: Recipe Input running
Using custom recipe
trying to open site
Traceback (most recent call last):
  File "site.py", line 77, in main
  File "site-packages/calibre/ebooks/conversion/cli.py", line 391, in main
  File "site-packages/calibre/ebooks/conversion/plumber.py", line 1088, in run
  File "site-packages/calibre/customize/conversion.py", line 245, in __call__
  File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 118, in convert
  File "site-packages/calibre/web/feeds/news.py", line 903, in __init__
  File "<string>", line 31, in get_browser
  File "site-packages/mechanize/_mechanize.py", line 254, in open
  File "site-packages/mechanize/_mechanize.py", line 310, in _mech_open
httperror_seek_wrapper: HTTP Error 500: Internal Server Error

Using the Calibre 3.14 64bit Linux on Ubuntu 17.04

If we can get pass this error I will update and fix the recipe.

Thanks

[Kovid Goyal (kovid)]

On my linux system, doing:

calibre-debug -c "from calibre import browser; br = browser(); br.open('https://newrepublic.com/magazine')"

Is line 31 that open or the next one?

[darkom (darko-miletic)]

It is the next one. You are right. It fails on

br.open('https://newrepublic.com/api/subscriptions/cds', data)

[darkom (darko-miletic)]

Interestingly if I try to connect using curl cli everything is ok:

curl -v -c /tmp/cookiejar.txt -G --data-urlencode email=someemail --data-urlencode password=somepass https://newrepublic.com/api/subscriptions/cds

[Kovid Goyal (kovid)]

So if you specify data to urlopen, the request becomes a POST instead of a GET. That is probably what the server does not like. Instead create a Request object explicitly and use that. See the builtin hbr.recipe for an example of doing that.