can't use testbrowser to test redirects to remote sites

[James Henstridge]

I would like to be able to use zope.testbrowser to test a redirect to a remote site.

Currently this fails because testbrowser tries to get the zope publication code to render the remote site, which fails. It would be nice if I could tell testbrowser to not follow the redirect in this case, and be able to check the headers to verify that the redirect went to the appropriate web site.

Below is the traceback I get:

Failed example:
   beta_browser.open('http://launchpad.dev/ubuntu')
Exception raised:
   Traceback (most recent call last):
     File "lib/zope/testing/doctest.py", line 1256, in __run
       compileflags, 1) in test.globs
     File "<doctest xx-beta-testers-redirection.txt[4]>", line 1, in ?
     File "lib/zope/testbrowser/browser.py", line 220, in open
       self.mech_browser.open(url, data)
     File "/usr/lib/python2.4/site-packages/mechanize/_mechanize.py", line 254, in open
       return self._mech_open(url, data)
     File "/usr/lib/python2.4/site-packages/mechanize/_mechanize.py", line 277, in _mech_open
       self._response = UserAgent.open(self, self.request, data)
     File "urllib2.py", line 364, in open
       response = meth(req, response)
     File "/usr/lib/python2.4/site-packages/ClientCookie/_urllib2_support.py", line 625, in http_response
       response = self.parent.error(
     File "urllib2.py", line 396, in error
       result = self._call_chain(*args)
     File "urllib2.py", line 337, in _call_chain
       result = func(*args)
     File "/usr/lib/python2.4/site-packages/ClientCookie/_urllib2_support.py", line 154, in http_error_302
       return self.parent.open(new)
     File "/usr/lib/python2.4/site-packages/mechanize/_mechanize.py", line 254, in open
       return self._mech_open(url, data)
     File "/usr/lib/python2.4/site-packages/mechanize/_mechanize.py", line 277, in _mech_open
       self._response = UserAgent.open(self, self.request, data)
     File "urllib2.py", line 358, in open
       response = self._open(req, data)
     File "urllib2.py", line 376, in _open
       '_open', req)
     File "urllib2.py", line 337, in _call_chain
       result = func(*args)
     File "lib/zope/testbrowser/testing.py", line 112, in http_open
       return self.do_open(PublisherConnection, req)
     File "urllib2.py", line 993, in do_open
       h.request(req.get_method(), req.get_selector(), req.data, headers)
     File "lib/zope/testbrowser/testing.py", line 69, in request
       self.response = self.caller(request_string, handle_errors)
     File "lib/zope/app/testing/functional.py", line 581, in __call__
       environment)
     File "lib/zope/app/testing/functional.py", line 614, in chooseRequestClass
       return chooseClasses(method, environment)
     File "lib/zope/app/publication/httpfactory.py", line 34, in chooseClasses
       request_class, publication = factory()
   TypeError: 'NoneType' object is not callable

[Benji York (benji)]

> = Request - Entry #1 by jamesh on Feb 7, 2007 6:35 am
>
> I would like to be able to use zope.testbrowser to test a redirect to a
> remote site.

I've wanted to do this myself, but there are a couple of downsides. First, I believe this will complicate the code somewhat for what is a pretty rare need.

The other is in the class of "you shouldn't want to do that anyway". Requiring network connectivity for a test should be avoided when possible, and you don't want to test that the other site is available, only that your app redirects properly.

Here are some options (in the order I would prefer if this were my test):

* create a custom testbrowser subclass that disables the redirection plugin that urllib2 (and therefore mechanize and testbrowser) uses and then inspect the headers to ensure the redirect happens as you expect

* for this part of your test use the "old style" functional tests to make sure you get a 301 redirect to the site you want to go to.

* use the non-Zope publisher version of testbrowser for this test (this means your test setup will have to start a "real" Zope instance to listen on a port)

Here's a rough implementation of the first option (only lightly tested):

 from zope.testbrowser.testing import PublisherMechanizeBrowser
 from zope.testbrowser.browser import Browser

 my_features = PublisherMechanizeBrowser.default_features[:]
 my_features.remove('_redirect')

 class NonRedirectingMechanizeBrowser(PublisherMechanizeBrowser):
     default_features = my_features

 browser = Browser(mech_browser=NonRedirectingMechanizeBrowser())
 browser.open('http://launchpad.dev/ubuntu')
 assert browser.headers['location'] == 'redirected URL'

[James Henstridge (jamesh)]

I ended up doing the test something like this:

    >>> browser.mech_browser.set_handle_redirect(False)
    >>> from urllib2 import HTTPError
    >>> try:
    ... browser.open('url-that-should-redirect')
    ... except HTTPError, exc:
    ... print str(exc)
    ... print exc.headers.getheader('Location')
    HTTP Error 303: See Other
    redirect-destination-url

Accessing the network from the tests was never my intention: rather just to check that the destination of the redirect was correct.

Maybe some API to do the equivalent of the set_handle_redirect(False) call would be appropriate?

[Benji York (benji)]

> = Comment - Entry #3 by jamesh on Feb 7, 2007 10:57 am
>
> I ended up doing the test something like this:
>
> >>> browser.mech_browser.set_handle_redirect(False)

I'll note that the mechanize bits in testbrowser are not part of the public interface, and mechanize itself is evolving, so that code may break in the future. I suspect the risk is worth it to you though.

> Maybe some API to do the equivalent of the set_handle_redirect(False)
> call would be appropriate?

Perhaps. If you prepared a patch, it's fairly likely that the feature would make it in.

[James Henstridge (jamesh)]

I realise that accessing the mechanize browser object directly is not a supported API. I posted the method I used since I thought it would be useful in deciding on an appropriate API for testbrowser.

[Brian Sutherland (jinty)]

With zope.testbrowser.wsgi.Browser in zope.testbrowser 4.0.0 a HostNotAllowed error is now raised when you try to access a domain that is not localhost, *.example.com, *.example.net or *.example.org:

    >>> browser.open('http://localhost/redirect.html?to=http://www.google.com')
    Traceback (most recent call last):
        ...
    HostNotAllowed: http://www.google.com/

I think this is sufficient to test redirects to remote sites without actually visiting them.

Also since the old zope.app.testing connection is deprecated, I'm going closing this issue.