Redirect to location protected by HTTP auth fails

[Glen Mailer]

When attempting to access a URL which redirects to a new URL, where the latter URL requires HTTP Authentication, Bzr throws a KeyError on the 'user' property of request.auth

I've set up a simple test case here:
bzr info http://glenjamin.co.uk/other/bzr/test

As far as I can tell, the auth dictionary is populated outside of the Request object initialisation, so when the Redirect creates a new request object, it doesnt get it's auth dict populated - leading to the key errors. I was hoping to be able to provide a fix, but I got lost in the program flow, here are the points of note:

Request.auth gets initialised as empty dict - bzrlib/transport/http/_urllib2_wrappers.py:353
Auth dict gets populated - bzrlib/transport/http/_urllib.py:69
Redirect creates new Request - bzrlib/transport/http/_urllib2_wrappers.py:749 (this one never gets its auth populated)

Bzr 1.16.1, reproduced on windows vista and mac os x

C:\temp>bzr info http://glenjamin.co.uk/other/bzr/test
bzr: ERROR: exceptions.KeyError: 'user' 0KB/s |

Traceback (most recent call last):
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\commands.py", line 729, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\commands.py", line 924, in run_bzr
    ret = run(*run_argv)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\commands.py", line 560, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\commands.py", line 939, in ignore_pipe
    result = func(*args, **kwargs)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\builtins.py", line 1366, in run
    show_bzrdir_info(bzrdir.BzrDir.open_containing(location)[0],
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\bzrdir.py", line 892, in open_containing
    return BzrDir.open_containing_from_transport(transport)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\bzrdir.py", line 913, in open_containing_from_transport
    result = BzrDir.open_from_transport(a_transport)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\bzrdir.py", line 867, in open_from_transport
    redirected)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\lazy_import.py", line 125, in __call__
    return obj(*args, **kwargs)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\transport\__init__.py", line 1642, in do_catching_redirections
    return action(transport)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\bzrdir.py", line 854, in find_format
    transport, _server_formats=_server_formats)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\bzrdir.py", line 1785, in find_format
    return format.probe_transport(transport)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\plugins\svn\format.py", line 109, in probe_transport
    transport = get_svn_ra_transport(transport)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\plugins\svn\transport.py", line 104, in get_svn_ra_transport
    resp = bzr_transport._perform(req)
  File "C:\dev\bin\python25\Lib\site-packages\bzrlib\transport\http\_urllib.py", line 79, in _perform
    response = self._opener.open(request)
  File "C:\dev\bin\python25\lib\urllib2.py", line 387, in open
    response = meth(req, response)
  File "c:\dev\bin\python25\lib\site-packages\bzrlib\transport\http\_urllib2_wrappers.py", line 1530, in http_response
    code, msg, hdrs)
  File "C:\dev\bin\python25\lib\urllib2.py", line 419, in error
    result = self._call_chain(*args)
  File "C:\dev\bin\python25\lib\urllib2.py", line 360, in _call_chain
    result = func(*args)
  File "c:\dev\bin\python25\lib\site-packages\bzrlib\transport\http\_urllib2_wrappers.py", line 815, in http_error_302
    return self.parent.open(redirected_req)
  File "C:\dev\bin\python25\lib\urllib2.py", line 387, in open
    response = meth(req, response)
  File "c:\dev\bin\python25\lib\site-packages\bzrlib\transport\http\_urllib2_wrappers.py", line 1530, in http_response
    code, msg, hdrs)
  File "C:\dev\bin\python25\lib\urllib2.py", line 419, in error
    result = self._call_chain(*args)
  File "C:\dev\bin\python25\lib\urllib2.py", line 360, in _call_chain
    result = func(*args)
  File "c:\dev\bin\python25\lib\site-packages\bzrlib\transport\http\_urllib2_wrappers.py", line 1441, in http_error_401
    return self.auth_required(req, headers)
  File "c:\dev\bin\python25\lib\site-packages\bzrlib\transport\http\_urllib2_wrappers.py", line 1054, in auth_required
    matching_handler = self.auth_match(server_header, auth)
  File "c:\dev\bin\python25\lib\site-packages\bzrlib\transport\http\_urllib2_wrappers.py", line 1290, in auth_match
    if auth['user'] is None or auth['password'] is None:
KeyError: 'user'

bzr 1.16.1 on python 2.5.4 (win32)
arguments: ['C:\\pys\\bzr.py', 'info', 'http://glenjamin.co.uk/other/bzr/test']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  bzrtools c:\dev\bin\python25\lib\site-packages\bzrlib\plugins\bzrtools [1.16]
  gtk C:\Users\Glen\AppData\Roaming\bazaar\2.0\plugins\gtk [0.95.0.final.1]
  launchpad c:\dev\bin\python25\lib\site-packages\bzrlib\plugins\launchpad [1.16.1]
  netrc_credential_store c:\dev\bin\python25\lib\site-packages\bzrlib\plugins\netrc_credential_store [1.16.1]
  qbzr c:\dev\bin\python25\lib\site-packages\bzrlib\plugins\qbzr [0.9.9]
  svn c:\dev\bin\python25\lib\site-packages\bzrlib\plugins\svn [0.6.2]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

[Neil Martinsen-Burrell (nmb)]

I have the same problem with a Loggerhead instance that is proxied by Apache. The Apache config has authentication required for the path that is being passed on the local loggerhead instance. I see the same error in this case when doing ``bzr branch http://server.example.com/bzr/branch_name`` where everything under /bzr is being passed on to the loggerhead server.

[Vincent Ladeuil (vila)]

I tried reproducing the problem with a recent version (> 2.0) and got prompted as I expected for user and password.

@Neil: Do you have a way to reproduce the problem ?

I had a look at your patch but I don't think the approach is right as it will blindly propagate the credentials from the original host to the redirected one, which is not right in all cases.

Instead, there is a couple of places (auth_match only as far as I can see) where we just want to replace the auth['user'] by auth.get('user', None).

[Neil Martinsen-Burrell (nmb)]

I can reproduce this using bzr.dev r4795 on the original poster's URL:

$ bzr info http://glenjamin.co.uk/other/bzr/test
bzr: ERROR: exceptions.KeyError: 'user'

Traceback (most recent call last):
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/commands.py", line 843, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/commands.py", line 1038, in run_bzr
    ret = run(*run_argv)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/commands.py", line 655, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/commands.py", line 1053, in ignore_pipe
    result = func(*args, **kwargs)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/builtins.py", line 1468, in run
    show_bzrdir_info(bzrdir.BzrDir.open_containing(location)[0],
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/bzrdir.py", line 898, in open_containing
    return BzrDir.open_containing_from_transport(transport)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/bzrdir.py", line 919, in open_containing_from_transport
    result = BzrDir.open_from_transport(a_transport)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/bzrdir.py", line 873, in open_from_transport
    redirected)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/lazy_import.py", line 125, in __call__
    return obj(*args, **kwargs)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/transport/__init__.py", line 1644, in do_catching_redirections
    return action(transport)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/bzrdir.py", line 860, in find_format
    transport, _server_formats=_server_formats)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/bzrdir.py", line 1820, in find_format
    return format.probe_transport(transport)
  File "/Users/nmb/.bazaar/plugins/svn/format.py", line 102, in probe_transport
    transport = get_svn_ra_transport(transport)
  File "/Users/nmb/.bazaar/plugins/svn/transport.py", line 102, in get_svn_ra_transport
    resp = bzr_transport._perform(req)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/transport/http/_urllib.py", line 79, in _perform
    response = self._opener.open(request)
  File "/Library/Frameworks/Python.framework/Versions/5.0.0/lib/python2.5/urllib2.py", line 387, in open
    response = meth(req, response)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/transport/http/_urllib2_wrappers.py", line 1551, in http_response
    code, msg, hdrs)
  File "/Library/Frameworks/Python.framework/Versions/5.0.0/lib/python2.5/urllib2.py", line 419, in error
    result = self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/5.0.0/lib/python2.5/urllib2.py", line 360, in _call_chain
    result = func(*args)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/transport/http/_urllib2_wrappers.py", line 836, in http_error_302
    return self.parent.open(redirected_req)
  File "/Library/Frameworks/Python.framework/Versions/5.0.0/lib/python2.5/urllib2.py", line 387, in open
    response = meth(req, response)
  File "/Users/nmb/src/bzr/bzr.dev/bzrlib/transport/http/_urllib2_wrappers.py", line 1551, in http_response
    code, msg, hdrs)
  File "/Library/Frameworks/Python.framework/Versions/5.0.0/lib/python2.5/urllib2.py", line 419, in error
    result = self._call_cha...

[Neil Martinsen-Burrell (nmb)]

Using lp:~vila/bzr/395714-auth-redirect gives me the same problem in a different place:

nmb@guttle[~/src/bzr/395714]$ ./bzr info http://glenjamin.co.uk/other/bzr/test
bzr: ERROR: exceptions.KeyError: 'user'

Traceback (most recent call last):
  File "/Users/nmb/src/bzr/395714/bzrlib/commands.py", line 843, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/Users/nmb/src/bzr/395714/bzrlib/commands.py", line 1038, in run_bzr
    ret = run(*run_argv)
  File "/Users/nmb/src/bzr/395714/bzrlib/commands.py", line 655, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/Users/nmb/src/bzr/395714/bzrlib/commands.py", line 1053, in ignore_pipe
    result = func(*args, **kwargs)
  File "/Users/nmb/src/bzr/395714/bzrlib/builtins.py", line 1468, in run
    show_bzrdir_info(bzrdir.BzrDir.open_containing(location)[0],
  File "/Users/nmb/src/bzr/395714/bzrlib/bzrdir.py", line 898, in open_containing
    return BzrDir.open_containing_from_transport(transport)
  File "/Users/nmb/src/bzr/395714/bzrlib/bzrdir.py", line 919, in open_containing_from_transport
    result = BzrDir.open_from_transport(a_transport)
  File "/Users/nmb/src/bzr/395714/bzrlib/bzrdir.py", line 873, in open_from_transport
    redirected)
  File "/Users/nmb/src/bzr/395714/bzrlib/lazy_import.py", line 125, in __call__
    return obj(*args, **kwargs)
  File "/Users/nmb/src/bzr/395714/bzrlib/transport/__init__.py", line 1644, in do_catching_redirections
    return action(transport)
  File "/Users/nmb/src/bzr/395714/bzrlib/bzrdir.py", line 860, in find_format
    transport, _server_formats=_server_formats)
  File "/Users/nmb/src/bzr/395714/bzrlib/bzrdir.py", line 1820, in find_format
    return format.probe_transport(transport)
  File "/Users/nmb/.bazaar/plugins/svn/format.py", line 102, in probe_transport
    transport = get_svn_ra_transport(transport)
  File "/Users/nmb/.bazaar/plugins/svn/transport.py", line 102, in get_svn_ra_transport
    resp = bzr_transport._perform(req)
  File "/Users/nmb/src/bzr/395714/bzrlib/transport/http/_urllib.py", line 79, in _perform
    response = self._opener.open(request)
  File "/Library/Frameworks/Python.framework/Versions/5.0.0/lib/python2.5/urllib2.py", line 387, in open
    response = meth(req, response)
  File "/Users/nmb/src/bzr/395714/bzrlib/transport/http/_urllib2_wrappers.py", line 1552, in http_response
    code, msg, hdrs)
  File "/Library/Frameworks/Python.framework/Versions/5.0.0/lib/python2.5/urllib2.py", line 419, in error
    result = self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/5.0.0/lib/python2.5/urllib2.py", line 360, in _call_chain
    result = func(*args)
  File "/Users/nmb/src/bzr/395714/bzrlib/transport/http/_urllib2_wrappers.py", line 836, in http_error_302
    return self.parent.open(redirected_req)
  File "/Library/Frameworks/Python.framework/Versions/5.0.0/lib/python2.5/urllib2.py", line 387, in open
    response = meth(req, response)
  File "/Users/nmb/src/bzr/395714/bzrlib/transport/http/_urllib2_wrappers.py", line 1552, in http_response
    code, msg, hdrs)
  File "/Library/Frameworks/Python.framework/Versions/5.0.0/lib/python2.5/urllib2.py", line 419, in er...

[Neil Martinsen-Burrell (nmb)]

Is it true that you are unable to reproduce this error on this URL: http://glenjamin.co.uk/other/bzr/test? If so, perhaps we should be looking at differences between our setups to pin down where this is coming from.

[Vincent Ladeuil (vila)]

Yes, I'm unable to reproduce the problem with http://glenjamin.co.uk/other/bzr/test :
./bzr info http://glenjamin.co.uk/other/bzr/test
http://glenjamin.co.uk/other/bzr/test/ is redirected to http://glenjamin.co.uk/other/bzr/httpauth/test/
HTTP glenjamin.co.uk, Realm: 'Restricted Files' username: dfg
HTTP <email address hidden>, Realm: 'Restricted Files' password: sdfg

bzr: ERROR: Invalid http response for http://glenjamin.co.uk/other/bzr/httpauth/test/.bzr/branch-format: Unable to handle http code 401: Authorization Required

But that may be because the redirection is to the same host for me but nor for you due to some internal LAN configuration ?

I've pushed an additional patch (revno 4799).

Is there a way we can meet on IRC to accelerate the resolution :) ?

[Vincent Ladeuil (vila)]

So, the summary is that bzr-svn was using the old code path to handle the http redirections and that code path
wasn't handling the authentication.

The associated branch implements that.