POST to authenticating proxy causes "necessary data rewind wasn't possible" error

[Carra Bussa]

Hi, new Bazaar user. New WinXP SP3 install, all patched and working. Install bazaar via bzr-setup-1.5. Quick sanity check OK (init, add, commit, diff, log), all local.

I'm behind an authenticating proxy, so entered: set http_proxy=http://id:<email address hidden>:8080/
[ wget.exe fails w/o set; add set, wget works. Credentials and settings *are* valid.]

Tried to do the example in the user guide, 2.1.5, step 2 -- bzr branch http://bazaar-vcs.org/bzr/bzr.dev

Starts, then fails with dump (below.) Going there (http://bazaar-vcs.org/bzr/bzr.dev) in a browser give a nice directory listing of 19 files and dirs. I don't have a python install except for bzr-setup, so if I'm missing a support module -- well, I probably *am*.

10:16:42 C:\temp\xyz>bzr branch http://bazaar-vcs.org/bzr/bzr.dev
bzr: ERROR: pycurl.error: (65, "necessary data rewind wasn't possible\n")

Traceback (most recent call last):
  File "bzrlib\commands.pyc", line 846, in run_bzr_catch_errors
  File "bzrlib\commands.pyc", line 797, in run_bzr
  File "bzrlib\commands.pyc", line 499, in run_argv_aliases
  File "bzrlib\builtins.pyc", line 916, in run
  File "bzrlib\bzrdir.pyc", line 789, in open_tree_or_branch
  File "bzrlib\bzrdir.pyc", line 667, in open
  File "bzrlib\bzrdir.pyc", line 704, in open_from_transport
  File "bzrlib\lazy_import.pyc", line 125, in __call__
  File "bzrlib\transport\__init__.pyc", line 1616, in do_catching_redirections
  File "bzrlib\bzrdir.pyc", line 681, in find_format
  File "bzrlib\bzrdir.pyc", line 1440, in find_format
  File "bzrlib\bzrdir.pyc", line 2398, in probe_transport
  File "bzrlib\smart\medium.pyc", line 421, in protocol_version
  File "bzrlib\smart\protocol.pyc", line 620, in query_version
  File "bzrlib\smart\protocol.pyc", line 488, in call
  File "bzrlib\smart\medium.pyc", line 348, in finished_writing
  File "bzrlib\transport\http\__init__.pyc", line 539, in _finished_writing
  File "bzrlib\transport\http\__init__.pyc", line 518, in send_http_smart_request
  File "bzrlib\transport\http\_pycurl.pyc", line 268, in _post
  File "bzrlib\transport\http\_pycurl.pyc", line 325, in _curl_perform
error: (65, "necessary data rewind wasn't possible\n")

bzr 1.5 on python 2.5.2 (win32)
arguments: ['bzr', 'branch', 'http://bazaar-vcs.org/bzr/bzr.dev']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  launchpad C:\Program Files\Bazaar\plugins\launchpad [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

[Andrew Bennetts (spiv)]

That's pretty weird.

Could you try these URLs instead:

 - http+urllib://bazaar-vcs.org/bzr/bzr.dev
 - nosmart+http://bazaar-vcs.org/bzr/bzr.dev

I'm guessing they'll both workaround this problem, but it would be good to confirm.

[anatoly techtonik (techtonik)]

I can try, because I have the sane "necessary data rewind wasn't possible\n" bzr 1.5 error when specifying proxy with http_proxy=http://id:<email address hidden>:8080 on windows 2000 in domain environment.

 - http+urllib://bazaar-vcs.org/bzr/bzr.dev
gives
   bzr: ERROR: Invalid http response for http://bazaar-vcs.org/bzr/bzr.dev/.bzr/branch-format: Unable to handle http code 407: Proxy Authentication Required
or if I omit "http://" prefix from proxy definition, i.e. http_proxy=id:<email address hidden>:8080
   bzr: ERROR: Invalid url supplied to transport: "Host empty in: id:<email address hidden>:8080"

 - nosmart+http://bazaar-vcs.org/bzr/bzr.dev
works ok with both prefixed with "http://" and plain URL http_proxy setting

[Andrew Bennetts (spiv)]

techtonik, thanks very much for that! That's a great help.

[Carra Bussa (q-launchpad-bussa-com)]

Using http+urllib://bazaar-vcs.org/bzr/bzr.dev worked (!), but using http:// didn't.

The problem seems to be in a proxy support library -- look at the attached sniffer traces. Using HTTP: fails, but then never, ever retries. http+urllib: starts the same way, but then immediately uses the proxy environment set and keeps on going.

Once again, I'm on a Win32 SP3 system with the default Win32 Bazaar install, but also behind a blocking, authenticating proxy -- if I don't first successfullly authenticate to the proxy on 8080, I don't go _ANY_where.

I made this a bug because I might be the only one that has this right now, but I bet I won't be in the future. Thanks!

[Andrew Schulman (andrex)]

I have the same error when I try to branch from an http server with digest authentication, and no proxy.

$ bzr branch -v http://<email address hidden>/tst1/tst2/tst3
bzr: ERROR: pycurl.error: (65, "necessary data rewind wasn't possible")

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 916, in run
    from_location)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 789, in open_tree_or_branch
    bzrdir = klass.open(location)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 667, in open
    return BzrDir.open_from_transport(t, _unsupported=_unsupported)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 704, in open_from_transport
    redirected)
  File "/usr/lib/python2.5/site-packages/bzrlib/lazy_import.py", line 125, in __call__
    return obj(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/__init__.py", line 1616, in do_catching_redirections
    return action(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 681, in find_format
    transport, _server_formats=_server_formats)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 1440, in find_format
    return format.probe_transport(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 2398, in probe_transport
    server_version = medium.protocol_version()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 421, in protocol_version
    self._protocol_version = client_protocol.query_version()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/protocol.py", line 620, in query_version
    self.call('hello')
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/protocol.py", line 488, in call
    self._request.finished_writing()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 348, in finished_writing
    self._finished_writing()
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/http/__init__.py", line 539, in _finished_writing
    data = self._medium.send_http_smart_request(self._buffer)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/http/__init__.py", line 518, in send_http_smart_request
    code, body_filelike = self._post(bytes)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/http/_pycurl.py", line 268, in _post
    self._curl_perform(curl, header, ['Expect: '])
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/http/_pycurl.py", line 325, in _curl_perform
    curl.perform()
error: (65, "necessary data rewind wasn't possible")

bzr 1.5 on python 2.5.2 (linux2)
arguments: ['/usr/bin/bzr', 'branch', '-v', 'http://<email address hidden>/tst1/tst2/tst3']
encoding: 'ISO-8859-1', fsenc: 'ISO-8859-1', lang: 'en_US'
plugins:
  bzrtools /usr/lib/python2.5/site-package...

[Michael B. Trausch (mtrausch)]

Using bzr 1.7rc1 (Windows installer bundle) on Windows XP Professional Service Pack 3.

I've a project that I am working on for a client and using bzr for, and I uploaded it to the client's Web server (https with basic HTTP authentication). When the client (who uses Windows) attempted to do a bzr branch on it, he was greeted with the "necessary data rewind wasn't possible" error, which appears to have originated from bzr's use of pycurl. Using https+urllib:// for the protocol designation in the branch operation works.

I was able to confirm this on a second XP Pro SP3 machine, as well.

The operation succeeds on an Ubuntu 8.04 machine running bzr 1.7rc1 as well as 1.6.1 so it appears to be specific to the combination of Windows and pycurl.

[Andrew Schulman (andrex)]

> it appears to be specific to the combination of Windows and pycurl.

No, since I'm running Debian.

[Michael B. Trausch (mtrausch)]

On Mon, Sep 15, 2008 at 09:15:59PM -0000, Andrew Schulman wrote:
> > it appears to be specific to the combination of Windows and pycurl.
>
> No, since I'm running Debian.
>

Sorry, I should have said "it appears to be specific to the default
install of bzr, Windows, and pycurl" since (at least on my Ubuntu
system) this problem does not occur out-of-the-box, whereas it does on
Windows out-of-the-box.

   --- Mike

--
My sigfile ran away and is on hiatus.

[Ned Batchelder (nedbatchelder)]

I also have this problem on Windows. I don't think there's a proxy between me and my repo. I'm using bzr against an SVN repo. I tried the suggestions in this thread:

C:\dev> bzr branch https://myrepohost.cpqcorp.net/myrepo/trunk bzrtrunk
bzr: ERROR: Invalid http response for https://myrepohost.cpqcorp.net/myrepo/trunk/.bzr/branch-format: Unable to handle http code
401: expected 200 or 404 for full response.

C:\dev> bzr branch https://ned:<email address hidden>/bitscrap/InternalProjects/Sources/bitty/trunk bzrtrunk
bzr: ERROR: pycurl.error: (65, "necessary data rewind wasn't possible")

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 857, in run_bzr_catch_errors
  File "bzrlib\commands.pyo", line 797, in run_bzr
  File "bzrlib\commands.pyo", line 499, in run_argv_aliases
  File "bzrlib\builtins.pyo", line 853, in run
  File "bzrlib\bzrdir.pyo", line 894, in open_tree_or_branch
  File "bzrlib\bzrdir.pyo", line 772, in open
  File "bzrlib\bzrdir.pyo", line 809, in open_from_transport
  File "bzrlib\lazy_import.pyo", line 125, in __call__
  File "bzrlib\transport\__init__.pyo", line 1616, in do_catching_redirections
  File "bzrlib\bzrdir.pyo", line 786, in find_format
  File "bzrlib\bzrdir.pyo", line 1636, in find_format
  File "bzrlib\bzrdir.pyo", line 2619, in probe_transport
  File "bzrlib\smart\medium.pyo", line 531, in protocol_version
  File "bzrlib\smart\protocol.pyo", line 769, in query_version
  File "bzrlib\smart\protocol.pyo", line 617, in call
  File "bzrlib\smart\medium.pyo", line 416, in finished_writing
  File "bzrlib\transport\http\__init__.pyo", line 553, in _finished_writing
  File "bzrlib\transport\http\__init__.pyo", line 520, in send_http_smart_request
  File "bzrlib\transport\http\_pycurl.pyo", line 267, in _post
  File "bzrlib\transport\http\_pycurl.pyo", line 341, in _curl_perform
error: (65, "necessary data rewind wasn't possible")

bzr 1.7.1 on python 2.5.2 (win32)
arguments: ['bzr', 'branch', 'https://ned:<email address hidden>/bitscrap/InternalProjects/Sources/bitty/trunk', 'bzrtrunk']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files\Bazaar\plugins\bzrtools [1.7.0]
  launchpad C:\Program Files\Bazaar\plugins\launchpad [unknown]
  qbzr C:\Program Files\Bazaar\plugins\qbzr [0.9.4]
  svn C:\Program Files\Bazaar\plugins\svn [0.4.13]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

C:\dev> bzr branch https+urllib://ned:<email address hidden>/bitscrap/InternalProjects/Sources/bitty/trunk bzrtrunk
bzr: ERROR: Unprintable exception RaRequestFailed: dict={'_preformatted_string': None, 'mesage': "OPTIONS of 'https+urllib://<email address hidden>/bitscrap/InternalProjects/Sources/bitty/trunk': could not connect to server (https+urllib://code.mro.cpqcorp.net:0)"}, fmt='A Subversion remote access command failed: %(message)', error=ValueError('incomplete format',)

C:\dev> bzr branch http+urllib://ned:<email address hidden>/bit...

[Ned Batchelder (nedbatchelder)]

Sorry, just to clarify: I meant to simplify the repo url through that session, but it only changed once. Consider "myrepohost.cpqcorp.net/myrepo/trunk" and "code.mro.cpqcorp.net/bitscrap/InternalProjects/Sources/bitty/trunk" as the same thing.

And to sum up the final result: once I used "svn+http:" as the protocol, it worked, even though it told me to use a different syntax which didn't work.