curl error: necessary data rewind wasn't possible

[Josh Matthews]

I've been using bzr 1.13 and bzr-svn 0.5.3 with Fedora 8 and pushing and pulling to/from a SourceForge project svn repo. I recently updated to Fedora 10, and I now receive this error on `bzr up`:

[t_mattjo@Phaethon fheroes2-trunk]$ bzr switch https://<email address hidden>/svnroot/fheroes2/trunk/fheroes2
bzr: ERROR: pycurl.error: (65, "necessary data rewind wasn't possible")

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 716, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 911, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 547, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 5170, in run
    to_branch = Branch.open(to_location)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 123, in open
    possible_transports=possible_transports)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 799, in open
    return BzrDir.open_from_transport(t, _unsupported=_unsupported)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 827, in open_from_transport
    redirected)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/__init__.py", line 1680, in do_catching_redirections
    return action(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 814, in find_format
    transport, _server_formats=_server_formats)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 1667, in find_format
    return format.probe_transport(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 2779, in probe_transport
    server_version = medium.protocol_version()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 611, in protocol_version
    client_protocol.query_version()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/protocol.py", line 777, in query_version
    self.call('hello')
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/protocol.py", line 617, in call
    self._request.finished_writing()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 436, in finished_writing
    self._finished_writing()
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/http/__init__.py", line 647, in _finished_writing
    data = self._medium.send_http_smart_request(self._buffer)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/http/__init__.py", line 616, in send_http_smart_request
    code, body_filelike = t._post(bytes)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/http/_pycurl.py", line 268, in _post
    self._curl_perform(curl, header, ['Expect: '])
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/http/_pycurl.py", line 356, in _curl_perform
    curl.perform()
error: (65, "necessary data rewind wasn't possible")

bzr 1.13.1 on python 2.5.2 (linux2)
arguments: ['/usr/bin/bzr', 'switch', 'https://<email address hidden>/svnroot/fheroes2/trunk/fheroes2']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  automv /home/t_mattjo/.bazaar/plugins/automv [unknown]
  bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.13]
  dns_cache /home/t_mattjo/.bazaar/plugins/dns_cache [unknown]
  extmerge /home/t_mattjo/.bazaar/plugins/extmerge [unknown]
  lastlog /home/t_mattjo/.bazaar/plugins/lastlog [unknown]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
  lessdiff /home/t_mattjo/.bazaar/plugins/lessdiff [unknown]
  lesslog /home/t_mattjo/.bazaar/plugins/lesslog [unknown]
  netrc_credential_store /usr/lib/python2.5/site-packages/bzrlib/plugins/netrc_credential_store [unknown]
  precommit_bugtraq /home/t_mattjo/.bazaar/plugins/precommit_bugtraq [unknown]
  search /home/t_mattjo/.bazaar/plugins/search [1.7dev]
  stats /home/t_mattjo/.bazaar/plugins/stats [unknown]
  svn /home/t_mattjo/.bazaar/plugins/svn [0.5.3]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

I've tried using svn+https:// along with http://, but neither makes any difference.

[Josh Matthews (joshmatthews)]

I tried updating to bzr 1.14rc2 and bzr-svn 0.5.4, and this is still broken.

[Jelmer Vernooij (jelmer)]

This problem is unrelated to bzr-svn.

[Jelmer Vernooij (jelmer)]

Uninstalling pycurl may fix it.

[Josh Matthews (joshmatthews)]

Unfortunately uninstalling pycurl via yum tries to uninstall bazaar as well, which is not very helpful. However, I seem to have been able to work around this problem by creating a new checkout with `bzr checkout svn+http://fheroes2.svn.sourceforge.net/svnroot/fheroes2/trunk/fheroes2`

[Vincent Ladeuil (vila)]

This bug comes back rarely but we never get enough information to reproduce it.

If you're stiil experiencing it, can you check against both http and https
 (I know that may be difficult as you may not be able to access the same repo with both protocols).

[unchiujar (vasile-jureschi)]

Any bazaar command run with authentication gives this
bzr log bzr+https://<email address hidden>/project/aport/master/trunk

bzr: ERROR: pycurl.error: (65, "necessary data rewind wasn't possible")

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    attaching the crash file
        /home/vasile/.cache/crash/bzr-20091113112957-10814.crash
    and including a description of the problem.

[unchiujar (vasile-jureschi)]

Installing the latest version from the nightly build gives the same error.

[unchiujar (vasile-jureschi)]

Found and the problem and hacked around it.

It seems bazaar doesn't like python2.5 and 2.6. Changing the system version to 2.4 fixed the problem.

These are the steps I did to fix it on my system.
 # Install python2.4
 # edit: /usr/share/python/debian_defaults
   and change
   default-version=python2.4
 # Change the /usr/bin/python symlink from /usr/bin/python2.6 or 2.5 to /usr/bin/python2.4
   rm /usr/bin/python
   ln -s /usr/bin/python2.4 /usr/bin/python
 # Add to $PYTHONPATH the path to bzrlin. I did export PYTHONPATH=/usr/share/pyshared:$PYTHONPATH

Do this only if you really need bazaar because this is certain to break programs that need 2.5 or 2.6

If someone from the bazaar team that actually knows how to fix this problem would get around to it, it would be nice.

Thanks :)

[unchiujar (vasile-jureschi)]

An alternative method of hacking around this problem is to install virtualenv http://pypi.python.org/pypi/virtualenv

Create virtual environment:
virtualenv -p python2.4 /usr/share/virtual_python2.4

Edit the bazaar executable:

nano -w $(which bzr)

and replace #!/usr/bin/python with #!/usr/share/virtual_python2.4/bin/python

Also run export PYTHONPATH=/usr/share/pyshared to give the path to bzrlib

This keeps the current version of python while running bzr with python 2.4

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

unchiujar wrote:
> An alternative method of hacking around this problem is to install
> virtualenv http://pypi.python.org/pypi/virtualenv
>
> Create virtual environment:
> virtualenv -p python2.4 /usr/share/virtual_python2.4
>
> Edit the bazaar executable:
>
> nano -w $(which bzr)
>
> and replace #!/usr/bin/python with
> #!/usr/share/virtual_python2.4/bin/python
>
> Also run export PYTHONPATH=/usr/share/pyshared to give the path to
> bzrlib
>
> This keeps the current version of python while running bzr with python
> 2.4
>

My guess is that it isn't python2.5 or 2.6 but that you don't have
pycurl installed in python2.4.

You can directly work around it with "http+urllib://" style urls, or
disable/uninstall pycurl.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksBhG0ACgkQJdeBCYSNAAPWEgCePR17WjQNmS1CEgrYwQh9ztsy
OQAAoJoccsa1YSOHP4WpvDAeAzqUDlS2
=Rsnp
-----END PGP SIGNATURE-----

[unchiujar (vasile-jureschi)]

The issue appears when I specifically use https (I am forced to use https by the way the repository I work with is configured).

"My guess is that it isn't python2.5 or 2.6 but that you don't have
pycurl installed in python2.4."

I just checked and after removing pycurl bzr works with python 2.6 so I you are right.

"http+urllib://" also works :)

Thanks :D

[Vincent Ladeuil (vila)]

Could you attach the relevant part of your .bzr.log ('bzr version' will tell you where it is) after issuing
the failing command adding '-Dhttp' in the parameters ?

That will provide more information about how the bug occurs.
As mentioned in previous comments, this behavior is rare and hard to reproduce, so while the workaround is perfectly
valid, it will be good to better understand the issue.

[unchiujar (vasile-jureschi)]

Log after running:

 bzr info -Dhttp bzr+https://<email address hidden>/project/aport/master/trunk

is attached.

Hope this helps :)

[Khairul Aizat Kamarudzzaman (fenris)]

cannot remove python-pycurl because ... some other application depend on it such as gwibber-services

[Vincent Ladeuil (vila)]

I'm pretty sure this has been fixed with recent versions of bzr/pycurl/gnutls, re-open and attach your log file as explained above if you run into this bug again.

[Martin Pool (mbp)]

Given the recent apparent dupe bug 722001 I think this is not actually fixed.

I think this is something do do with http 100-continue. It may be specific to servers (or proxies) that either do send it or don't send it, or perhaps send it even when they're not asked to.

[Vincent Ladeuil (vila)]

I never thought about http 100-continue (and my gut feeling/confused memories on the topic is that the client initiates the 100-continue and we explicitly don't do that), but I may be wrong.

I thought this bug was triggered by some 401 error requiring resending the request (hence the rewind).

In both cases, a .bzr.log file for the failing command with -Dhttp added is the best way to gather the data needed for the diagnosis.

So if the failing command is:

   bzr pull <url>

do

  bzr pull -Dhttp <url>

and attach the relevant part of ~/.bzr.log (bzr version will tell you where it's located).

Make sure there is no sensitive header contents in the excerpt (any header with auth in its name is potentially leaking credentials). If there are such headers, edit their *value* only. We don't need the value for the diagnosis, but *not* having the header at *all* will led us to wrong conclusions.

I filed bug #723074 about credentials in .bzr.log.

[Marius Kruger (amanica)]

Failing log as requested. I did not see any headers content, maybe I'm running the wrong thing. Let me know if you need me to run/test anything else.
(This is on natty with a fairly recent bzr.dev)

[Marius Kruger (amanica)]

I have this problem with bzr and bzr-svn to the same server, here is a log when trying with svn.
With bzr branches I can work arround it with https+urllib:// , but with bzr-svn I get another error:
$ bzr push https+urllib://<svn branch>
bzr: ERROR: Transport operation not possible: http does not support mkdir()

[Marius Kruger (amanica)]

Ok now https://<svn branch> and https+urllib://<svn branch> works for me again (that error was just a bad way of saying bzr-svn is not installed)

[Lluís Vilanova (lluis.vilanova)]

I tried these two commands in the recent bazaar 2.5.0, resulting in the attached failure log:

bzr co https://code.server/bzr/project
bzr co https://<email address hidden>/bzr/project

As others said, using "https+urllib" works around the problem, but it is still a hassle to have it failing by default.

[Jaap Karssenberg (jaap.karssenberg)]

Got the same error while trying to set up bzr access through corporate proxy server with authentication.

I set the "http_proxy" environment to include user and password, this gives the following error:

bzr: ERROR: pycurl.error: (65, "necessary data rewind wasn't possible\n")

Traceback (most recent call last):
  File "bzrlib\commands.pyc", line 846, in run_bzr_catch_errors
  File "bzrlib\commands.pyc", line 797, in run_bzr
  File "bzrlib\commands.pyc", line 499, in run_argv_aliases
  File "bzrlib\builtins.pyc", line 916, in run
  File "bzrlib\bzrdir.pyc", line 789, in open_tree_or_branch
  File "bzrlib\bzrdir.pyc", line 667, in open
  File "bzrlib\bzrdir.pyc", line 704, in open_from_transport
  File "bzrlib\lazy_import.pyc", line 125, in __call__
  File "bzrlib\transport\__init__.pyc", line 1616, in do_catching_redirections
  File "bzrlib\bzrdir.pyc", line 681, in find_format
  File "bzrlib\bzrdir.pyc", line 1440, in find_format
  File "bzrlib\bzrdir.pyc", line 2398, in probe_transport
  File "bzrlib\smart\medium.pyc", line 421, in protocol_version
  File "bzrlib\smart\protocol.pyc", line 620, in query_version
  File "bzrlib\smart\protocol.pyc", line 488, in call
  File "bzrlib\smart\medium.pyc", line 348, in finished_writing
  File "bzrlib\transport\http\__init__.pyc", line 539, in _finished_writing
  File "bzrlib\transport\http\__init__.pyc", line 518, in send_http_smart_reques
t
  File "bzrlib\transport\http\_pycurl.pyc", line 268, in _post
  File "bzrlib\transport\http\_pycurl.pyc", line 325, in _curl_perform
error: (65, "necessary data rewind wasn't possible\n")

bzr 1.5 on python 2.5.2 (win32)
arguments: ['C:\\localdata\\Bazaar\\bzr.exe', 'branch', 'http://xxx']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  launchpad C:\localdata\Bazaar\plugins\launchpad [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

Using "http+urllib" instead results in
bzr: ERROR: Invalid http response for http://xxx: Unable to handle http code 407: Proxy Authentication Required

[Vincent Ladeuil (vila)]

@jaap: You're using bzr-1.5 which was released more than 4 years ago and is not maintained any more (but newer versions are backward compatible).

Can you try again with a more recent version (see https://launchpad.net/bzr/+download)

[Callum Macdonald (chmac)]

I receive this error when pushing to a bzr+https:// repository since installing pycurl. Prior to pycurl's installation, the push command worked. Since I installed pycurl, it now throws the "necessary data rewind wasn't possible" error. In my case, using https+urllib:// is not an option as my host does not support the plain HTTPS protocol.

Is there a way to force bzr to use urllib insteasd of pycurl while also using the bzr+https:// protocol? I tried bzr+https+urllib:// for example, but that does not work.

[Vincent Ladeuil (vila)]

@Callum: Please read the comments on this bug and follow the instructions there. Telling us which bzr version you're using is also needed to diagnose what is happening in your case.

Also, from your comment, it's a bit unclear what your context is, you say: "my host does not support the plain HTTPS protocol." and "I receive this error when pushing to a bzr+https:// repository". Either you expect https to work or you don't.

What do you mean ?

Note that in almost all cases where we got this error message from pycurl, it was tracked down as a bug in pycurl or one of its dependencies, so please make sure you're using an up-to-date system and tell us which versions you're using.

Finally, as recommended in some earlier comment, what happens if you uninstall pycurl ? Recent versions of bzr's urllib implementation support https.

[Wim Lewis (wiml-omni)]

FWIW, I think this can happen any time you set curl's READFUNCTION but do not set its SEEKFUNCTION.