TypeError on HTTP smart server

[Glen Mailer]

This only happened with a branch of a rails project I had, and it seemed to be a certain project only - if necessary I may be able to provide a copy of the branch.

When attempting to push the branch to an HTTP smart server (apache+mod_wsgi) I get the following traceback on the server:

2302.623 _iter_with_errors caught error
2302.625 Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/protocol.py", line 1277, in _iter_with_errors
    yield None, iterator.next()
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/repository.py", line 482, in body_stream
    for bytes in byte_stream:
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/repository.py", line 506, in _stream_to_byte_stream
    for record in substream:
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/groupcompress_repo.py", line 1114, in wrap_and_count
    for record in stream:
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/groupcompress_repo.py", line 1095, in _filter_id_to_entry
    chk_map._bytes_to_text_key):
  File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/groupcompress_repo.py", line 1214, in _filter_text_keys
    text_keys_update([bytes_to_text_key(b) for n,b in items])
  File "_chk_map_pyx.pyx", line 416, in bzrlib._chk_map_pyx._bytes_to_text_key
TypeError: bytes must be a string

If I access the same server using bzr+ssh, I can push the branch successfully (but I can't branch/pull with bzr+http still).

Disabling the _chk_map_pyx module by renaming the .so file allowed the pure-python fallback to function correctly.

Bazaar (bzr) 2.2.1
  Python interpreter: /usr/bin/python 2.6.5
  Python standard library: /usr/lib/python2.6
  Platform: Linux-2.6.32-24-server-x86_64-with-Ubuntu-10.04-lucid
  bzrlib: /usr/lib/python2.6/dist-packages/bzrlib

[Andrew Bennetts (spiv)]

I'm glad you have a successful workaround!

This is a bit strange, I wonder what's going on. At a guess it might be a unicode vs. bytes issue. I've got a simple patch to make that error slightly more informative, to hopefully confirm if that's the case.

[Glen Mailer (glenjamin)]

I hate this bug.

I branched bzr's trunk, modified the Pyrex source to improve the error message, and ran make.

I replaced the .so in my existing bazaar installation with the one i'd just built from trunk, and I can no longer reproduce the bug.

The only other thing I can think of that might be different (assuming the _chk_map_pyx file hasn't changed between 2.2.1 and head), is that the Bazaar I have installed is from the PPA, and the one I built required me to install Cython - I'm unsure what was used in the PPA version to build the pyx files.

[John Szakmeister (jszakmeister)]

I'm seeing this problem as well with bzr 2.2.1. I can't make the branches available (sorry) but I can help test a fix.

[Vincent Ladeuil (vila)]

@John: A branch is associated with this bug, can you try it, making sure you compile the right extension and give feedback ?

[Glen Mailer (glenjamin)]

I've had this issue come back to bite me again after upgrading. The patch from this bug improved the error message.

30.782 Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/request.py", line 355, in _call_converting_errors
    return callable(*args, **kwargs)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/repository.py", line 760, in _inserter_thread
    stream, src_format, self.tokens)
  File "/usr/lib/pymodules/python2.6/bzrlib/repository.py", line 4105, in insert_stream
    hint = self.target_repo.commit_write_group()
  File "/usr/lib/pymodules/python2.6/bzrlib/repository.py", line 1640, in commit_write_group
    result = self._commit_write_group()
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/pack_repo.py", line 2333, in _commit_write_group
    hint = self._pack_collection._commit_write_group()
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/pack_repo.py", line 2167, in _commit_write_group
    problems = self._check_new_inventories()
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/groupcompress_repo.py", line 761, in _check_new_invento$
    chk_map._bytes_to_text_key):
  File "/usr/lib/pymodules/python2.6/bzrlib/repofmt/groupcompress_repo.py", line 1338, in _filter_text_keys
    text_keys_update([bytes_to_text_key(b) for n,b in items])
  File "_chk_map_pyx.pyx", line 416, in bzrlib._chk_map_pyx._bytes_to_text_key
TypeError: bytes must be a string, got <type 'bzrlib._static_tuple_c.StaticTuple'>

[Vincent Ladeuil (vila)]

@Glen: After upgrading to which version ?

Is there a server involved too that needs to be upgraded ?

[Glen Mailer (glenjamin)]

The server was upgraded to 2.3.0, the clients are unchanged - I checked one that I think produced the error and it was on 2.2.1.

[Vincent Ladeuil (vila)]

That's weird, this sounds like a very old bug or may be an out-to-date extension.

Do you run from sources ? Did you run make to recompile the extensions ? (Both client and server depending on how you installed them).

Re-reading the comment I realized you probably checked the extensions already but...

[Glen Mailer (glenjamin)]

The server is installed via the Bazaar PPA on Ubuntu 10.10.

Interestingly, the error has been ocurring at roughly the same time each day this week (circa 11am).

The reason it stopped happening between reporting the bug and now would be my removal of the _chk_map_pyx.so file, which the upgrade restored.

It happens on commit over HTTPS+urllib transport, so my presumption is there's some weird edgecase in a checksum that involves the time over this transport. Which sounds crazy, and is merely a vaguely educated guess.

[Vincent Ladeuil (vila)]

A bug in the httpS stack sounds too scary to be true, the consequences would crop up all over the place.

The fact that deleting an extension makes a change is more likely (the python fallback implementation is different enough to explain a bug specific to the extension and also ~rules out a problem in the HTTPS stack).

The string/StaticTuple mismatch in particular has been encountered in other bugs, maybe jam knows better than me there.

[Glen Mailer (glenjamin)]

The remaining odd part is that with the extension enabled, the error happens over HTTPS, but not SSH. Unfortunately it's intermittent enough that I've been unable to try it over other transports (since HTTPS and SSH are the two we have set up for the server).

[John A Meinel (jameinel)]

Looking at the error, it looks like something where we are expecting a raw text key, but are seeing an unexpanded tuple in the CHKMap key. Which is valid if a subreference hasn't been expanded. I'd have to poke through the code a bit more to figure out why it is expecting text keys to be expanded there.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 3/8/2011 9:25 PM, John A Meinel wrote:
> Looking at the error, it looks like something where we are expecting a
> raw text key, but are seeing an unexpanded tuple in the CHKMap key.
> Which is valid if a subreference hasn't been expanded. I'd have to poke
> through the code a bit more to figure out why it is expecting text keys
> to be expanded there.
>

Actually, being HTTP might be a clue. HTTP is truly stateless, while SSH
is not.

Which means that if you start passing in some data, then resume it
later, on SSH the data will still be cached. Something to think about,
at least.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk12wZIACgkQJdeBCYSNAAOs+QCggLPNnE2vguh/TDQ05bYFQlx+
m5YAoMi+2KR4N3AEm66CZEEnij7/hrmq
=hPne
-----END PGP SIGNATURE-----

[James Gregory-Monk (jamgregory)]

So we're back to seeing the error that @Glen described again, and rather than deleting a library, I'd like to try and find out why this is happening.

The Apache server is using WSGI to serve up the Bazaar branches - could this be an influencing factor? I'm not sure why that was chosen over FastCGI/native Python - might it be worth trying a different method to see if that has an effect?

[Raman Muthukrishnan (7-raman)]

We faced the same problem. We have python2.7.2, mod_wsgi 3.3 on Ubuntu 11.10
Only with checkout we see the issue. With checkout --lightweight the issue is not seen.
We removed the WSGIApplicationGroup %{GLOBAL}, and the problem did not occur. But this is not a workaround as I think this should be defined for plugins to work.
Any solution or workaround will be helpful.

Thanks