qlog: TooManyConcurrentRequests searching on a remote branch while revisions are loading due to scrolling.

[Gary van der Merwe]

bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium 'SmartSSHClientMedium(connected=True, username=u'garyvdm', host='bazaar.launchpad.net', port=None)' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

Traceback (most recent call last):
  File "/home/garyvdm/qbzr/wd/qbzr/lib/uifactory.py", line 31, in decorate
    r = f(*args, **kargs)
  File "/home/garyvdm/qbzr/wd/qbzr/lib/log.py", line 560, in update_search
    self.log_list.set_search(search_text, field)
  File "/home/garyvdm/qbzr/wd/qbzr/lib/logwidget.py", line 316, in set_search
    self.graph_provider.set_search(str, field)
  File "/home/garyvdm/qbzr/wd/qbzr/lib/loggraphprovider.py", line 1571, in set_search
    revisions_loaded = revisions_loaded)
  File "/home/garyvdm/qbzr/wd/qbzr/lib/loggraphprovider.py", line 1602, in load_revisions
    *args, **kargs)
  File "/home/garyvdm/qbzr/wd/qbzr/lib/lazycachedrevloader.py", line 109, in load_revisions
    for rev in repo.get_revisions(batch_revids):
  File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/remote.py", line 1589, in get_revisions
    return self._real_repository.get_revisions(revision_ids)
  File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1855, in get_revisions
    return self._get_revisions(revision_ids)
  File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1861, in _get_revisions
    for revid, rev in self._iter_revisions(revision_ids):
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1885, in _iter_revisions
    for record in stream:
  File "/usr/lib/python2.6/dist-packages/bzrlib/knit.py", line 1398, in get_record_stream
    ordering, include_delta_closure):
  File "/usr/lib/python2.6/dist-packages/bzrlib/knit.py", line 1409, in _get_remaining_record_stream
    positions = self._get_components_positions(keys, allow_missing=True)
  File "/usr/lib/python2.6/dist-packages/bzrlib/knit.py", line 1166, in _get_components_positions
    build_details = self._index.get_build_details(pending_components)
  File "/usr/lib/python2.6/dist-packages/bzrlib/knit.py", line 3015, in get_build_details
    for entry in entries:
  File "/usr/lib/python2.6/dist-packages/bzrlib/knit.py", line 3043, in _get_entries
    for node in self._graph_index.iter_entries(keys):
  File "/usr/lib/python2.6/dist-packages/bzrlib/index.py", line 1290, in iter_entries
    for node in index.iter_entries(keys):
  File "/usr/lib/python2.6/dist-packages/bzrlib/btree_index.py", line 1101, in iter_entries
    if not self.key_count():
  File "/usr/lib/python2.6/dist-packages/bzrlib/btree_index.py", line 1383, in key_count
    self._get_root_node()
  File "/usr/lib/python2.6/dist-packages/bzrlib/btree_index.py", line 879, in _get_root_node
    self._get_internal_nodes([0])
  File "/usr/lib/python2.6/dist-packages/bzrlib/btree_index.py", line 904, in _get_internal_nodes
    return self._get_nodes(self._internal_node_cache, node_indexes)
  File "/usr/lib/python2.6/dist-packages/bzrlib/btree_index.py", line 896, in _get_nodes
    found.update(self._get_and_cache_nodes(needed))
  File "/usr/lib/python2.6/dist-packages/bzrlib/btree_index.py", line 686, in _get_and_cache_nodes
    for node_pos, node in self._read_nodes(sorted(nodes)):
  File "/usr/lib/python2.6/dist-packages/bzrlib/btree_index.py", line 1494, in _read_nodes
    for offset, data in data_ranges:
  File "/usr/lib/python2.6/dist-packages/bzrlib/transport/remote.py", line 367, in _readv
    [(c.start, c.length) for c in cur_request])
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/client.py", line 172, in call_with_body_readv_array
    args[0], args[1:], readv_body=body, expect_response_body=True)
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/client.py", line 76, in _call_and_read_response
    readv_body=readv_body, body_stream=body_stream)
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/client.py", line 42, in _send_request
    protocol_version)
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/client.py", line 112, in _construct_protocol
    request = self._medium.get_request()
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/medium.py", line 699, in get_request
    return SmartClientStreamMediumRequest(self)
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/medium.py", line 904, in __init__
    raise errors.TooManyConcurrentRequests(self._medium)
TooManyConcurrentRequests: The medium 'SmartSSHClientMedium(connected=True, username=u'garyvdm', host='bazaar.launchpad.net', port=None)' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

bzr 2.0rc1 on python 2.6.2 (linux2)
arguments: ['/usr/bin/bzr', 'qlog', 'lp:qbzr']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_ZA.UTF-8'
plugins:
  builddeb /home/garyvdm/qbzr/wd/builddeb [2.1.1dev]
  launchpad /usr/lib/python2.6/dist-packages/bzrlib/plugins/launchpad [2.0rc1]
  netrc_credential_store /usr/lib/python2.6/dist-packages/bzrlib/plugins/netrc_credential_store [2.0rc1]
  qbzr /home/garyvdm/qbzr/wd/qbzr [0.15dev]

[Gary van der Merwe (garyvdm)]

Note to self..

This bug is going to need a bit of refactoring to fix correctly. lazycachedrevloader.load_revisions should allways be run in the loading queue, but as some of it callers call it in a synchronous fasion, while others in a asynchronous fasion, hence we cannot just decorate it with @runs_in_loading_queue.

There are 2 way we can fix this...
Option 1:
---------
We need to change the synchronous callers to be asynchronous fasion. Then just decorate lazycachedrevloader.load_revisions with @runs_in_loading_queue

* Pro: Better design ??? - I guess this will make it easier to change to the much talked about 2 thread design.
* Con: The synchronous callers become alot more complicated.

Option 2:
---------
Have 2 api calls, one for synchronous, one for asynchronous. The asynchronous call will run in the loading queue, but the synchronous one will not, and it will be the callers responsibility to run in the loading queue.

* Pro: The synchronous callers don't change much (stay simple).

Synchronous Calls
  C:\Program Files\Bazaar\plugins\qbzr\lib\annotate.py
 Line 478: rev = gp.load_revisions([revid])[revid]
  C:\Program Files\Bazaar\plugins\qbzr\lib\commit.py
 Line 200: rev = self.graph_provider.load_revisions([revid])[revid]
  C:\Program Files\Bazaar\plugins\qbzr\lib\log.py
 Line 435: all_revs = gp.load_revisions(all_revids)
  C:\Program Files\Bazaar\plugins\qbzr\lib\loggraphprovider.py
 Line 422: head_revs = self.load_revisions(self.head_revids)
Most of the synchronous callers (the first 3 listed) are using the results to show using format_revision_html. These should be refactored to make to code common. This whould mean that we would only need to make the call asynchronous in one place (good.)

The other synchronous call is during the loggraphprovider load call. The whole load call needs to be made asynchronous.

Asynchronous Calls
  C:\Program Files\Bazaar\plugins\qbzr\lib\loggraphprovider.py
 Line 1566: self.load_revisions(revids,
  C:\Program Files\Bazaar\plugins\qbzr\lib\revtreeview.py
 Line 131: load_revisions(revids, model.get_repo(),