Simultaneous recipe builds fail with 503 error

[Erich E. Hoover]

When we build our package on the build server we have to make sure we only build one or two at a time, otherwise the packages all fail to build with "Unable to handle http code 503: Service Temporarily Unavailable".

Relevant log portion:
bzr: ERROR: bzrlib.errors.InvalidHttpResponse: Invalid http response for http://bazaar.launchpad.net/~vcs-imports/wine/master-git/.bzr/repository/indices/9973a10a406130078505cc3c175d75cd.rix: Unable to handle http code 503: Service Temporarily Unavailable

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 930, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 1121, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 673, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 697, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 136, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 166, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/builder/cmds.py", line 360, in run
    build_tree(base_branch, working_directory)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/builder/recipe.py", line 661, in build_tree
    possible_transports=possible_transports)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/builder/recipe.py", line 638, in _build_inner_tree
    instruction.apply(target_path, tree_to, br_to)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/builder/recipe.py", line 721, in apply
    possible_transports=possible_transports)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/builder/recipe.py", line 447, in merge_branch
    other_branch=child_branch.branch, tree_branch=br_to)
  File "/usr/lib/python2.7/dist-packages/bzrlib/merge.py", line 425, in from_revision_ids
    merger.find_base()
  File "/usr/lib/python2.7/dist-packages/bzrlib/merge.py", line 544, in find_base
    lcas = self.revision_graph.find_lca(revisions[0], revisions[1])
  File "/usr/lib/python2.7/dist-packages/bzrlib/graph.py", line 294, in find_lca
    border_common, common, sides = self._find_border_ancestors(revisions)
  File "/usr/lib/python2.7/dist-packages/bzrlib/graph.py", line 762, in _find_border_ancestors
    new_ancestors = searcher.step()
  File "/usr/lib/python2.7/dist-packages/bzrlib/graph.py", line 1387, in step
    return self.next()
  File "/usr/lib/python2.7/dist-packages/bzrlib/graph.py", line 1409, in next
    self._advance()
  File "/usr/lib/python2.7/dist-packages/bzrlib/graph.py", line 1443, in _advance
    found, ghosts, next, parents = self._do_query(self._next_query)
  File "/usr/lib/python2.7/dist-packages/bzrlib/graph.py", line 1467, in _do_query
    parent_map = self._parents_provider.get_parent_map(revisions)
  File "/usr/lib/python2.7/dist-packages/bzrlib/graph.py", line 118, in get_parent_map
    new_found = parents_provider.get_parent_map(remaining)
  File "/usr/lib/python2.7/dist-packages/bzrlib/vf_repository.py", line 1882, in get_parent_map
    self.revisions.get_parent_map(query_keys).iteritems():
  File "/usr/lib/python2.7/dist-packages/bzrlib/groupcompress.py", line 1356, in get_parent_map
    return self._get_parent_map_with_sources(keys)[0]
  File "/usr/lib/python2.7/dist-packages/bzrlib/groupcompress.py", line 1375, in _get_parent_map_with_sources
    new_result = source.get_parent_map(missing)
  File "/usr/lib/python2.7/dist-packages/bzrlib/groupcompress.py", line 2097, in get_parent_map
    for node in nodes:
  File "/usr/lib/python2.7/dist-packages/bzrlib/groupcompress.py", line 2069, in _get_entries
    for node in self._graph_index.iter_entries(keys):
  File "/usr/lib/python2.7/dist-packages/bzrlib/index.py", line 1368, in iter_entries
    for node in index.iter_entries(keys):
  File "/usr/lib/python2.7/dist-packages/bzrlib/btree_index.py", line 1194, in iter_entries
    nodes, nodes_and_keys = self._walk_through_internal_nodes(needed_keys)
  File "/usr/lib/python2.7/dist-packages/bzrlib/btree_index.py", line 1148, in _walk_through_internal_nodes
    nodes = self._get_leaf_nodes(node_indexes)
  File "/usr/lib/python2.7/dist-packages/bzrlib/btree_index.py", line 989, in _get_leaf_nodes
    found = self._get_nodes(self._leaf_node_cache, node_indexes)
  File "/usr/lib/python2.7/dist-packages/bzrlib/btree_index.py", line 966, in _get_nodes
    found.update(self._get_and_cache_nodes(needed))
  File "/usr/lib/python2.7/dist-packages/bzrlib/btree_index.py", line 743, in _get_and_cache_nodes
    for node_pos, node in self._read_nodes(sorted(nodes)):
  File "/usr/lib/python2.7/dist-packages/bzrlib/btree_index.py", line 1566, in _read_nodes
    for offset, data in data_ranges:
  File "/usr/lib/python2.7/dist-packages/bzrlib/transport/http/__init__.py", line 201, in _readv
    for cur_coal, rfile in self._coalesce_readv(relpath, coalesced):
  File "/usr/lib/python2.7/dist-packages/bzrlib/transport/http/__init__.py", line 293, in _coalesce_readv
    for c, rfile in get_and_yield(relpath, ranges):
  File "/usr/lib/python2.7/dist-packages/bzrlib/transport/http/__init__.py", line 256, in get_and_yield
    code, rfile = self._get(relpath, coalesced)
  File "/usr/lib/python2.7/dist-packages/bzrlib/transport/http/_urllib.py", line 120, in _get
    response = self._perform(request)
  File "/usr/lib/python2.7/dist-packages/bzrlib/transport/http/_urllib.py", line 76, in _perform
    response = self._opener.open(request)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/dist-packages/bzrlib/transport/http/_urllib2_wrappers.py", line 1806, in http_response
    code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/dist-packages/bzrlib/transport/http/_urllib2_wrappers.py", line 1823, in http_error_default
    % (code, msg))
InvalidHttpResponse: Invalid http response for http://bazaar.launchpad.net/~vcs-imports/wine/master-git/.bzr/repository/indices/9973a10a406130078505cc3c175d75cd.rix: Unable to handle http code 503: Service Temporarily Unavailable
resource.struct_rusage(ru_utime=60.278022, ru_stime=2.641441, ru_maxrss=242228, ru_ixrss=0, ru_idrss=0, ru_isrss=0, ru_minflt=79783, ru_majflt=24, ru_nswap=0, ru_inblock=9192, ru_oublock=372024, ru_msgsnd=0, ru_msgrcv=0, ru_nsignals=0, ru_nvcsw=14630, ru_nivcsw=1732)

[Sebastian Lackner (slackner)]

The problem is getting worse. Until about three days ago it was still possible to push at least two builds at the same time, but now even with just a single build at a time it triggers the 503 error most of the time. This makes it almost impossible to push out new packages. We had a release yesterday, and I am still busy with clicking the "rebuild" button all the time, until it eventually succeeds. :/

Here a recent example of such a build failure:
https://launchpadlibrarian.net/200945517/buildlog.txt.gz

[Michael Müller (mqchael)]

The situation is so bad now that it is almost impossible to build any wine packages. You need like 20 attempts before it finally works. Our commercial subscription is pretty much useless now and we think about migrating away from launchpad.

[William Grant (wgrant)]

Things indeed got markedly worse around Monday, finally making the proxy connection pooling problem clear. We've applied a fix and I've tested recipe builds from your branches, and they now work fine. Let us know if you see any more issues.