PermissionDenied test failures on windows with sftp

[Martin Packman]

Various tests involving an sftp server on windows fail randomly, but now more consistently, with PermissionDenied while trying to operate on a .pack file. This is similar but not obviously related to the earlier problems bug 681047 covers. A example set of failing tests can be seen on babune:

<http://babune.ladeuil.net:24842/view/Windows/job/selftest-windows/593/>

There are a couple of other issues causing failures (see bug 842233 and bug 8891582 for those), the relevent ones from that run are:

  bb.test_too_much.SFTPTestsAbsoluteSibling.test_push
  bb.test_too_much.SFTPTestsRelative.test_push

  bb.test_branch.TestRemoteBranch.test_branch_remote_remote

  bt.commands.test_commit.TestCommitWithBoundBranch.test_commit_mine_modified
  bt.commands.test_pull.TestPull.test_pull_with_bound_branch

  bt.per_branch.test_stacking.TestStackingConnections.test_open_stacked(RemoteBranchFormat-default)
  bt.per_branch.test_stacking.TestStackingConnections.test_open_stacked(RemoteBranchFormat-v2)
  bt.per_branch.test_stacking.TestStackingConnections.test_open_stacked_relative(RemoteBranchFormat-default)

These tests on babune were also affected the big regression in r6079 and started failing consistently with "Attempt to escape test isolation":

<http://babune.ladeuil.net:24842/view/Windows/job/selftest-windows/503/>

That problem is covered in bug 842223, but with the fix PermissionDenied errors returned, and appear more widespread. Another change in the meanwhile may well have been caused or aggravated this. Some of the tests at least were already failing before then, for instance:

<http://babune.ladeuil.net:24842/view/Windows/job/selftest-windows/494/testReport/bzrlib.tests.blackbox.test_branch/TestRemoteBranch/test_branch_remote_remote/>

The general form of the tracebacks is:

Traceback (most recent call last):
  File "...\bzrlib\tests\commands\test_pull.py", line 56, in test_pull_with_bound_branch
    pull.run(self.get_url('remote'), directory='local')
  File "...\bzrlib\commands.py", line 698, in run
    return self._operation.run_simple(*args, **kwargs)
  File "...\bzrlib\cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "...\bzrlib\cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "...\bzrlib\builtins.py", line 1106, in run
    local=local, show_base=show_base)
  File "...\bzrlib\decorators.py", line 217, in write_locked
    result = unbound(self, *args, **kwargs)
  File "...\bzrlib\workingtree.py", line 989, in pull
    local=local)
  File "...\bzrlib\branch.py", line 1127, in pull
    possible_transports=possible_transports, *args, **kwargs)
  File "...\bzrlib\decorators.py", line 217, in write_locked
    result = unbound(self, *args, **kwargs)
  File "...\bzrlib\branch.py", line 3427, in pull
    run_hooks=False)
  File "...\bzrlib\branch.py", line 1127, in pull
    possible_transports=possible_transports, *args, **kwargs)
  File "...\bzrlib\decorators.py", line 217, in write_locked
    result = unbound(self, *args, **kwargs)
  File "...\bzrlib\branch.py", line 3432, in pull
    merge_tags_to_master=not source_is_master)
  File "...\bzrlib\branch.py", line 3562, in _pull
    graph=graph)
  File "...\bzrlib\decorators.py", line 217, in write_locked
    result = unbound(self, *args, **kwargs)
  File "...\bzrlib\branch.py", line 3375, in _update_revisions
    self.fetch(stop_revision=stop_revision)
  File "...\bzrlib\decorators.py", line 217, in write_locked
    result = unbound(self, *args, **kwargs)
  File "...\bzrlib\branch.py", line 3352, in fetch
    fetch_spec=fetch_spec)
  File "...\bzrlib\repository.py", line 714, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "...\bzrlib\decorators.py", line 217, in write_locked
    result = unbound(self, *args, **kwargs)
  File "...\bzrlib\vf_repository.py", line 2518, in fetch
    find_ghosts=find_ghosts)
  File "...\bzrlib\fetch.py", line 76, in __init__
    self.__fetch()
  File "...\bzrlib\fetch.py", line 103, in __fetch
    self._fetch_everything_for_search(search_result)
  File "...\bzrlib\fetch.py", line 131, in _fetch_everything_for_search
    stream, from_format, [])
  File "...\bzrlib\vf_repository.py", line 1986, in insert_stream
    hint = self.target_repo.commit_write_group()
  File "...\bzrlib\repository.py", line 631, in commit_write_group
    result = self._commit_write_group()
  File "...\bzrlib\repofmt\pack_repo.py", line 1711, in _commit_write_group
    hint = self._pack_collection._commit_write_group()
  File "...\bzrlib\repofmt\pack_repo.py", line 1596, in _commit_write_group
    self._new_pack.finish()
  File "...\bzrlib\repofmt\pack_repo.py", line 509, in finish
    self.upload_transport.move(self.random_name, new_name)
  File "...\bzrlib\transport\sftp.py", line 753, in move
    self._rename_and_overwrite(path_from, path_to)
  File "...\bzrlib\transport\sftp.py", line 747, in _rename_and_overwrite
    ': unable to rename to %r' % (abs_to))
  File "...\bzrlib\transport\sftp.py", line 686, in _translate_io_exception
    self._translate_error(e, path, raise_generic=False)
  File "...\bzrlib\transport\__init__.py", line 335, in _translate_error
    raise errors.PermissionDenied(path, extra=e)
PermissionDenied: Permission denied: "/C:/docume~1/babune/locals~1/temp/testbzr-cb_kiq.tmp/ll.test_pull_with_bound_branch/work/master/.bzr/repository/upload/g58ycl2gboebk16gd8s7.pack": [Errno 13] Permission denied

The initial forward slash on the path renders it invalid, but must be just a cosmetic problem with the error reporting as the tests do sometimes pass.

A number of tests also report a traceback during cleanup like:

Traceback (most recent call last):
  File "...\bzrlib\repository.py", line 280, in abort_write_group
    self._abort_write_group()
  File "...\bzrlib\repofmt\pack_repo.py", line 1692, in _abort_write_group
    self._pack_collection._abort_write_group()
  File "...\bzrlib\repofmt\pack_repo.py", line 1549, in _abort_write_group
    operation.run_simple()
  File "...\bzrlib\cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "...\bzrlib\cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "...\bzrlib\repofmt\pack_repo.py", line 432, in abort
    self.write_stream.close()
  File "...\bzrlib\transport\__init__.py", line 237, in close
    del _file_streams[self.transport.abspath(self.relpath)]
KeyError: 'hooked://foo@127.0.0.1:1340/C%3A/docume~1/babune/locals~1/temp/testbzr-cb_kiq.tmp/ll.test_pull_with_bound_branch/work/master/.bzr/repository/upload/g58ycl2gboebk16gd8s7.pack'

Which is likely a seperate bug in the rollback code.

[Martin Packman (gz)]

Missed bug 788130 which is about this issue on one of the tests, from when it was a more intermittent and less prevalent problem.