update fails with File exists

[basur]

We recently started getting the following errors intermittendly.
They sometimes seem to be going away when a new commit has been added to the branch the checkout is from.
Or they may indeed result in a out of memory error of bzr (maybe that should be another bug report?)

The problem is that in a checkout of the bound branch we try to execute update.
This seems to fail every single time with the error that some ignored error for a pack file followed by a file exists error to the same file.
The file will be different with every update attempt.

A bzr status will show that the working tree is now out of date.
So I thought I try a bzr unbind followed by a bzr update.
This succeeds as below snippet shows.
After that I bound the checkout back to the bound branch location.

This needs to be addressed urgently because it is preventing our automated build agent to run properly and it is costing in productivity of our developers.

bzr version is : 2.3.4 but we have seen the same issue with 2.4b5
bzr is showing the issue on Windows 7, Windows XP and Windows Server 2008 R2.
We were running the standalone install build which seems to be using a python 2.6.6 internally.

Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

<path to checkout>bzr up
Connected (version 2.0, client OpenSSH_4.3)
Authentication (publickey) successful!
Secsh channel 1 opened.
bzr: ERROR (ignored): 'file:///<path to shared repo>/.bzr/repository/upload/m4u3imj5ho2f81806lkk.pack'
bzr: ERROR: File exists: u'<path to shared repo>/.bzr/repository/upload/m4u3imj5ho2f81806lkk.pack': [Error 183] Cannot create a file when that file already exists

<path to checkout>bzr info
Repository checkout (format: unnamed)
Location:
  repository checkout root: .
        checkout of branch: <bound branch location>
         shared repository: <path to shared repo>

<path to checkout>bzr stat
working tree is out of date, run 'bzr update'

<path to checkout>bzr up
Connected (version 2.0, client OpenSSH_4.3)
Authentication (publickey) successful!
Secsh channel 1 opened.
bzr: ERROR (ignored): 'file:///<path to shared repo>/.bzr/repository/upload/pqu389lavz9517d5oucm.pack'
bzr: ERROR: File exists: u'<path to shared repo>/.bzr/repository/upload/pqu389lavz9517d5oucm.pack': [Error 183] Cannot create a file when that file already exists

<path to checkout>bzr info
Repository checkout (format: unnamed)
Location:
  repository checkout root: .
        checkout of branch: <bound branch location>
         shared repository: <path to shared repo>

<path to checkout>bzr unbind

<path to checkout>bzr up
<files being updated>
All changes applied successfully.
Updated to revision 5318 of branch <path to shared repo>/CPlusPlus/xml-data/build-dir/CPP-TR-VSB

<path to checkout>

[John A Meinel (jameinel)]

You have a garbage file in .bzr/repository/upload which is confusing us while fetching new content.

If you have no bzr process running, then you should have no content in .bzr/repository/upload (it is a temporary staging area.)

You can try just deleting all files in that directory (do not delete the directory itself). And see if 'bzr update' works again. It sounds like it is failing to download because it got the same content as a previous download that was killed.

On POSIX systems (Mac and Linux) this won't fail, because it will just overwrite the target file.

It would also be useful to see the full traceback (from your .bzr.log file, see 'bzr --version' for information on where that file lives.)

[basur (basur-au)]

Here is the contents of the bzr.log. The upload folder was empty before the bzr update. After the update the mentioned file did exist.

Mon 2011-08-08 11:45:18 +1000
0.062 bazaar version: 2.3.4
0.062 bzr arguments: [u'update', u'-r5325', u'E:\\BambooAgentData\\CPlusPlus\\xml-data\\build-dir\\CPP-TR-VSB']
0.078 looking for plugins in C:/Users/build/AppData/Roaming/bazaar/2.0/plugins
0.078 looking for plugins in C:/Program Files (x86)/Bazaar/plugins
0.171 encoding stdout as osutils.get_user_encoding() 'cp1252'
0.187 opening working tree '<path to bound checkout>'
0.218 bzr-svn: using Subversion 1.6.6 (), subvertpy 0.7.5
0.265 falling back to default implementation
0.265 failed to load system host keys: [Errno 2] No such file or directory: 'C:\\Users\\build/.ssh/known_hosts'
[13332] 2011-08-08 11:45:18.361 INFO: Connected (version 2.0, client OpenSSH_4.3)
[13332] 2011-08-08 11:45:18.562 INFO: Authentication (publickey) successful!
[13332] 2011-08-08 11:45:18.578 INFO: Secsh channel 1 opened.
0.873 Using fetch logic to copy between RemoteRepository(<remote repository location>)(RemoteRepositoryFormat(_network_name='Bazaar repository format 2a (needs bzr 1.16 or later)\n')) and CHKInventoryRepository('<path to shared repository>/.bzr/repository/')(RepositoryFormat2a())
0.873 fetch up to rev {some-revision-id}
6.287 abort_write_group failed
6.287 Traceback (most recent call last):
  File "bzrlib\repository.pyo", line 1019, in abort_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 2314, in _abort_write_group
  File "bzrlib\repofmt\pack_repo.pyo", line 2129, in _abort_write_group
  File "bzrlib\cleanup.pyo", line 135, in run_simple
  File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
  File "bzrlib\repofmt\pack_repo.pyo", line 438, in abort
  File "bzrlib\transport\__init__.pyo", line 240, in close
KeyError: '<path to shared repository>/.bzr/repository/upload/2ztkfnfy72d4pwc232iq.pack'

[13332] 2011-08-08 11:45:24.367 INFO: bzr: ERROR (ignored): '<path to shared repository>/.bzr/repository/upload/2ztkfnfy72d4pwc232iq.pack'
6.302 Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 927, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1127, in run_bzr
  File "bzrlib\commands.pyo", line 692, in run_argv_aliases
  File "bzrlib\commands.pyo", line 714, in run
  File "bzrlib\cleanup.pyo", line 135, in run_simple
  File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
  File "bzrlib\builtins.pyo", line 1413, in run
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\branch.pyo", line 2803, in update
  File "bzrlib\branch.pyo", line 1089, in pull
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\branch.pyo", line 3492, in pull
  File "bzrlib\branch.pyo", line 3602, in _pull
  File "bzrlib\branch.pyo", line 1038, in update_revisions
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\branch.pyo", line 3435, in update_revisions
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\branch.pyo", line 687, in fetch
  File "bzrlib\repository.pyo", line 1781, in fetch
  File "bzrlib\decorators.pyo", line 194, in write_l...

[basur (basur-au)]

Do you need anything else to look into it further?

[Martin Pool (mbp)]

basur, is this stored on a network drive of any kind?

[basur (basur-au)]

No. The local shared repository and its checkout that cause the problem are all located on a local hard disk.

The remote repository where the bound branch of the checkouts are located is accessed via bzr+ssh and is located on a different server in the network.